From 0db31d584caa42ff9a1e9fec89ddf9b1ffca87b2 Mon Sep 17 00:00:00 2001 From: Zangwei Zheng Date: Sun, 17 Mar 2024 22:16:27 +0800 Subject: [PATCH] update readme --- README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index bb1c5bb..4a1839c 100644 --- a/README.md +++ b/README.md @@ -26,14 +26,14 @@ inference, and more. Our provided [checkpoints](#model-weights) can produce 2~5s ## 🎥 Latest Demo -| **2s 512×512** | **2s 512×512** | **2s 512×512** | -| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [](https://github.com/hpcaitech/Open-Sora/assets/99191637/de1963d3-b43b-4e68-a670-bb821ebb6f80) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/13f8338f-3d42-4b71-8142-d234fbd746cc) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/fa6a65a6-e32a-4d64-9a9e-eabb0ebb8c16) | -| A serene night scene in a forested area. [...] The video is a time-lapse, capturing the transition from day to night, with the lake and forest serving as a constant backdrop. | A soaring drone footage captures the majestic beauty of a coastal cliff, [...] The water gently laps at the rock base and the greenery that clings to the top of the cliff. | The majestic beauty of a waterfall cascading down a cliff into a serene lake. [...] The camera angle provides a bird's eye view of the waterfall. | -| | | | -| A bustling city street at night, filled with the glow of car headlights and the ambient light of streetlights. [...] | The vibrant beauty of a sunflower field. The sunflowers are arranged in neat rows, creating a sense of order and symmetry. [...] | A serene underwater scene featuring a sea turtle swimming through a coral reef. The turtle, with its greenish-brown shell [...] | +| **2s 512×512** | **2s 512×512** | **2s 512×512** | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +| [](https://github.com/hpcaitech/Open-Sora/assets/99191637/de1963d3-b43b-4e68-a670-bb821ebb6f80) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/13f8338f-3d42-4b71-8142-d234fbd746cc) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/fa6a65a6-e32a-4d64-9a9e-eabb0ebb8c16) | +| A serene night scene in a forested area. [...] The video is a time-lapse, capturing the transition from day to night, with the lake and forest serving as a constant backdrop. | A soaring drone footage captures the majestic beauty of a coastal cliff, [...] The water gently laps at the rock base and the greenery that clings to the top of the cliff. | The majestic beauty of a waterfall cascading down a cliff into a serene lake. [...] The camera angle provides a bird's eye view of the waterfall. | +| | | | +| A bustling city street at night, filled with the glow of car headlights and the ambient light of streetlights. [...] | The vibrant beauty of a sunflower field. The sunflowers are arranged in neat rows, creating a sense of order and symmetry. [...] | A serene underwater scene featuring a sea turtle swimming through a coral reef. The turtle, with its greenish-brown shell [...] | -Videos are downsampled to `.gif` for display. Click for original videos. Texts are trimmed for display, see [here](/assets/texts/t2v_samples.txt) for full texts. +Videos are downsampled to `.gif` for display. Click for original videos. Texts are trimmed for display, see [here](/assets/texts/t2v_samples.txt) for full texts. See more samples at our [gallery](https://hpcaitech.github.io/Open-Sora/). ## 🔆 New Features/Updates @@ -112,12 +112,12 @@ After installation, we suggest reading [structure.md](docs/structure.md) to lear ## Model Weights -| Resoluion | Data | #iterations | Batch Size | GPU days (H800) | URL | -| ---------- | ------ | ----------- | ---------- | --------------- | --- | -| 16×256×256 | 366K | 80k | 8×64 | 117 | | -| 16×256×256 | 20K HQ | 24k | 8×64 | 45 | | -| 16×512×512 | 20K HQ | 20k | 2×64 | 35 | | -| 64×512×512 | 50K HQ | | 4×64 | | | +| Resoluion | Data | #iterations | Batch Size | GPU days (H800) | URL | +| ---------- | ------ | ----------- | ---------- | --------------- | ---------- | +| 16×256×256 | 366K | 80k | 8×64 | 117 | [:link:]() | +| 16×256×256 | 20K HQ | 24k | 8×64 | 45 | [:link:]() | +| 16×512×512 | 20K HQ | 20k | 2×64 | 35 | [:link:]() | +| 64×512×512 | 50K HQ | | | | TBD | Our model's weight is partially initialized from [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha). The number of parameters is 724M. More information about training can be found in our **[report](/docs/report_v1.md)**. More about dataset can be found in [dataset.md](/docs/dataset.md). HQ means high quality.