nicolabs/Open-Sora
1
0
Fork 0
mirror of https://github.com/hpcaitech/Open-Sora.git synced 2026-05-21 11:59:01 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

update docs

Browse source
This commit is contained in:
Zangwei Zheng 2024-03-16 23:29:14 +08:00
parent f41ac61290
commit acc1627c7d
2 changed files with 24 additions and 22 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
README.md
View file

@ -30,7 +30,7 @@ Videos are downsampled to `.gif` for display. Click the video for original ones.
## 🔆 New Features/Updates
* 📍 Open-Sora-v1 released. Model weights are available [here](#model-weights). With only 400K video clips and 200 H800 days, we are able to generate 2s 512×512 videos.
* 📍 Open-Sora-v1 released. Model weights are available [here](#model-weights). With only 400K video clips and 200 H800 days (compared with 152M samples in Stable Video Diffusion), we are able to generate 2s 512×512 videos.
* ✅ Three stages training from an image diffusion model to a video diffusion model. We provide the weights for each stage.
* ✅ Support training acceleration including accelerated transformer, faster T5 and VAE, and sequence parallelism. Open-Sora improve **55%** training speed when training on 64x512x512 videos. Details locates at [acceleration.md](docs/acceleration.md).
* ✅ We provide video cutting and captioning tools for data preprocessing. Instructions can be found [here](tools/data/README.md) and our data collection plan can be found at [datasets.md](docs/datasets.md).
@ -49,7 +49,7 @@ Videos are downsampled to `.gif` for display. Click the video for original ones.
### TODO list sorted by priority
* [ ] Complete the data processing pipeline (including dense optical flow, aesthetics scores, text-image similarity, deduplication, etc.). See [datasets.md]() for more information. **[WIP]**
* [ ] Complete the data processing pipeline (including dense optical flow, aesthetics scores, text-image similarity, deduplication, etc.). See [datasets.md](/docs/datasets.md) for more information. **[WIP]**
* [ ] Training Video-VAE. **[WIP]**
<details>
@ -113,6 +113,8 @@ After installation, we suggest reading [structure.md](docs/structure.md) to lear
Our model's weight is partially initialized from [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha). The number of parameters is 724M. More information about training can be found in [report_v1.md](/docs/report_v1.md). More about dataset can be found in [dataset.md](/docs/dataset.md).
**LIMITATION**: Our model is trained on a limited budget. The quality and text alignment are not strong. The model performs badly especially on generating human activities and cannot follow detailed instructions. We are working on improving the quality and text alignment.
## Inference
To run inference with our provided weights, first download [T5](https://huggingface.co/DeepFloyd/t5-v1_1-xxl/tree/main) weights into `pretrained_models/t5_ckpts/t5-v1_1-xxl`. Then run the following commands to generate samples. See [here](docs/structure.md#inference-config-demos) to customize the configuration.

40
docs/acceleration.md
View file

@ -12,11 +12,11 @@ Open-Sora boosts the training speed by:
Our training speed on images is comparable to [OpenDiT](https://github.com/NUS-HPC-AI-Lab/OpenDiT), an project to accelerate DiT training. The training speed is measured on 8 H800 GPUs with batch size 128, image size 256x256.
| Model | Throughput (img/s) | Throughput (tokens/s) |
| -------- | ------------------ | --------------------- |
| DiT | | |
| OpenDiT | | |
| OpenSora | 175 | 45k |
| Model | Throughput (img/s/GPU) | Throughput (tokens/s/GPU) |
| -------- | ---------------------- | ------------------------- |
| DiT | | |
| OpenDiT | 1100 | |
| OpenSora | 1400 | 45k |
## Efficient STDiT
@ -24,15 +24,15 @@ Our STDiT adopts spatial-temporal attention to model the video data. Compared wi
The training speed is measured on 8 H800 GPUs with acceleration techniques applied, GC means gradient checkpointing. Both with T5 conditioning like PixArt.
| Model | Setting | Throughput (sample/s) | Throughput (tokens/s) |
| ---------------- | -------------- | --------------------- | --------------------- |
| DiT | 16x256 (4k) | 7.20 | 29k |
| STDiT | 16x256 (4k) | 7.00 | 28k |
| DiT | 16x512 (16k) | 0.85 | 14k |
| STDiT | 16x512 (16k) | 1.45 | 23k |
| DiT (GC) | 64x512 (65k) | 0.08 | 5k |
| STDiT (GC) | 64x512 (65k) | 0.40 | 25k |
| STDiT (GC, sp=2) | 360x512 (370k) | 0.10 | 18k |
| Model | Setting | Throughput (sample/s/GPU) | Throughput (tokens/s/GPU) |
| ---------------- | -------------- | ------------------------- | ------------------------- |
| DiT | 16x256 (4k) | 7.20 | 29k |
| STDiT | 16x256 (4k) | 7.00 | 28k |
| DiT | 16x512 (16k) | 0.85 | 14k |
| STDiT | 16x512 (16k) | 1.45 | 23k |
| DiT (GC) | 64x512 (65k) | 0.08 | 5k |
| STDiT (GC) | 64x512 (65k) | 0.40 | 25k |
| STDiT (GC, sp=2) | 360x512 (370k) | 0.10 | 18k |
With a 4x downsampling in the temporal dimension with Video-VAE, an 24fps video has 450 frames. The gap between the speed of STDiT (28k tokens/s) and DiT on images (up to 45k tokens/s) mainly comes from the T5 and VAE encoding, and temperal attention.
@ -49,9 +49,9 @@ For option 2, we boost T5 speed and memory requirement. According to [OpenDiT](h
The training speed is measured on 8 H800 GPUs with STDiT.
| Acceleration | Setting | Throughput (img/s) | Throughput (tokens/s) |
| ------------ | ------------- | ------------------ | --------------------- |
| Baseline | 16x256 (4k) | 6.16 | 25k |
| w. faster T5 | 16x256 (4k) | 7.00 | 29k |
| Baseline | 64x512 (65k) | 0.94 | 15k |
| w. both | 64x512 (65k) | 1.45 | 23k |
| Acceleration | Setting | Throughput (img/s/GPU) | Throughput (tokens/s/GPU) |
| ------------ | ------------- | ---------------------- | ------------------------- |
| Baseline | 16x256 (4k) | 6.16 | 25k |
| w. faster T5 | 16x256 (4k) | 7.00 | 29k |
| Baseline | 64x512 (65k) | 0.94 | 15k |
| w. both | 64x512 (65k) | 1.45 | 23k |