mirror of
https://github.com/hpcaitech/Open-Sora.git
synced 2026-04-11 05:13:31 +02:00
update docs
This commit is contained in:
Zangwei Zheng
parent
c07cfd4387
commit
4da80a4778
30
README.md
30
README.md
|
|
@ -37,15 +37,16 @@ With Open-Sora, we aim to inspire innovation, creativity, and inclusivity in the
|
|||
|
||||
## 🎥 Latest Demo
|
||||
|
||||
More samples are available in our [gallery](https://hpcaitech.github.io/Open-Sora/).
|
||||
|
||||
| **2s 240×426** | **2s 240×426** |
|
||||
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| [<img src="assets/demo/sample_16x240x426_9.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c31ebc52-de39-4a4e-9b1e-9211d45e05b2) | [<img src="assets/demo/sora_16x240x426_26.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c31ebc52-de39-4a4e-9b1e-9211d45e05b2) |
|
||||
| [<img src="assets/demo/sora_16x240x426_27.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/f7ce4aaa-528f-40a8-be7a-72e61eaacbbd) | [<img src="assets/demo/sora_16x240x426_40.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/5d58d71e-1fda-4d90-9ad3-5f2f7b75c6a9) |
|
||||
| **2s 240×426** | **2s 240×426** |
|
||||
| ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| [<img src="assets/demo/sample_16x240x426_9.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c31ebc52-de39-4a4e-9b1e-9211d45e05b2) | [<img src="assets/demo/sora_16x240x426_26.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c31ebc52-de39-4a4e-9b1e-9211d45e05b2) |
|
||||
| [<img src="assets/demo/sora_16x240x426_27.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/f7ce4aaa-528f-40a8-be7a-72e61eaacbbd) | [<img src="assets/demo/sora_16x240x426_40.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/5d58d71e-1fda-4d90-9ad3-5f2f7b75c6a9) |
|
||||
|
||||
| **2s 426×240** | **2s 426×240** | **4s 480×854** |
|
||||
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| [<img src="assets/demo/sora_16x426x240_24.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/34ecb4a0-4eef-4286-ad4c-8e3a87e5a9fd) | [<img src="assets/demo/sora_16x426x240_3.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/3e892ad2-9543-4049-b005-643a4c1bf3bf) | [<img src="assets/demo/sample_32x480x854_9.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c1619333-25d7-42ba-a91c-18dbc1870b18) |
|
||||
| **2s 426×240** | **2s 426×240** | **4s 480×854** |
|
||||
| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| [<img src="assets/demo/sora_16x426x240_24.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/34ecb4a0-4eef-4286-ad4c-8e3a87e5a9fd) | [<img src="assets/demo/sora_16x426x240_3.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/3e892ad2-9543-4049-b005-643a4c1bf3bf) | [<img src="assets/demo/sample_32x480x854_9.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c1619333-25d7-42ba-a91c-18dbc1870b18) |
|
||||
|
||||
|
||||
<details>
|
||||
|
|
@ -63,8 +64,6 @@ see [here](/assets/texts/t2v_samples.txt) for full prompts.
|
|||
|
||||
</details>
|
||||
|
||||
More samples are available in our [gallery](https://hpcaitech.github.io/Open-Sora/).
|
||||
|
||||
## 🔆 New Features/Updates
|
||||
|
||||
* 📍 **Open-Sora 1.1** released. Model weights are available [here](). It is trained on **0s~15s, 144p to 720p, various aspect ratios** videos. See our **[report 1.1](docs/report_02.md)** for more discussions.
|
||||
|
|
@ -176,11 +175,10 @@ pip install -v .
|
|||
|
||||
### Open-Sora 1.1 Model Weights
|
||||
|
||||
| Resolution | Data | #iterations | Batch Size | URL |
|
||||
| ---------- | --------------------- | ----------- | ---------- | --------------------------------------------------------------------------------------------- |
|
||||
| dynamic | 10M videos + 2M images | 100 | dynamic | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v2-stage2) |
|
||||
| dynamic | 20K HQ | 4k | dynamic | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v2-stage3) |
|
||||
|
||||
| Resolution | Data | #iterations | Batch Size | URL |
|
||||
| ------------------ | -------------------------- | ----------- | ------------------------------------------------- | -------------------------------------------------------------------- |
|
||||
| mainly 144p & 240p | 10M videos + 2M images | 100k | [dynamic](/configs/opensora-v1-1/train/stage2.py) | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v2-stage2) |
|
||||
| 144p to 720p | 500K HQ videos + 1M images | 4k | [dynamic](/configs/opensora-v1-1/train/stage3.py) | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v2-stage3) |
|
||||
|
||||
### Open-Sora 1.0 Model Weights
|
||||
|
||||
|
|
@ -223,12 +221,12 @@ This will launch a Gradio application on your localhost. If you want to know mor
|
|||
Since Open-Sora 1.1 supports inference with dynamic input size, you can pass the input size as an argument.
|
||||
|
||||
```bash
|
||||
# video sampling
|
||||
# text to video
|
||||
python scripts/inference.py configs/opensora-v1-1/inference/sample.py \
|
||||
--ckpt-path CKPT_PATH --prompt "A beautiful sunset over the city" --num-frames 32 --image-size 480 854
|
||||
```
|
||||
|
||||
See [here](docs/commands.md#inference-with-open-sora-11) for more instructions.
|
||||
See [here](docs/commands.md#inference-with-open-sora-11) for more instructions including text-to-image, image-to-video, video-to-video, and infinite time generation.
|
||||
|
||||
### Open-Sora 1.0 Command Line Inference
|
||||
|
||||
|
|
|
|||
|
|
@ -106,7 +106,7 @@ To summarize, the training of Open-Sora 1.1 requires approximately **9 days** on
|
|||
|
||||
As we get one step closer to the replication of Sora, we find many limitations for the current model, and these limitations point to the future work.
|
||||
|
||||
- **Generation Failure**: we fine many cases (especially when the total token number is large or the content is complex), our model fails to generate the scene. There may be a collapse in the temporal attention and we have identified a potential bug in our code. We are working hard to fix it.
|
||||
- **Generation Failure**: we fine many cases (especially when the total token number is large or the content is complex), our model fails to generate the scene. There may be a collapse in the temporal attention and we have identified a potential bug in our code. We are working hard to fix it. Besides, we will increase our model size and training data to improve the generation quality in the next version.
|
||||
- **Noisy generation and influency**: we find the generated model is sometimes noisy and not fluent, especially for long videos. We think the problem is due to not using a temporal VAE. As [Pixart-Sigma](https://arxiv.org/abs/2403.04692) finds that adapting to a new VAE is simple, we plan to develop a temporal VAE for the model in the next version.
|
||||
- **Lack of time consistency**: we find the model cannot generate videos with high time consistency. We think the problem is due to the lack of training FLOPs. We plan to collect more data and continue training the model to improve the time consistency.
|
||||
- **Bad human generation**: We find the model cannot generate high-quality human videos. We think the problem is due to the lack of human data. We plan to collect more human data and continue training the model to improve the human generation.
|
||||
|
|
|
|||
Loading…
Reference in a new issue