mirror of
https://github.com/hpcaitech/Open-Sora.git
synced 2026-05-21 11:59:01 +02:00
Merge branch 'main' of https://github.com/hpcaitech/Open-Sora into dev/v1.2
This commit is contained in:
zhengzangw
commit
7aa940f20d
|
|
@ -24,7 +24,7 @@ With Open-Sora, our goal is to foster innovation, creativity, and inclusivity wi
|
|||
|
||||
## 📰 News
|
||||
|
||||
- **[2024.06.17]** 🔥 We released **Open-Sora 1.2**, which includes **3D-VAE**, **rectified flow**, and **score condition**. The video quality is greatly improved. [[checkpoints]](#open-sora-10-model-weights) [[report]](/docs/report_03.md)
|
||||
- **[2024.06.17]** 🔥 We released **Open-Sora 1.2**, which includes **3D-VAE**, **rectified flow**, and **score condition**. The video quality is greatly improved. [[checkpoints]](#open-sora-10-model-weights) [[report]](/docs/report_03.md) [[blog]](https://hpc-ai.com/blog/open-sora-from-hpc-ai-tech-team-continues-open-source-generate-any-16-second-720p-hd-video-with-one-click-model-weights-ready-to-use)
|
||||
- **[2024.04.25]** 🤗 We released the [Gradio demo for Open-Sora](https://huggingface.co/spaces/hpcai-tech/open-sora) on Hugging Face Spaces.
|
||||
- **[2024.04.25]** We released **Open-Sora 1.1**, which supports **2s~15s, 144p to 720p, any aspect ratio** text-to-image, **text-to-video, image-to-video, video-to-video, infinite time** generation. In addition, a full video processing pipeline is released. [[checkpoints]]() [[report]](/docs/report_02.md)
|
||||
- **[2024.03.18]** We released **Open-Sora 1.0**, a fully open-source project for video generation.
|
||||
|
|
@ -297,7 +297,7 @@ In the Gradio application, the basic options are as follows:
|
|||
|
||||
The easiest way to generate a video is to input a text prompt and click the "**Generate video**" button (scroll down if you cannot find). The generated video will be displayed in the right panel. Checking the "**Enhance prompt with GPT4o**" will use GPT-4o to refine the prompt, while "**Random Prompt**" button will generate a random prompt by GPT-4o for you. Due to the OpenAI's API limit, the prompt refinement result has some randomness.
|
||||
|
||||
Then, you can choose the **resolution**, **duration**, and **aspect ratio** of the generated video. Different resolution and video length will affect the video generation speed. On a 80G H100 GPU, the generation speed and peak memory usage is:
|
||||
Then, you can choose the **resolution**, **duration**, and **aspect ratio** of the generated video. Different resolution and video length will affect the video generation speed. On a 80G H100 GPU, the generation speed (with `num_sampling_step=30`) and peak memory usage is:
|
||||
|
||||
| | Image | 2s | 4s | 8s | 16s |
|
||||
| ---- | ------- | -------- | --------- | --------- | --------- |
|
||||
|
|
|
|||
|
|
@ -11,11 +11,11 @@ OpenAI的Sora在生成一分钟高质量视频方面非常出色。然而，它
|
|||
如图中所示，在STDiT（ST代表时空）中，我们在每个空间注意力之后立即插入一个时间注意力。这类似于Latte论文中的变种3。然而，我们并没有控制这些变体的相似数量的参数。虽然Latte的论文声称他们的变体比变种3更好，但我们在16x256x256视频上的实验表明，相同数量的迭代次数下，性能排名为：DiT（完整）> STDiT（顺序）> STDiT（并行）≈ Latte。因此，我们出于效率考虑选择了STDiT（顺序）。[这里](/docs/acceleration.md#efficient-stdit)提供了速度基准测试。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
为了专注于视频生成，我们希望基于一个强大的图像生成模型来训练我们的模型。PixArt-α是一个经过高效训练的高质量图像生成模型，具有T5条件化的DiT结构。我们使用[PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha)初始化我们的模型，并将插入的时间注意力的投影层初始化为零。这种初始化在开始时保留了模型的图像生成能力，而Latte的架构则不能。插入的注意力将参数数量从5.8亿增加到7.24亿。
|
||||
|
||||
|
||||
|
||||
|
||||
借鉴PixArt-α和Stable Video Diffusion的成功，我们还采用了渐进式训练策略：在366K预训练数据集上进行16x256x256的训练，然后在20K数据集上进行16x256x256、16x512x512和64x512x512的训练。通过扩展位置嵌入，这一策略极大地降低了计算成本。
|
||||
|
||||
|
|
@ -26,7 +26,7 @@ OpenAI的Sora在生成一分钟高质量视频方面非常出色。然而，它
|
|||
|
||||
我们发现数据的数量和质量对生成视频的质量有很大的影响，甚至比模型架构和训练策略的影响还要大。目前，我们只从[HD-VG-130M](https://github.com/daooshee/HD-VG-130M)准备了第一批分割（366K个视频片段）。这些视频的质量参差不齐，而且字幕也不够准确。因此，我们进一步从提供免费许可视频的[Pexels](https://www.pexels.com/)收集了20k相对高质量的视频。我们使用LLaVA，一个图像字幕模型，通过三个帧和一个设计好的提示来标记视频。有了设计好的提示，LLaVA能够生成高质量的字幕。
|
||||
|
||||
|
||||
|
||||
|
||||
由于我们更加注重数据质量，我们准备收集更多数据，并在下一版本中构建一个视频预处理流程。
|
||||
|
||||
|
|
@ -38,12 +38,12 @@ OpenAI的Sora在生成一分钟高质量视频方面非常出色。然而，它
|
|||
|
||||
16x256x256 预训练损失曲线
|
||||
|
||||
|
||||
|
||||
|
||||
16x256x256 高质量训练损失曲线
|
||||
|
||||
|
||||
|
||||
|
||||
16x512x512 高质量训练损失曲线
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -100,7 +100,7 @@ def build_models(model_type, config, enable_optimization=False):
|
|||
# handle model download logic in HuggingFace Space
|
||||
from opensora.models.stdit.stdit3 import STDiT3
|
||||
|
||||
model_kwargs = {k: v for k, v in config.model.items() if k not in ("type", "from_pretrained")}
|
||||
model_kwargs = {k: v for k, v in config.model.items() if k not in ("type", "from_pretrained", "force_huggingface")}
|
||||
stdit = STDiT3.from_pretrained(HF_STDIT_MAP[model_type], **model_kwargs)
|
||||
stdit = stdit.cuda()
|
||||
|
||||
|
|
|
|||
|
|
@ -15,6 +15,11 @@ def timestep_transform(
|
|||
scale=1.0,
|
||||
num_timesteps=1,
|
||||
):
|
||||
# Force fp16 input to fp32 to avoid nan output
|
||||
for key in ["height", "width", "num_frames"]:
|
||||
if model_kwargs[key].dtype == torch.float16:
|
||||
model_kwargs[key] = model_kwargs[key].float()
|
||||
|
||||
t = t / num_timesteps
|
||||
resolution = model_kwargs["height"] * model_kwargs["width"]
|
||||
ratio_space = (resolution / base_resolution).sqrt()
|
||||
|
|
|
|||
Loading…
Reference in a new issue