update eval api and docs

README.md

@@ -119,7 +119,16 @@ More samples are available in our [gallery](https://hpcaitech.github.io/Open-Sor
 * [Training](#training)
 * [Contribution](#contribution)
 * [Acknowledgement](#acknowledgement)
-* [Citation](#citation)
+
+Other useful documents and links are listed below.
+
+* Report: [1.0](docs/report_01.md), [1.1](docs/report_02.md), [acceleration.md](docs/acceleration.md)
+* Repo structure: [structure.md](docs/structure.md)
+* Config file explanation: [config.md](docs/config.md)
+* Useful commands: [commands.md](docs/commands.md)
+* Data processing pipeline and dataset: [datasets.md](docs/datasets.md)
+* Each data processing tool's README: [dataset conventions and management](/tools/datasets/README.md), [scene cutting](/tools/scene_cut/README.md), [scoring](/tools/scoring/README.md), [caption](/tools/caption/README.md)
+* Gallery: [gallery](https://hpcaitech.github.io/Open-Sora/)
 
 ## Installation
 
@@ -267,8 +276,14 @@ python -m tools.datasets.csvutil ~/dataset_ready.csv --fmin 48
 
 ## Training
 
-To launch training, first download [T5](https://huggingface.co/DeepFloyd/t5-v1_1-xxl/tree/main) weights
+### Open-Sora 1.1 Training
-into `pretrained_models/t5_ckpts/t5-v1_1-xxl`. Then run the following commands to launch training on a single node.
+
+### Open-Sora 1.0 Training
+
+<details>
+<summary>View more</summary>
+
+Once you prepare the data in a `csv` file, run the following commands to launch training on a single node.
 
 ```bash
 # 1 GPU, 16x256x256
@@ -287,6 +302,8 @@ colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py configs/o
 
 For training other models and advanced usage, see [here](docs/commands.md) for more instructions.
 
+</details>
+
 ## Contribution
 
 Thanks goes to these wonderful contributors ([emoji key](https://allcontributors.org/docs/en/emoji-key)

configs/opensora-v1-1/inference/sample-ref.py

@@ -8,11 +8,11 @@ multi_resolution = "STDiT2"
 prompt_path = None
 prompt = [
     "A car driving on the ocean.",
-    'Drone view of waves crashing against the rugged cliffs along Big Sur\'s garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff\'s edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff\'s edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.{"reference_path": "assets/images/condition/cliff.png", "mask_strategy": "0,0,0,1,0"}',
+    'Drone view of waves crashing against the rugged cliffs along Big Sur\'s garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff\'s edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff\'s edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.{"reference_path": "assets/images/condition/cliff.png", "mask_strategy": "0"}',
     "In an ornate, historical hall, a massive tidal wave peaks and begins to crash. Two surfers, seizing the moment, skillfully navigate the face of the wave.",
 ]
 
-loop = 1
+loop = 2
 condition_frame_length = 4
 reference_path = [
     "https://cdn.openai.com/tmp/s/interp/d0.mp4",
@@ -22,9 +22,9 @@ reference_path = [
 # valid when reference_path is not None
 # (loop id, ref id, ref start, length, target start)
 mask_strategy = [
-    "0,0,0,8,0,0.3",
+    "0,0,0,0,8,0.3",
     None,
-    "0,0,0,1,0",
+    "0",
 ]
 
 # Define model

docs/report_01.md

@@ -10,11 +10,11 @@ The video training involves a large amount of tokens. Considering 24fps 1min vid
 
 As shown in the figure, we insert a temporal attention right after each spatial attention in STDiT (ST stands for spatial-temporal). This is similar to variant 3 in Latte's paper. However, we do not control a similar number of parameters for these variants. While Latte's paper claims their variant is better than variant 3, our experiments on 16x256x256 videos show that with same number of iterations, the performance ranks as: DiT (full) > STDiT (Sequential) > STDiT (Parallel) ≈ Latte. Thus, we choose STDiT (Sequential) out of efficiency. Speed benchmark is provided [here](/docs/acceleration.md#efficient-stdit).
 
-![Architecture Comparison](https://i0.imgs.ovh/2024/03/15/eLk9D.png)
+![Architecture Comparison](/assets/readme/report_arch_comp.png)
 
 To focus on video generation, we hope to train the model based on a powerful image generation model. [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha) is an efficiently trained high-quality image generation model with T5-conditioned DiT structure. We initialize our model with PixArt-α and initialize the projection layer of inserted temporal attention with zero. This initialization preserves model's ability of image generation at beginning, while Latte's architecture cannot. The inserted attention increases the number of parameter from 580M to 724M.
 
-![Architecture](https://image.jiqizhixin.com/uploads/editor/ff49eaba-6b19-43d7-b65d-ad2ecdb9d555/640.jpeg)
+![Architecture](/assets/readme/report_arch.jpg)
 
 Drawing from the success of PixArt-α and Stable Video Diffusion, we also adopt a progressive training strategy: 16x256x256 on 366K pretraining datasets, and then 16x256x256, 16x512x512, and 64x512x512 on 20K datasets. With scaled position embedding, this strategy greatly reduces the computational cost.
 
@@ -24,7 +24,7 @@ We also try to use a 3D patch embedder in DiT. However, with 2x downsampling on
 
 We find that the number and quality of data have a great impact on the quality of generated videos, even larger than the model architecture and training strategy. At this time, we only prepared the first split (366K video clips) from [HD-VG-130M](https://github.com/daooshee/HD-VG-130M). The quality of these videos varies greatly, and the captions are not that accurate. Thus, we further collect 20k relatively high quality videos from [Pexels](https://www.pexels.com/), which provides free license videos. We label the video with LLaVA, an image captioning model, with three frames and a designed prompt. With designed prompt, LLaVA can generate good quality of captions.
 
-![Caption](https://i0.imgs.ovh/2024/03/16/eXdvC.png)
+![Caption](/assets/readme/report_caption.png)
 
 As we lay more emphasis on the quality of data, we prepare to collect more data and build a video preprocessing pipeline in our next version.
 
@@ -36,14 +36,14 @@ With a limited training budgets, we made only a few exploration. We find learnin
 
 16x256x256 Pretraining Loss Curve
 
-![16x256x256 Pretraining Loss Curve](https://i0.imgs.ovh/2024/03/16/erXQj.png)
+![16x256x256 Pretraining Loss Curve](/assets/readme/report_loss_curve_1.png)
 
 16x256x256 HQ Training Loss Curve
 
-![16x256x256 HQ Training Loss Curve](https://i0.imgs.ovh/2024/03/16/ernXv.png)
+![16x256x256 HQ Training Loss Curve](/assets/readme/report_loss_curve_2.png)
 
 16x512x512 HQ Training Loss Curve
 
-![16x512x512 HQ Training Loss Curve](https://i0.imgs.ovh/2024/03/16/erHBe.png)
+![16x512x512 HQ Training Loss Curve](/assets/readme/report_loss_curve_3.png)
 
 > Core Contributor: Zangwei Zheng*, Xiangyu Peng*, Shenggui Li, Hongxing Liu, Yang You

docs/report_02.md

@@ -1,7 +1,20 @@
 # Open-Sora 1.1 Report
 
-> Core Contributor:
+- [Model Architecture Modification](#model-architecture-modification)
->
+- [Support for Multi-time/resolution/aspect ratio/fps Training](#support-for-multi-timeresolutionaspect-ratiofps-training)
-> * **Algorithm & Acceleration**: Zangwei Zheng, Xiangyu Peng, Shenggui Li, Hongxing Liu, Yukun Zhou
+- [Masked DiT as Image/Video-to-Video Model](#masked-dit-as-imagevideo-to-video-model)
-> * **Data Collection & Pipeline**: Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Young, Junjie Wang, Chenfeng Yu
+- [Data Collection \& Pipeline](#data-collection--pipeline)
-> * **System Maintenance**: Wenjun Li, Shenggui Li
+- [Training Details](#training-details)
+
+## Model Architecture Modification
+
+## Support for Multi-time/resolution/aspect ratio/fps Training
+
+## Masked DiT as Image/Video-to-Video Model
+
+## Data Collection & Pipeline
+
+## Training Details
+
+> **Algorithm & Acceleration**: Zangwei Zheng, Xiangyu Peng, Shenggui Li, Hongxing Liu, Yukun Zhou
+> **Data Collection & Pipeline**: Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Young, Junjie Wang, Chenfeng Yu

eval/sample.sh

@@ -91,7 +91,7 @@ function run_video_c() { # 30min
 
 function run_video_d() { # 30min
   # 2.4 16x480x854
-  eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 16 --image-size 480 854 --sample-name short_16x480x854
+  eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 16 --image-size 480 854 --sample-name sora_16x480x854
 }
 
 function run_video_e() { # 30min
@@ -111,14 +111,14 @@ function run_video_edit() { # 23min
     --num-frames 16 --image-size 240 426 \
     --loop 5 --condition-frame-length 4 \
     --reference-path assets/images/condition/cliff.png assets/images/condition/wave.png assets/images/condition/ship.png \
-    --mask-strategy "0,0,0,1,0" "0,0,0,1,0" "0,0,0,1,0"
+    --mask-strategy "0" "0" "0"
 
   eval $CMD_REF --ckpt-path $CKPT --save-dir $OUTPUT --sample-name ref_L10C4_64x240x426 \
     --prompt-path assets/texts/t2v_ref.txt --start-index 0 --end-index 3 \
     --num-frames 64 --image-size 240 426 \
     --loop 5 --condition-frame-length 16 \
     --reference-path assets/images/condition/cliff.png assets/images/condition/wave.png assets/images/condition/ship.png \
-    --mask-strategy "0,0,0,1,0" "0,0,0,1,0" "0,0,0,1,0"
+    --mask-strategy "0" "0" "0"
 
   # 3.2
   eval $CMD_REF --ckpt-path $CKPT --save-dir $OUTPUT --sample-name ref_L1_128x240x426 \
@@ -126,7 +126,7 @@ function run_video_edit() { # 23min
     --num-frames 128 --image-size 240 426 \
     --loop 1 \
     --reference-path assets/images/condition/cliff.png "assets/images/condition/cactus-sad.png\;assets/images/condition/cactus-happy.png" https://cdn.openai.com/tmp/s/interp/d0.mp4 \
-    --mask-strategy "0,0,0,1,0\;0,0,0,1,-1" "0,0,0,1,0\;0,1,0,1,-1" "0,0,0,64,0,0.5"
+    --mask-strategy "0\;0,0,0,-1,1" "0\;0,1,0,-1,1" "0,0,0,0,64,0.5"
 }
 
 # vbench has 950 samples

scripts/inference-long.py

@@ -35,6 +35,26 @@ def collect_references_batch(reference_paths, vae, image_size):
     return refs_x
 
 
+def process_mask_strategy(mask_strategy):
+    mask_batch = []
+    mask_strategy = mask_strategy.split(";")
+    for mask in mask_strategy:
+        mask_group = mask.split(",")
+        assert len(mask_group) >= 1 and len(mask_group) <= 6, f"Invalid mask strategy: {mask}"
+        if len(mask_group) == 1:
+            mask_group.extend(["0", "0", "0", "1", "0"])
+        elif len(mask_group) == 2:
+            mask_group.extend(["0", "0", "1", "0"])
+        elif len(mask_group) == 3:
+            mask_group.extend(["0", "1", "0"])
+        elif len(mask_group) == 4:
+            mask_group.extend(["1", "0"])
+        elif len(mask_group) == 5:
+            mask_group.append("0")
+        mask_batch.append(mask_group)
+    return mask_batch
+
+
 def apply_mask_strategy(z, refs_x, mask_strategys, loop_i):
     masks = []
     for i, mask_strategy in enumerate(mask_strategys):
@@ -42,11 +62,9 @@ def apply_mask_strategy(z, refs_x, mask_strategys, loop_i):
         if mask_strategy is None:
             masks.append(mask)
             continue
-        mask_strategy = mask_strategy.split(";")
+        mask_strategy = process_mask_strategy(mask_strategy)
         for mst in mask_strategy:
-            mask_batch = mst.split(",")
+            loop_id, m_id, m_ref_start, m_target_start, m_length, edit_ratio = mst
-            loop_id, m_id, m_ref_start, m_length, m_target_start = mask_batch[:5]
-            edit_ratio = mask_batch[5] if len(mask_batch) == 6 else 0.0
             loop_id = int(loop_id)
             if loop_id != loop_i:
                 continue
@@ -247,7 +265,7 @@ def main():
                         mask_strategy[j] += ";"
                     mask_strategy[
                         j
-                    ] += f"{loop_i},{len(refs)-1},-{cfg.condition_frame_length},{cfg.condition_frame_length},0"
+                    ] += f"{loop_i},{len(refs)-1},-{cfg.condition_frame_length},0,{cfg.condition_frame_length}"
             masks = apply_mask_strategy(z, refs_x, mask_strategy, loop_i)
 
             # 4.6. diffusion sampling