nicolabs/Open-Sora
1
0
Fork 0
mirror of https://github.com/hpcaitech/Open-Sora.git synced 2026-05-21 11:59:01 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

add video captioning docs

Browse source
This commit is contained in:
Zangwei Zheng 2024-03-17 19:36:03 +08:00
parent 921e3138b3
commit 937f52f5bb
3 changed files with 30 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -117,7 +117,7 @@ Our model's weight is partially initialized from [PixArt-α](https://github.com/
## Inference
To run inference with our provided weights, first download [T5](https://huggingface.co/DeepFloyd/t5-v1_1-xxl/tree/main) weights into `pretrained_models/t5_ckpts/t5-v1_1-xxl`. Then run the following commands to generate samples. See [here](docs/structure.md#inference-config-demos) to customize the configuration.
To run inference with our provided weights, first download [T5](https://huggingface.co/DeepFloyd/t5-v1_1-xxl/tree/main) weights into `pretrained_models/t5_ckpts/t5-v1_1-xxl`. Then download the model weights. Run the following commands to generate samples. See [here](docs/structure.md#inference-config-demos) to customize the configuration.
```bash
# Sample 16x256x256 (5s/sample)

12
docs/report_v1.md
View file

@ -14,7 +14,7 @@ As shown in the figure, we insert a temporal attention right after each spatial
To focus on video generation, we hope to train the model based on a powerful image generation model. [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha) is an efficiently trained high-quality image generation model with T5-conditioned DiT structure. We initialize our model with PixArt-α and initialize the projection layer of inserted temporal attention with zero. This initialization preserves model's ability of image generation at beginning, while Latte's architecture cannot. The inserted attention increases the number of parameter from 580M to 724M.
[Figure]
![Architecture](https://i0.imgs.ovh/2024/03/16/erC1d.png)
Drawing from the success of PixArt-α and Stable Video Diffusion, we also adopt a progressive training strategy: 16x256x256 on 366K pretraining datasets, and then 16x256x256, 16x512x512, and 64x512x512 on 20K datasets. With scaled position embedding, this strategy greatly reduces the computational cost.
@ -36,16 +36,12 @@ With a limited training budgets, we made only a few exploration. We find learnin
16x256x256 Pretraining Loss Curve
[Figure]
![16x256x256 Pretraining Loss Curve](https://i0.imgs.ovh/2024/03/16/erXQj.png)
16x256x256 HQ Training Loss Curve
[Figure]
![16x256x256 HQ Training Loss Curve](https://i0.imgs.ovh/2024/03/16/ernXv.png)
16x512x512 HQ Training Loss Curve
[Figure]
64x512x512 HQ Training Loss Curve
[Figure]
![16x512x512 HQ Training Loss Curve](https://i0.imgs.ovh/2024/03/16/erHBe.png)

25
tools/caption/README.md
View file

@ -0,0 +1,25 @@
# Video Captioning
Human labeling of videos is expensive and time-consuming. We adopt powerful image captioning models to generate captions for videos. Although GPT-4V achieves a better performance, its 20s/sample speed is too slow for us. With batch inference, we can achieve a speed of 3s/sample with LLaVA, and the quality is comparable. LLaVA is the second best open-source model in [MMMU](https://mmmu-benchmark.github.io/) and accepts any resolution.
![Caption](https://i0.imgs.ovh/2024/03/16/eXdvC.png)
## GPT-4V Captioning
Run the following command to generate captions for videos with GPT-4V:
```bash
python -m tools.caption.caption_gpt4 FOLDER_WITH_VIDEOS output.csv --key $OPENAI_API_KEY
```
The cost is approximately $0.01 per video (3 frames per video). The output is a CSV file with path and caption.
## LLaVA Captioning
First, install LLaVA according to their [official instructions](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#install). We use the `liuhaotian/llava-v1.6-34b` model for captioning, which can be download [here](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b). Then, run the following command to generate captions for videos with LLaVA:
```bash
CUDA_VISIBLE_DEVICES=0,1 python -m tools.caption.caption_llava samples output.csv
```
The Yi-34B requires 2 80GB GPUs and 3s/sample. The output is a CSV file with path and caption.