update datasets.md

This commit is contained in:
Tom Young 2024-06-17 14:38:40 +00:00
parent c40f6c681b
commit 5218d5a928

View file

@ -1,16 +1,28 @@
# Datasets
For Open-Sora 1.1, we conduct mixed training with both images and videos. The main datasets we use are listed below.
For Open-Sora 1.2, we conduct mixed training with both images and videos. The main datasets we use are listed below.
Please refer to [README](/README.md#data-processing) for data processing.
## Video
### Webvid-10M
[Webvid-10M](https://github.com/m-bain/webvid) contains 10 million video-text pairs scraped from the stock footage sites.
We first train the model on this dataset (40k hours) for 30k steps (2 epochs).
### Panda-70M
[Panda-70M](https://github.com/snap-research/Panda-70M) is a large-scale dataset with 70M video-caption pairs.
We use the [training-10M subset](https://github.com/snap-research/Panda-70M/tree/main/dataset_dataloading) for training,
which contains ~10M videos of better quality.
### Mixkit
[Mixkit](https://mixkit.co/) is a video website where we obtained 9k videos.
### Pixabay
[Pixabay](https://pixabay.com/videos/) is video website where we obtained 60.5k videos.
### Pexels
[Pexels](https://www.pexels.com/) is a popular online platform that provides high-quality stock photos, videos, and music for free.
@ -28,6 +40,16 @@ The dataset is proposed for super-resolution tasks. We use the dataset for HQ fi
The caption is generated by BLIP-2.
We find the scene and the text quality are relatively poor. For OpenSora 1.0, we only use ~350K samples from this dataset.
### MiraData
[MiraData](https://github.com/mira-space/MiraData): a high-quality dataset with 77k long videos, mainly from games and city/scenic exploration.
### Vript
[Vript](https://github.com/mutonix/Vript/tree/main): a densely annotated dataset of 400k videos.
## Image
### Midjourney-v5-1.7M