Open-Sora/docs/data_processing.md

Data Processing

We establish a complete pipeline for video/image data processing. The pipeline is shown below.

First, raw videos, either from the Internet or public datasets, are split into shorter clips based on scene detection. Then, we evaluate these videos by predicting multiple scores using existing models. We first predict the aesthetic score and the optical flow score for a video. We also conduct OCR to detect texts in the video. Only videos with satisfactory evaluation results are sent to the next step for captioning. After captioning, the matching score is also calculated as an assessment of video-text alignment. Finally, we filter samples based on the matching score and conduct camera motion detection for the remaining samples. In summary, our pipeline produces video-text pairs which have high aesthetic quality, large video motion and strong semantic consistency.

For more information, please refer to:

• Dataset Management
• Scene Detection and Video Splitting
• Scoring and Filtering
• Captioning