diff --git a/eval/README.md b/eval/README.md index 7ae6e32..652d3a3 100644 --- a/eval/README.md +++ b/eval/README.md @@ -48,8 +48,14 @@ First, generate the relevant videos with the following commands: ```bash # vbench task, if evaluation all set start_index to 0, end_index to 2000 bash eval/sample.sh /path/to/ckpt num_frames model_name_for_log -4 start_index end_index + # Alternatively, launch 8 jobs at once (you must read the script to understand the details) bash eval/vbench/launch.sh /path/to/ckpt num_frames model_name + +# in addition, you can specify resolution, aspect ratio, sampling steps, flow, and llm-refine +bash eval/vbench/launch.sh /path/to/ckpt num_frames model_name res_value aspect_ratio_value steps_value flow_value llm_refine_value +# for example +# bash eval/vbench/launch.sh /mnt/jfs-hdd/sora/checkpoints/outputs/042-STDiT3-XL-2/epoch1-global_step16200_llm_refine/ema.pt 51 042-STDiT3-XL-2 240p 9:16 30 2 True ``` After generation, install the VBench package following our [installation](../docs/installation.md)'s sections of "Evaluation Dependencies". Then, run the following commands to evaluate the generated samples. diff --git a/eval/vbench/launch.sh b/eval/vbench/launch.sh index f3a5b62..e7c1165 100644 --- a/eval/vbench/launch.sh +++ b/eval/vbench/launch.sh @@ -24,6 +24,8 @@ TASK_ID_LIST=(4a 4b 4c 4d 4e 4f 4g 4h) # for log records only START_INDEX_LIST=(0 120 240 360 480 600 720 840) END_INDEX_LIST=(120 240 360 480 600 720 840 2000) +## Modify the following to run on multiple machines for faster results +## 720p will take quite long on a single machine # START_INDEX_LIST=(60 180 300 420 540 660 780 900) # END_INDEX_LIST=(120 240 360 480 600 720 840 2000) # LOG_BASE=$(dirname $CKPT)/eval/last_60