nicolabs/Open-Sora
1
0
Fork 0
mirror of https://github.com/hpcaitech/Open-Sora.git synced 2026-04-10 21:01:26 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Dev/pxy (#72)

Browse source 
* update scoring/matching

* update scoring/matching

* update scoring/matching

* update scoring/matching

* update scoring/matching

* update scoring/matching

* update scoring/matching

* update scoring/matching

* update scoring/matching

* update scene_cut

* update scene_cut

* update scene_cut[A

* update scene_cut

* update scene_cut

* update scene_cut

* update scene_cut

* update scene_cut

* update scene_cut

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* update readme

* update readme

* extract frames using opencv everywhere

* extract frames using opencv everywhere

* extract frames using opencv everywhere

* filter panda10m

* filter panda10m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* m

* ocr

* add ocr

* add main.sh

* add ocr

* add ocr

* add ocr

* add ocr

* add ocr

* add ocr

* update scene_cut

* update remove main.sh

* update scoring

* update scoring

* update scoring

* update README

* update readme

* update scene_cut

* update readme

* update scoring

* update readme

* update readme

* update filter_panda10m

* update readme

* update readme

* update launch.ipynb

* update scene_cut

* update scene_cut

* update readme

* update launch.ipynb
This commit is contained in:
xyupeng 2024-04-24 15:52:07 +08:00 committed by GitHub
parent d110689f29
commit 2219fc8703
7 changed files with 113 additions and 630 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

36
README.md
View file

@ -247,39 +247,11 @@ To lower the memory usage, set a smaller `vae.micro_batch_size` in the config (s
</details>
## Data Processing
High-quality data is crucial for training good generation models.
To this end, we establish a complete pipeline for data processing, which could seamlessly convert raw videos to high-quality video-text pairs.
The pipeline is shown below. For detailed information, please refer to [data processing](docs/data_processing.md).
Also check out the [datasets](docs/data_processing.md) we use.
Te be modified
High-quality Data is the key to high-quality models. Our used datasets and data collection plan
is [here](/docs/datasets.md). We provide tools to process video data. Our data processing pipeline includes
the following steps:
1. Manage datasets. [[docs](/tools/datasets/README.md)]
2. Scene detection and video splitting. [[docs](/tools/scene_cut/README.md)]
3. Score and filter videos. [[docs](/tools/scoring/README.md)]
4. Generate video captions. [[docs](/tools/caption/README.md)]
Below is an example workflow to process data. However, we recommend you to read the detailed documentation for each tool, and decide which tools to use based on your needs. This pipeline applies to both image and video data. Full pipeline is available in [datasets.md](/tools/datasets/README.md#data-process-pipeline).
```bash
# Suppose videos and images under ~/dataset/
# 1. Convert dataset to CSV (meta.csv)
python -m tools.datasets.convert video ~/dataset --output meta.csv
# 2. Get video information (meta_info_fmin1.csv)
python -m tools.datasets.datautil meta.csv --info --fmin 1
# 3. Get caption information
torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava meta_info_fmin1.csv --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video
# merge generated results (meta_caption.csv)
python -m tools.datasets.datautil meta_info_fmin1_caption_part*.csv --output meta_caption.csv
# clean caption (meta_caption_processed.csv)
python -m tools.datasets.datautil meta_caption.csv --clean-caption --refine-llm-caption --remove-empty-caption --output meta_caption_processed.csv
# 4. Scoring (meta_caption_processed_aes.csv)
torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference meta_caption_processed.csv --bs 1024 --num_workers 16
# Filter videos by aesthetic scores (meta_aes_aesmin5.csv)
python -m tools.datasets.csvutil meta_caption_processed_aes.csv --aesmin 5 --output meta_aes_aesmin5.csv
# 5. Additional filtering
python -m tools.datasets.csvutil ~/dataset_ready.csv --fmin 48
```
![Data Processing Pipeline](assets/readme/report_data_pipeline.png)

65
docs/data_processing.md
View file

@ -13,8 +13,65 @@ conduct camera motion detection for the remaining samples.
In summary, our pipeline produces video-text pairs which have high aesthetic quality, large video motion and strong
semantic consistency.
Below is an example workflow to process videos.
```bash
ROOT_VIDEO="/path/to/video/folder"
ROOT_CLIPS="/path/to/video/clips/folder"
ROOT_META="/path/to/meta/folder"
# 1.1 Create a meta file from a video folder. This should output ${ROOT_META}/meta.csv
python -m tools.datasets.convert video ${ROOT_VIDEO} --output ${ROOT_META}/meta.csv
# 1.2 Get video information and remove broken videos. This should output ${ROOT_META}/meta_info_fmin1.csv
python -m tools.datasets.datautil ${ROOT_META}/meta.csv --info --fmin 1
# 2.1 Detect scenes. This should output ${ROOT_META}/meta_info_fmin1_timestamp.csv
python -m tools.scene_cut.scene_detect ${ROOT_META}/meta_info_fmin1.csv
# 2.2 Cut video into clips based on scenes. This should produce video clips under ${ROOT_CLIPS}
python -m tools.scene_cut.cut ${ROOT_META}/meta_info_fmin1_timestamp.csv --save_dir ${ROOT_CLIPS}
# 2.3 Create a meta file for video clips. This should output ${ROOT_META}/meta_clips.csv
python -m tools.datasets.convert video ${ROOT_CLIPS} --output ${ROOT_META}/meta_clips.csv
# 2.4 Get clips information and remove broken ones. This should output ${ROOT_META}/meta_clips_info_fmin1.csv
python -m tools.datasets.datautil ${ROOT_META}/meta_clips.csv --info --fmin 1
# 3.1 Predict aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_part*.csv
torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference \
${ROOT_META}/meta_clips_info_fmin1.csv \
--bs 1024 \
--num_workers 16
# 3.2 Merge files; This should output ${ROOT_META}/meta_clips_info_fmin1_aes.csv
python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes_part*.csv --output ${ROOT_META}/meta_clips_info_fmin1_aes.csv
# 3.2 Filter by aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv
python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes.csv --aesmin 5
# 4.1 Generate caption. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv
torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava \
${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv \
--dp-size 8 \
--tp-size 1 \
--model-path /path/to/llava-v1.6-mistral-7b \
--prompt video
# 4.2 Merge caption results. This should output ${ROOT_META}/meta_clips_caption.csv
python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv --output ${ROOT_META}/meta_clips_caption.csv
# 4.3 Clean caption. This should output ${ROOT_META}/meta_clips_caption_cleaned.csv
python -m tools.datasets.datautil \
${ROOT_META}/meta_clips_caption.csv \
--clean-caption \
--refine-llm-caption \
--remove-empty-caption \
--output ${ROOT_META}/meta_clips_caption_cleaned.csv
```
For more information, please refer to:
- [Dataset Management](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/datasets/README.md)
- [Scene Detection and Video Splitting](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/scene_cut/README.md)
- [Scoring and Filtering](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/scoring/README.md)
- [Captioning](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/caption/README.md)
- [Dataset Management](../tools/datasets/README.md)
- [Scene Detection and Video Splitting](../tools/scene_cut/README.md)
- [Scoring and Filtering](../tools/scoring/README.md)
- [Captioning](../tools/caption/README.md)

620
notebooks/launch.ipynb
View file

@ -16,13 +16,14 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"OPEN_SORA_HOME = \"/home/zhaowangbo/zangwei/opensora/\"\n",
"# TODO: change to your own project path!!!\n",
"OPEN_SORA_HOME = \"/path/to/Open-Sora/\"\n",
"\n",
"\n",
"def convert_dataset_cmd(input_dir, output_file, datatype=\"video\"):\n",
@ -123,6 +124,17 @@
" return \" && \".join(commands), output_file\n",
"\n",
"\n",
"def get_ocr(input_file):\n",
" commands = []\n",
" base, ext = os.path.splitext(input_file)\n",
" output_file = f\"{base}_match{ext}\"\n",
"\n",
" commands.append(f'echo \"Getting match score of {input_file} to {output_file}\"')\n",
" commands.append(f\"cd {OPEN_SORA_HOME}\")\n",
" commands.append(f\"torchrun --standalone --nproc_per_node 8 -m tools.scoring.ocr.inference {input_file}\")\n",
" return \" && \".join(commands), output_file\n",
"\n",
" \n",
"def get_match_score(input_file):\n",
" commands = []\n",
" base, ext = os.path.splitext(input_file)\n",
@ -167,18 +179,18 @@
"source": [
"### Remote Launch via Paramiko\n",
"\n",
"First, you should add hosts in your ~/.ssh/config file"
"First, add hosts to `~/.ssh/config`"
]
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import paramiko\n",
"\n",
"HOSTS = [\"h800-80\", \"h800-81\", \"h800-82\", \"h800-83\", \"h800-84\", \"h800-85\", \"h800-86\", \"h800-170\", \"h800-171\"]\n",
"HOSTS = [\"host-0\", \"host-1\", \"host-2\", \"host-3\", \"host-4\", \"host-5\", \"host-6\", \"host-7\"]\n",
"\n",
"# load from ~/.ssh/config\n",
"ssh_config = paramiko.SSHConfig()\n",
@ -246,7 +258,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -259,9 +271,9 @@
"\n",
"def nvitop(host=None):\n",
" if host:\n",
" run_command(f\"/home/zhaowangbo/.local/bin/nvitop -1\", host)\n",
" run_command(f\"/home/user/.local/bin/nvitop -1\", host)\n",
" else:\n",
" run_command_all_hosts(\"/home/zhaowangbo/.local/bin/nvitop -1\")\n",
" run_command_all_hosts(\"/home/user/.local/bin/nvitop -1\")\n",
"\n",
"\n",
"def ps(host=None, interest=\"python|sleep|torchrun|colossal\", all=True):\n",
@ -290,389 +302,22 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Examples\n",
"### Example\n",
"\n",
"The following is the pipeline for panda."
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"echo \"Getting aesthetic score of /mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info.csv to /mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info_aes.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && torchrun --standalone --nproc_per_node 8 -m tools.scoring.aesthetic.inference /mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info.csv && python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info_part*.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info_aes.csv --format csv --sort aes && echo \"All Done!\"\n",
"/mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info_aes.csv\n"
]
}
],
"source": [
"# panda\n",
"host = \"h800-83\"\n",
"split = 16\n",
"input_dir = f\"/mnt/disk1/data-panda/{split}\"\n",
"log_file = os.path.join(OPEN_SORA_HOME, f\"logs/data-panda-{split}-split.log\")\n",
"output_file = f\"/mnt/hdd/data/panda70m_by/raw/meta/split-{split}/meta.csv\"\n",
"cmd, output_file = get_commands(\n",
" [\n",
" # {\n",
" # \"cmd\": convert_dataset_cmd,\n",
" # \"input_dir\": input_dir,\n",
" # \"output_file\": output_file,\n",
" # },\n",
" # {\n",
" # \"cmd\": get_caption_load,\n",
" # },\n",
" # {\n",
" # \"cmd\": get_video_info_torchvision,\n",
" # },\n",
" # {\n",
" # \"cmd\": get_aesthetic_score,\n",
" # },\n",
" # {\n",
" # \"cmd\": get_flow_score,\n",
" # },\n",
" # {\n",
" # \"cmd\": get_match_score,\n",
" # },\n",
" # {\n",
" # \"cmd\": get_cmotion_score,\n",
" # },\n",
" ]\n",
")\n",
"print(cmd)\n",
"print(output_file)"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"echo \"Getting info of /mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean.csv to /mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1 && echo \"All Done!\"\n",
"/mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean_info.csv\n"
]
}
],
"source": [
"# panda\n",
"host = \"h800-82\"\n",
"split = 7\n",
"log_file = os.path.join(OPEN_SORA_HOME, f\"logs/data-panda-{split}-split.log\")\n",
"cmd, output_file = get_commands(\n",
" [\n",
" {\n",
" \"cmd\": get_video_info_torchvision,\n",
" \"input_file\": f\"/mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean.csv\",\n",
" },\n",
" ]\n",
")\n",
"print(cmd)\n",
"print(output_file)"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"echo \"Getting info of /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted.csv to /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && conda activate llava2 && torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted.csv --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video && conda activate opensora && python -m tools.datasets.datautil /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted_caption_part*.csv --output /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted_info.csv --format csv --intersection /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted.csv --clean-caption --refine-llm-caption --remove-empty-caption && echo \"All Done!\"\n",
"/home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted_info.csv\n"
]
}
],
"source": [
"# v2text\n",
"host = \"h800-86\"\n",
"log_file = os.path.join(OPEN_SORA_HOME, f\"logs/data-v2text-18.log\")\n",
"input_file = \"/home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted.csv\"\n",
"cmd, output_file = get_commands(\n",
" [\n",
" {\n",
" \"cmd\": get_caption_llava7b_video,\n",
" \"input_file\": input_file,\n",
" },\n",
" ]\n",
")\n",
"print(cmd)\n",
"print(output_file)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Remote launch via paramiko."
]
},
{
"cell_type": "code",
"execution_count": 67,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HOST: h800-80\n",
"COMMAND: nohup bash -ic 'echo \"Getting info of /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv to /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && conda activate llava2 && torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video && conda activate opensora && python -m tools.datasets.datautil /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_part*.csv --output /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_info.csv --format csv --intersection /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv --clean-caption --refine-llm-caption --remove-empty-caption && echo \"All Done!\"' >> /home/zhaowangbo/zangwei/opensora/logs/data-v2text-12.log 2>&1 &\n",
"HOST: h800-80\n",
"COMMAND: bash -ic 'ps ux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" zhaowan+ 3482147 2.0 0.0 8496 5244 ? S 13:24 0:00 bash -ic echo \"Getting info of /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv to /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && conda activate llava2 && torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video && conda activate opensora && python -m tools.datasets.datautil /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_part*.csv --output /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_info.csv --format csv --intersection /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv --clean-caption --refine-llm-caption --remove-empty-caption && echo \"All Done!\"\n",
"zhaowan+ 3482641 10.0 0.0 8488 5088 ? Ss 13:24 0:00 bash -ic ps ux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"zhaowan+ 3482792 42.0 0.0 2933116 227692 ? R 13:24 0:00 /home/zhaowangbo/.conda/envs/llava2/bin/python /home/zhaowangbo/.conda/envs/llava2/bin/torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video\n",
"zhaowan+ 3482808 0.0 0.0 6412 728 ? S 13:24 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"\n"
]
}
],
"outputs": [],
"source": [
"sleep = None\n",
"run_command(cmd, host, log_file=log_file, nohup=True, sleep=sleep)\n",
"ps(host)"
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HOST: h800-84\n",
"COMMAND: bash -ic 'ps ux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" zhaowan+ 697488 0.8 0.0 21302928 982860 ? Sl 07:54 3:19 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 756910 2.3 0.0 28226540 982328 ? Sl 07:55 9:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757066 22.0 0.1 56222740 4023872 ? Il 07:55 86:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757068 21.2 0.2 62354592 4400760 ? Il 07:55 83:44 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757115 20.6 0.3 61031756 8265648 ? Il 07:55 81:23 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757132 22.6 0.2 68694032 4909496 ? Il 07:55 89:27 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757134 26.8 0.2 63051476 5462788 ? Rl 07:55 105:47 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757181 23.8 0.2 49112896 4392012 ? Il 07:55 94:00 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757183 23.2 0.2 66357972 5091892 ? Rl 07:55 91:41 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757186 22.2 0.2 73577628 5263940 ? Il 07:55 87:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757194 21.2 0.2 61154288 4409388 ? Il 07:55 83:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757220 22.5 0.2 56297716 5071064 ? Il 07:55 88:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757222 21.0 0.4 80284000 8926020 ? Rl 07:55 83:07 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757224 21.3 0.2 61690852 4616640 ? Il 07:55 84:16 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757300 21.4 0.2 63578668 4423544 ? Il 07:55 84:35 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757348 21.7 0.2 79282760 5193352 ? Il 07:55 85:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757350 26.9 0.2 60714784 4810316 ? Rl 07:55 106:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757367 22.2 0.2 70532632 5095456 ? Il 07:55 87:39 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757399 22.4 0.2 65492460 4838288 ? Il 07:55 88:23 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757401 24.0 0.2 53068672 5885536 ? Rl 07:55 94:41 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757418 21.8 0.2 61157180 4504976 ? Il 07:55 86:16 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757435 22.8 0.2 62984764 5448604 ? Il 07:55 90:09 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757437 22.0 0.2 67874668 5040264 ? Rl 07:55 86:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757439 22.7 0.2 68867624 4337728 ? Il 07:55 89:29 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757456 20.2 0.2 68795064 4935428 ? Il 07:55 79:45 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757458 21.7 0.2 61065776 4931596 ? Il 07:55 85:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757475 22.3 0.2 58261592 4732108 ? Il 07:55 88:08 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757673 20.5 0.1 56322500 4223948 ? Il 07:55 80:59 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757722 22.2 0.4 72053868 10389480 ? Il 07:55 87:41 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757737 21.2 0.2 60275360 4627408 ? Il 07:55 83:54 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757769 20.7 0.2 64496484 4712216 ? Il 07:55 81:50 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757816 19.5 0.2 49375580 5353560 ? Rl 07:55 76:57 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757848 20.8 0.2 67125764 4799684 ? Dl 07:55 82:05 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757865 20.1 0.1 68462464 3823092 ? Il 07:55 79:35 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757897 21.4 0.2 72993788 5109220 ? Il 07:55 84:22 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757914 21.0 0.2 58775368 4358980 ? Rl 07:55 82:48 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757946 27.0 0.2 65172932 4311496 ? Rl 07:55 106:46 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757963 22.8 0.1 60634860 2854188 ? Il 07:55 90:02 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 757980 27.8 0.2 70003380 4786480 ? Il 07:55 109:48 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758027 21.4 0.2 61093344 5021592 ? Dl 07:55 84:40 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758055 21.4 0.2 63173892 4465772 ? Il 07:55 84:38 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758076 27.6 0.2 63197068 4837008 ? Il 07:55 109:08 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758078 19.0 0.2 57450172 4515412 ? Il 07:55 74:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758095 22.8 0.2 63550712 5040656 ? Il 07:55 90:09 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758112 22.3 0.1 60771048 3967276 ? Il 07:55 88:11 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758129 20.9 0.2 76323044 5139840 ? Il 07:55 82:25 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758161 22.9 0.1 63293068 4105016 ? Il 07:55 90:38 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758238 20.7 0.1 58782420 3733868 ? Il 07:55 81:41 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758240 23.6 0.2 60366724 4870712 ? Il 07:55 93:15 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758287 20.4 0.2 67594208 4885716 ? Il 07:55 80:37 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758334 20.9 0.2 60524216 5200828 ? Il 07:55 82:47 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758420 21.9 0.2 61651496 4922668 ? Il 07:55 86:25 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758461 20.0 0.2 58606524 4798940 ? Il 07:55 78:58 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758490 25.6 0.2 57541972 4511752 ? Il 07:55 101:00 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758522 21.4 0.2 57452244 4556340 ? Il 07:55 84:30 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758539 21.8 0.2 64637932 4421864 ? Il 07:55 86:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758541 21.4 0.2 63791100 5004104 ? Il 07:55 84:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758543 21.1 0.2 72491116 4859764 ? Il 07:55 83:26 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758575 21.7 0.2 54628964 4677276 ? Il 07:55 85:45 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758622 20.1 0.2 54905520 4660216 ? Il 07:55 79:34 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758654 20.6 0.2 65326772 4237344 ? Il 07:55 81:24 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758671 21.3 0.2 67287820 4826480 ? Rl 07:55 84:01 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758688 23.9 0.2 66189768 5231632 ? Rl 07:55 94:17 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758735 23.6 0.1 59340972 3407908 ? Il 07:55 93:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758767 21.1 0.2 59026076 4922724 ? Dl 07:55 83:19 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758799 24.9 0.2 51343480 4485852 ? Il 07:55 98:15 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758831 20.7 0.2 73143608 5161436 ? Rl 07:55 81:42 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758833 21.3 0.2 49991612 4309764 ? Il 07:55 84:07 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758836 21.4 0.2 63950844 4979736 ? Il 07:55 84:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758853 24.8 0.2 58920328 4975560 ? Rl 07:55 97:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758885 22.6 0.2 66055404 5366452 ? Il 07:55 89:13 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758932 21.1 0.2 63554972 4502792 ? Il 07:55 83:23 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758949 29.8 0.1 61384460 3507884 ? Il 07:55 117:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758981 22.2 0.1 69580512 4075980 ? Il 07:55 87:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 758983 22.9 0.2 60777712 5068084 ? Rl 07:55 90:38 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759000 23.8 0.2 64954152 4806056 ? Il 07:55 93:50 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759032 26.3 0.2 58059496 4858848 ? Il 07:55 104:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759094 25.8 0.2 65760368 5394312 ? Rl 07:55 101:49 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759126 26.5 0.2 59127380 4639020 ? Il 07:55 104:40 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759128 21.3 0.2 60868916 4568500 ? Il 07:55 84:19 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759130 20.5 0.2 61144416 4353776 ? Il 07:55 81:11 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759147 22.2 0.2 59077984 4776584 ? Dl 07:55 87:40 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759164 22.1 0.2 65963804 4400904 ? Il 07:55 87:09 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759166 21.1 0.2 60764900 5435900 ? Il 07:55 83:15 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759168 25.9 0.2 62305136 5418788 ? Rl 07:55 102:11 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759200 22.1 0.2 63089284 4373652 ? Il 07:55 87:29 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759202 19.9 0.2 49378748 4509304 ? Il 07:55 78:48 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759219 21.2 0.2 73910200 4413528 ? Il 07:55 83:35 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759311 19.9 0.2 58333480 4383732 ? Il 07:55 78:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759328 20.1 0.2 51891348 4394716 ? Il 07:55 79:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759345 21.3 0.2 62480824 4778464 ? Il 07:55 84:13 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759392 21.8 0.2 66537592 5815980 ? Il 07:55 86:02 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759409 21.0 0.2 61670672 4647044 ? Rl 07:55 83:07 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759426 21.0 0.2 67450788 4598964 ? Il 07:55 83:06 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759443 24.9 0.2 60838980 4473136 ? Il 07:55 98:25 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759460 22.9 0.1 64643588 3131576 ? Il 07:55 90:30 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759462 22.5 0.2 65706064 4940932 ? Dl 07:55 88:52 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 759494 20.4 0.2 56295000 4496504 ? Il 07:55 80:43 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 1262361 0.0 0.0 14180 5296 ? S 01:54 0:00 bash -ic sleep 3h; echo \"Getting info of /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv to /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1 && echo \"All Done!\"\n",
"zhaowan+ 1354819 0.0 0.0 14180 5232 ? S 01:54 0:00 bash -ic sleep 6h; echo \"Getting info of /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv to /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1 && echo \"All Done!\"\n",
"zhaowan+ 1984585 0.8 0.0 21234260 914656 ? Sl 04:54 4:44 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2064791 2.5 0.0 28001092 915132 ? Sl 04:55 14:24 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2064871 18.9 0.1 23901996 3583608 ? Il 04:55 108:52 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2064873 20.1 0.2 24702060 4261168 ? Il 04:55 115:25 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2064965 20.1 0.1 24774688 4017516 ? Sl 04:55 115:42 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065099 20.2 0.1 24040840 3599188 ? Il 04:55 116:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065101 19.3 0.1 23808764 3612580 ? Dl 04:55 111:09 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065118 19.0 0.1 23696028 3377124 ? Il 04:55 109:05 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065135 19.8 0.1 23934804 3577844 ? Sl 04:55 114:08 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065152 21.2 0.1 23751700 3432956 ? Il 04:55 121:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065184 20.4 0.1 23980124 3410420 ? Sl 04:55 117:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065186 20.9 0.1 23809520 3244896 ? Il 04:55 120:01 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065203 20.1 0.1 24171548 3601216 ? Sl 04:55 115:43 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065205 21.6 0.1 23632596 3062812 ? Sl 04:55 124:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065222 20.1 0.1 23794996 3315016 ? Sl 04:55 115:58 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065239 20.2 0.1 24125720 3522592 ? Sl 04:55 116:07 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065241 19.8 0.1 24244016 3764500 ? Sl 04:55 113:46 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065243 21.5 0.1 24222260 3530048 ? Sl 04:55 123:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065245 21.1 0.1 24215696 3735876 ? Sl 04:55 121:39 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065247 21.5 0.1 23730660 3535380 ? Il 04:55 123:58 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065264 21.2 0.1 23773068 3454216 ? Il 04:55 121:45 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065266 20.9 0.3 27030228 6811788 ? Rl 04:55 120:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065283 20.7 0.1 23630864 3312760 ? Il 04:55 119:16 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065300 23.4 0.1 24095968 3621376 ? Il 04:55 134:47 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065317 20.9 0.1 23896548 3454832 ? Il 04:55 120:02 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065319 20.4 0.1 23852780 3495144 ? Sl 04:55 117:08 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065321 22.3 0.2 24970292 4325780 ? Rl 04:55 128:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065545 25.8 0.1 24964468 4157640 ? Rl 04:55 148:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065562 22.1 0.1 23664124 3216992 ? Sl 04:55 126:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065564 23.3 0.2 25803564 5036196 ? Rl 04:55 133:46 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065566 23.3 0.1 23749396 3553960 ? Il 04:55 134:16 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065598 21.0 0.1 23867720 3516672 ? Il 04:55 120:35 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065615 21.7 0.1 24266168 3508132 ? Sl 04:55 124:50 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065627 22.0 0.1 23496084 3144832 ? Il 04:55 126:49 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065649 23.1 0.1 24012732 3694248 ? Il 04:55 132:51 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065780 20.5 0.1 24022680 3826792 ? Il 04:55 118:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065877 21.0 0.1 24020120 3507712 ? Sl 04:55 120:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065879 20.7 0.1 23247420 3051268 ? Il 04:55 119:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065881 24.8 0.1 23613316 3172020 ? Il 04:55 142:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065883 21.1 0.2 25951160 5594956 ? Sl 04:55 121:12 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065915 21.3 0.1 23865148 3295056 ? Sl 04:55 122:50 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065917 19.8 0.1 23812588 3299900 ? Sl 04:55 114:06 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065934 20.3 0.1 24185736 3616268 ? Sl 04:55 116:55 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065936 25.8 0.1 24135688 3816780 ? Il 04:55 148:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2065938 21.8 0.1 23936204 3617428 ? Il 04:55 125:40 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066168 20.6 0.1 24040208 3560608 ? Sl 04:55 118:44 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066254 21.7 0.1 24113416 3671912 ? Il 04:55 124:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066463 20.1 0.1 24137384 3567148 ? Sl 04:55 115:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066465 21.0 0.3 27174728 6728004 ? Sl 04:55 120:59 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066467 21.9 0.1 24238792 3674556 ? Il 04:55 126:17 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066469 21.7 0.2 26543524 6101716 ? Rl 04:55 124:40 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066471 20.3 0.3 27072852 6721744 ? Il 04:55 116:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066503 23.5 0.1 24265704 3700984 ? Il 04:55 135:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066535 20.4 0.1 24308408 4047644 ? Dl 04:55 117:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066537 23.8 0.1 23862900 3666564 ? Il 04:55 136:49 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066554 25.6 0.1 24714016 4075292 ? Rl 04:55 147:22 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066571 24.8 0.1 24000324 3681940 ? Il 04:55 142:25 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066573 21.4 0.2 25996736 5555128 ? Il 04:55 123:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066575 23.7 0.1 24301656 3949984 ? Il 04:55 136:22 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066577 20.1 0.3 26725368 6406652 ? Rl 04:55 115:54 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066579 24.9 0.1 23801220 3360128 ? Il 04:55 143:30 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066581 21.8 0.1 24127376 3554688 ? Il 04:55 125:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066583 19.5 0.4 30593056 10112812 ? Sl 04:55 112:16 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066585 19.6 0.1 23536080 3089880 ? Sl 04:55 113:05 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066602 19.7 0.1 24258636 3688524 ? Sl 04:55 113:24 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066604 21.4 0.1 23373520 3177488 ? Il 04:55 123:12 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066606 21.3 0.1 23965420 3273080 ? Sl 04:55 122:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066608 23.0 0.1 24060464 3741516 ? Il 04:55 132:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066625 23.1 0.1 24998568 3895104 ? Rl 04:55 132:42 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066627 22.6 0.1 24209284 3768280 ? Il 04:55 129:55 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066629 23.0 0.1 24071816 3721036 ? Rl 04:55 132:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066646 23.7 0.1 23595984 3277364 ? Il 04:55 136:21 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066648 21.9 0.3 26924956 6450372 ? Dl 04:55 125:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066650 23.3 0.1 23828300 3445160 ? Il 04:55 133:48 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066652 20.5 0.1 24249456 3561936 ? Rl 04:55 118:11 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066654 21.0 0.1 23778072 3427236 ? Il 04:55 120:37 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066656 26.0 0.1 24788272 4198708 ? Rl 04:55 149:23 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066658 20.9 0.1 23684832 3489420 ? Il 04:55 120:12 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066660 20.4 0.1 24054808 3580084 ? Il 04:55 117:17 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066662 24.6 0.2 24910136 4403304 ? Rl 04:55 141:44 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066664 20.4 0.1 24376528 4025124 ? Dl 04:55 117:21 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066666 22.4 0.1 23970000 3523596 ? Sl 04:55 128:52 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066733 19.9 0.1 23976636 3496212 ? Sl 04:55 114:45 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066877 23.6 0.2 24892000 4431224 ? Rl 04:55 136:02 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066879 20.1 0.1 23928356 3358612 ? Sl 04:55 115:27 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066911 21.0 0.1 23792432 3596812 ? Il 04:55 120:44 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066913 20.7 0.1 24435236 3628644 ? Sl 04:55 119:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066930 23.7 0.1 23690284 3371840 ? Il 04:55 136:12 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2066947 24.4 0.1 23663732 3517224 ? Il 04:55 140:38 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2067141 21.1 0.1 24832424 4074672 ? Sl 04:55 121:12 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2067143 19.4 0.1 24615160 3921920 ? Sl 04:55 111:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2067160 26.5 0.1 23807964 3456564 ? Il 04:55 152:21 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2067162 23.2 0.1 24068296 3503972 ? Il 04:55 133:17 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2067164 25.2 0.1 23698596 3346972 ? Il 04:55 144:42 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2067166 19.8 0.1 23898984 3456920 ? Il 04:55 114:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2067168 21.6 0.1 23833948 3637788 ? Il 04:55 124:05 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2067170 23.6 0.1 23952140 3511024 ? Rl 04:55 135:54 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 2067172 24.3 0.2 25158908 4555396 ? Rl 04:55 139:57 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n",
"zhaowan+ 3380048 10.0 0.0 14176 5312 ? Ss 14:29 0:00 bash -ic ps ux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"zhaowan+ 3381150 0.0 0.0 12124 2496 ? S 14:29 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"\n"
]
}
],
"source": [
"# pkill(\"split-6\", \"h800-84\")\n",
"ps(\"h800-84\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
@ -682,170 +327,9 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HOST: h800-80\n",
"COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" root 4838 0.0 0.0 29820 18308 ? Ss Apr08 0:01 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n",
"lisheng+ 1551124 0.0 0.0 3090356 185552 pts/9 Sl 11:48 0:02 /home/lishenggui/.conda/envs/opensora/bin/python /home/lishenggui/.conda/envs/opensora/bin/torchrun --master_addr 10.20.1.80 --master_port 29550 --nproc_per_node 8 --nnodes 8 --node_rank 0 /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 1565730 101 0.1 71931860 4155240 ? Ssl 11:48 377:44 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 1565747 101 0.4 71657092 8687376 ? Ssl 11:48 377:51 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 1565766 101 0.4 71851652 8879768 ? Rsl 11:48 377:36 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 1565775 101 0.4 71543296 8769772 ? Ssl 11:48 377:49 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 1565776 101 0.3 71417920 8448272 ? Ssl 11:48 377:38 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 1565777 101 0.3 71220924 8251132 ? Ssl 11:48 377:37 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 1565778 101 0.3 71397324 8431376 ? Ssl 11:48 377:47 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 1565779 101 0.4 71677852 8709120 ? Ssl 11:48 377:36 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"root 1569630 0.0 0.0 11152 580 ? S 18:01 0:00 sleep 30\n",
"zhaowan+ 1571020 4.5 0.0 8488 5184 ? Ss 18:01 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"zhaowan+ 1571252 0.0 0.0 6412 2356 ? S 18:01 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"root 2401123 0.0 0.0 193404 64564 ? S Apr10 0:00 /usr/bin/python3 /usr/local/bin/jupyter-lab --allow-root --ServerApp.base_url=lab\n",
"\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"\n",
"HOST: h800-81\n",
"COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" root 4828 0.0 0.0 29820 17552 ? Ss Mar19 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n",
"zhaowan+ 320182 0.2 0.0 1255960 460164 ? Sl 14:08 0:34 /home/zhaowangbo/.vscode-server/bin/1e790d77f81672c49be070e04474901747115651/node /home/zhaowangbo/.vscode-server/extensions/ms-python.vscode-pylance-2024.4.1/dist/server.bundle.js --cancellationReceive=file:ff42b41eb2644cbe0dab526c305f3ec2a5e3ca1694 --node-ipc --clientProcessId=294384\n",
"tom 382879 0.0 0.0 683308 47204 ? Sl Apr12 0:35 /home/zhaowangbo/.conda/envs/opensora/bin/python -m ipykernel_launcher --f=/home/tom/.local/share/jupyter/runtime/kernel-v2-362345kP0b0TEeW8vP.json\n",
"lisheng+ 672111 0.0 0.0 3016604 185384 ? S 11:48 0:01 /home/lishenggui/.conda/envs/opensora/bin/python /home/lishenggui/.conda/envs/opensora/bin/torchrun --master_addr 10.20.1.80 --master_port 29550 --nproc_per_node 8 --nnodes 8 --node_rank 1 /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 686018 101 0.3 70429568 7713964 ? Ssl 11:48 377:47 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 686019 101 0.3 70867092 7934756 ? Rsl 11:48 377:36 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 686026 101 0.3 70857736 7925428 ? Ssl 11:48 378:07 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 686037 101 0.4 71659836 8688480 ? Ssl 11:48 377:47 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 686038 101 0.4 71799792 8828624 ? Ssl 11:48 377:38 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 686039 101 0.4 71458224 8488680 ? Ssl 11:48 377:34 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 686042 101 0.4 71650732 8680932 ? Rsl 11:48 377:40 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"lisheng+ 686050 101 0.4 71747956 8778104 ? Ssl 11:48 377:39 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n",
"zhaowan+ 3388150 0.4 0.0 129732 45044 ? Sl 18:00 0:00 /usr/bin/python3 /home/zhaowangbo/.vscode-server/extensions/ms-python.black-formatter-2024.2.0/bundled/tool/lsp_server.py --stdio\n",
"zhaowan+ 3388671 5.0 0.0 8492 5064 ? Ss 18:01 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"zhaowan+ 3388767 0.0 0.0 6412 2264 ? S 18:01 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"\n",
"HOST: h800-82\n",
"COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" root 4798 0.0 0.0 30076 18704 ? Ss Mar18 0:02 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n",
"zhaowan+ 1606637 5.0 0.0 8488 5292 ? Ss 18:01 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"zhaowan+ 1606730 0.0 0.0 6412 2360 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"\n",
"HOST: h800-83\n",
"COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" root 4804 0.0 0.0 35356 14248 ? Ss Mar21 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n",
"lisheng+ 1171411 2.2 0.0 1289796 482508 ? Sl 17:43 0:25 /home/lishenggui/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node /home/lishenggui/.vscode-server/extensions/ms-python.vscode-pylance-2024.4.1/dist/server.bundle.js --cancellationReceive=file:5946790779c12b9e336d5758f1b93c58461b5cde95 --node-ipc --clientProcessId=1169435\n",
"zhaowan+ 1206902 10.0 0.0 14176 5304 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"zhaowan+ 1207036 0.0 0.0 12124 2548 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"lisheng+ 3761623 0.1 0.0 1219948 387940 ? Sl 11:48 0:25 /home/lishenggui/.vscode-server/cli/servers/Stable-5c3e652f63e798a5ac2f31ffd0d863669328dc4c/server/node /home/lishenggui/.vscode-server/extensions/ms-python.vscode-pylance-2024.4.1/dist/server.bundle.js --cancellationReceive=file:cac8d8fe58263fb931aacdabe459d3cf363759b6a6 --node-ipc --clientProcessId=3735637\n",
"\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"\n",
"HOST: h800-84\n",
"COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" root 4731 0.0 0.0 35612 15176 ? Ss Mar19 0:02 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n",
"zhaowan+ 2207851 5.0 0.0 14176 5360 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"zhaowan+ 2207944 0.0 0.0 12124 2672 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"\n",
"HOST: h800-85\n",
"COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" root 4901 0.0 0.0 29820 18840 ? Ss Apr04 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n",
"zhaowan+ 2201547 0.0 0.0 8700 3260 pts/2 T Apr13 0:00 bash -ic nohup sleep 3m\n",
"root 2388479 0.0 0.0 5613836 24020 pts/5 Sl+ Apr13 1:29 docker run -it --gpus all --entrypoint=bash image.luchentech.com/base/opensora:1.0.0-ubuntu20.04-python3.10-torch2.2.1-cuda121\n",
"litianyi 3016261 6.7 0.0 275912 41260 pts/28 Sl+ 12:01 24:23 /home/share/software/miniconda3/build/bin/python /home/litianyi/.local/bin/nvitop\n",
"litianyi 3169865 0.2 0.0 5781128 292448 pts/26 Sl+ 15:17 0:23 /home/litianyi/.conda/envs/opensorawb/bin/python /home/litianyi/.conda/envs/opensorawb/bin/torchrun --standalone --nproc_per_node 1 scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3169959 216 0.5 136100876 12633772 ? Rsl 15:17 357:05 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3170857 1.5 0.1 80501332 2850552 ? Sl 15:18 2:32 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3170920 1.5 0.1 80689012 3038212 ? Sl 15:18 2:30 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3170983 1.5 0.1 80685000 3034264 ? Sl 15:18 2:29 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171046 1.5 0.1 80600804 2950032 ? Sl 15:18 2:31 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171109 1.4 0.1 80714872 3063928 ? Sl 15:18 2:26 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171172 1.5 0.1 80746208 3095252 ? Sl 15:18 2:29 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171235 1.5 0.1 80663276 3012272 ? Sl 15:18 2:31 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171298 1.5 0.1 80682696 3031880 ? Sl 15:18 2:30 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171361 1.5 0.1 80556108 2905152 ? Sl 15:18 2:33 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171424 1.5 0.1 80678164 3026948 ? Sl 15:18 2:28 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171487 1.5 0.1 80644056 2993252 ? Sl 15:18 2:29 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171550 1.5 0.1 80768108 3117204 ? Sl 15:18 2:28 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171613 1.5 0.1 80692264 3041440 ? Sl 15:18 2:30 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171676 1.5 0.1 80629512 2978456 ? Sl 15:18 2:33 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171739 1.5 0.1 80599356 2948260 ? Sl 15:18 2:32 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3171802 1.5 0.1 80815668 3164776 ? Sl 15:18 2:29 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3317198 1.7 0.0 5782152 310412 pts/34 Sl+ 17:52 0:09 /home/litianyi/.conda/envs/opensorawb/bin/python /home/litianyi/.conda/envs/opensorawb/bin/torchrun --standalone --nproc_per_node 1 scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3317292 442 0.3 84751060 6609672 ? Ssl 17:52 42:48 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3318034 3.4 0.2 52835540 4631824 ? Sl 17:53 0:18 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3318097 3.4 0.2 52835552 4630380 ? Sl 17:53 0:18 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3318160 3.4 0.2 52835564 4629260 ? Sl 17:53 0:18 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3318223 3.3 0.2 52835576 4630124 ? Sl 17:53 0:17 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n",
"litianyi 3323247 0.0 0.0 5476 580 ? S 17:59 0:00 sleep 180\n",
"zhaowan+ 3325310 10.0 0.0 8496 5284 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"litianyi 3325318 0.0 0.0 5476 516 ? S 18:02 0:00 sleep 1\n",
"litianyi 3325354 0.0 0.0 5476 520 ? S 18:02 0:00 sleep 1\n",
"zhaowan+ 3325418 0.0 0.0 6444 2620 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"\n",
"HOST: h800-86\n",
"COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" root 4893 0.0 0.0 35360 18904 ? Ss Apr02 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n",
"zhaowan+ 25146 10.0 0.0 14600 5740 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"zhaowan+ 25327 0.0 0.0 12148 2536 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"\n",
"==== STDERR ====\n",
" bash: 无法设定终端进程组(-1): 对设备不适当的 ioctl 操作\n",
"bash: 此 shell 中无任务控制\n",
"\n",
"HOST: h800-170\n",
"COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" root 4085 0.0 0.0 35768 19104 ? Ss Mar20 0:03 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n",
"zhaowan+ 2338486 5.0 0.0 14176 5428 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"zhaowan+ 2338580 0.0 0.0 12124 2708 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"\n",
"HOST: h800-171\n",
"COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n",
"==== STDOUT ====\n",
" root 4813 0.0 0.0 29676 18824 ? Ss Mar19 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n",
"zhaowan+ 838735 10.0 0.0 8496 5208 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n",
"zhaowan+ 838828 0.0 0.0 6444 2612 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n",
"\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"\n"
]
}
],
"outputs": [],
"source": [
"ps()"
]
@ -877,7 +361,7 @@
},
{
"cell_type": "code",
"execution_count": 23,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -909,23 +393,14 @@
},
{
"cell_type": "code",
"execution_count": 24,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"cd /home/zhaowangbo/zangwei/opensora/ && colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py configs/opensora-v1-1/train/video.py --wandb True --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --load-path outputs/764-STDiT2-XL-2/epoch1-global_step6000\n"
]
}
],
"outputs": [],
"source": [
"host = \"h800-80\"\n",
"log_file = os.path.join(OPEN_SORA_HOME, \"logs/train_02.log\")\n",
"data_path = \"/home/zhaowangbo/data/csv/video_image_test_2.csv\"\n",
"ckpt_path = \"outputs/764-STDiT2-XL-2/epoch1-global_step6000\"\n",
"cmd = colossal_run(data_path, ckpt_path)\n",
"host = \"host-0\"\n",
"log_file = os.path.join(OPEN_SORA_HOME, \"logs/train.log\")\n",
"data_path = \"/path/to/meta.csv\"\n",
"cmd = colossal_run(data_path)\n",
"print(cmd)"
]
},
@ -940,39 +415,18 @@
},
{
"cell_type": "code",
"execution_count": 27,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HOST: h800-80\n",
"COMMAND: bash -ic 'cd /home/zhaowangbo/zangwei/opensora/ && cat hostfile | xargs -I \"{}\" ssh \"{}\" pkill -9 python'\n",
"==== STDERR ====\n",
" bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n",
"bash: no job control in this shell\n",
"pkill: killing pid 382879 failed: Operation not permitted\n",
"\n"
]
}
],
"outputs": [],
"source": [
"cmd = kill_all()\n",
"run_command(cmd, host)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
@ -986,9 +440,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
"version": "3.9.18"
}
},
"nbformat": 4,
"nbformat_minor": 2
"nbformat_minor": 4
}

4
tools/scene_cut/convert_id_to_path.py
View file

@ -7,10 +7,12 @@ from functools import partial
import numpy as np
import pandas as pd
from pandarallel import pandarallel
from tqdm import tqdm
import cv2
from mmengine.logging import print_log
from moviepy.editor import VideoFileClip
from tqdm import tqdm
tqdm.pandas()
def is_intact_video(video_path, mode="moviepy", verbose=False, logger=None):

14
tools/scene_cut/cut.py
View file

@ -9,6 +9,9 @@ from imageio_ffmpeg import get_ffmpeg_exe
from mmengine.logging import MMLogger, print_log
from pandarallel import pandarallel
from scenedetect import FrameTimecode
from tqdm import tqdm
tqdm.pandas()
def process_single_row(row, args, log_name=None):
@ -92,25 +95,18 @@ def split_video(
# for the remaining calls.
# cmd += ['-v', 'error']
# clip to cut
# -ss after -i is very slow; put -ss before -i
# input path
# cmd += ["-i", video_path]
# clip to cut
# cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-t", str(duration.get_seconds())]
# clip to cut
cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-i", video_path, "-t", str(duration.get_seconds())]
# target fps
# cmd += ['-vf', 'select=mod(n\,2)']
if target_fps is not None:
cmd += ["-r", f"{target_fps}"]
# aspect ratio
if shorter_size is not None:
cmd += ["-vf", f"scale='if(gt(iw,ih),-2,{shorter_size})':'if(gt(iw,ih),{shorter_size},-2)'"]
# cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"]
# cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"]
cmd += ["-map", "0", save_path]

2
tools/scene_cut/scene_detect.py
View file

@ -7,6 +7,8 @@ from pandarallel import pandarallel
from scenedetect import AdaptiveDetector, detect
from tqdm import tqdm
tqdm.pandas()
def process_single_row(row):
# windows

2
tools/scoring/README.md
View file

@ -30,7 +30,7 @@ Then, run the following command. **Make sure** the meta file has column `path` (
```bash
torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference /path/to/meta.csv --bs 1024 --num_workers 16
```
This will generate multiple part files. Run `python -m tools.datasets.datautil /path/to/meta_part1.csv /path/to/meta_part2.csv` to merge these part files.
This will generate multiple part files, each corresponding to a node . Run `python -m tools.datasets.datautil /path/to/meta_aes_part*.csv --output /path/to/meta_aes.csv` to merge them.
## Optical Flow Score