From d110689f29018d5ffb7d0fc3da574ceaffdc566e Mon Sep 17 00:00:00 2001 From: Yanjia0 <42895286+Yanjia0@users.noreply.github.com> Date: Wed, 24 Apr 2024 13:19:29 +0800 Subject: [PATCH 1/3] Create report_v2.md create new Chinese report for v1.1 --- docs/zh_CN/report_v2.md | 114 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 docs/zh_CN/report_v2.md diff --git a/docs/zh_CN/report_v2.md b/docs/zh_CN/report_v2.md new file mode 100644 index 0000000..0b59c0d --- /dev/null +++ b/docs/zh_CN/report_v2.md @@ -0,0 +1,114 @@ +# Open-Sora 1.1 技术报告 + +- [模型架构修改](#模型架构修改) +- [支持不同视频长度/分辨率/宽高比/帧率（fps）训练](#支持不同视频长度分辨率宽高比帧率fps训练) +- [使用Masked DiT作为图生视频/视频生视频模型](#使用masked-dit作为图生视频视频生视频模型) +- [数据收集和流程](#数据收集和流程) +- [训练详情](#训练详情) +- [结果和评价](#结果和评价) +- [不足和下一步计划](#不足和下一步计划) + +在Open-Sora1.1版本中，我们使用了10M数据来训练经过结构调优后的STDiT的700M模型（Open-Sora1.0版本仅用400K数据）。我们实现了[Sora报告](https://openai.com/research/video-generation-models-as-world-simulators)中提到的以下功能： + +- 可变的视频时长、分辨率、宽高比（包括采样灵活性、改进的取景范围和构图） +- 提示词增加图片和视频选项（使图像动起来、生成式增长视频、视频到视频编辑、连接不同视频） +- 图像生成功能 + +为了实现这一目标，我们在预训练阶段使用了多任务学习。对于扩散模型来说，用不同的采样时间步长进行训练已经是一种多任务学习。我们将这一思想在图像和视频的条件生成模型上，进一步扩展到多分辨率、宽高比、帧长、fps以及不同的掩码策略。我们在**0~15s、144p到720p、各种宽高比的视频**上训练模型。虽然由于训练FLOPs不足的限制，生成的视频在时间一致性上的表现没有那么高，但我们仍然可以看到这个模型的巨大潜力。 + +## 模型架构修改 + +我们对原始ST-DiT模型进行了以下修改，以获得更好的训练稳定性和模型性能（ST-DiT-2）： + +- **在时间注意力模块中添加[旋转位置编码](https://arxiv.org/abs/2104.09864)**：遵循目前LLM的最佳实践，我们将时间注意力模块中的正弦位置编码更改为旋转位置编码，因为它也算一项序列预测任务。 +- **在时间注意力模块中添加AdaIN和Layernormal**：我们将时间注意力与AdaIN和Layer范数作为空间注意力包裹起来，以稳定训练。 +- **[QK归一化](https://arxiv.org/abs/2302.05442)与[RMSNorm](https://arxiv.org/abs/1910.07467)**：和[SD3](https://arxiv.org/pdf/2403.03206.pdf)类似地，我们应用QK归一化来提高半精度训练的稳定性。 +- **支持动态输入大小和视频条件限定**：为了支持多分辨率、宽高比和fps训练，我们ST-DiT-2来接受任何输入大小。延申[PixArt-alpha](https://github.com/PixArt-alpha/PixArt-alpha)的想法，我们支持限定视频的高度、宽度、宽高比、帧长和fps。 +- **将T5token数量从120扩展到200**：我们使用的视频描述通常少于200个token，我们发现模型也可以很好地处理更长的文本。 + +## 支持不同视频长度/分辨率/宽高比/帧率（fps）训练 + +正如[Sora报告](https://openai.com/research/video-generation-models-as-world-simulators)中提到的，使用原始无损视频的分辨率、宽高比和视频长度进行训练可以增加采样灵活性，改善取景和构图。我们找到了三种实现这一目标的方法： +- [NaViT](https://arxiv.org/abs/2307.06304)：通过不同掩码策略支持在同一训练批次内使用不同大小的数据，并且训练效率下降很少。然而，该系统实现起来有点复杂，并且可能无法兼容kernal优化技术（如flashattention）。 +- 填充（[FiT](https://arxiv.org/abs/2402.12376)，[Open-Sora-Plan](https://github.com/PKU-YuanGroup/Open-Sora-Plan)）：通过填充支持同一批次内的不同大小的数据。然而，将不同的分辨率填充到相同的大小会导致效率降低。 +- 分桶训练（[SDXL](https://arxiv.org/abs/2307.01952)、[PixArt](https://arxiv.org/abs/2310.00426)）：支持通过分桶的方式在不同批次中动态调整大小，但在同一批次内数据大小必须相同，只能应用固定数量的数据大小。在一个批次中，我们不需要实现复杂的掩码或填充。 + +为了更便捷的实现，我们选择分桶训练的方式。我们预先定义了一些固定的分辨率，并将不同的样本分配到不同的桶中。下面列出了分桶方案中值得注意的点。但我们可以看到，这些在我们的实验中并不是一个大问题。 + +
+查看注意事项 + +- 桶大小被限制为固定数量：首先，在实际应用中，通常只使用少数宽高比（9:16、3:4）和分辨率（240p、1080p）。其次，我们发现经过训练的模型可以很好地推广到未见过的解决方案。 +- 每批的大小相同，打破了独立同分布（i.i.d.）假设：由于我们使用多个 GPU，因此不同 GPU 上的本地批次具有不同的大小。我们没有发现此问题导致性能显着下降。 +- 可能没有足够的样本来填充每个桶，并且分布可能有偏差：首先，当本地批量大小不太大时，我们的数据集足够大以填充每个桶。其次，我们应该分析数据大小的分布并相应地定义桶大小。第三，分配不平衡并没有显着影响训练过程。 +- 不同的分辨率和帧长可能有不同的处理速度：与PixArt只处理相似分辨率（相似token数）的宽高比不同，我们需要考虑不同分辨率和帧长的处理速度。我们可以使用“bucket_config”来定义每个桶的批量大小，以确保处理速度相似。 + +
+ +![bucket](/assets/readme/report_bucket.png) + +如图所示，桶是（分辨率，帧数量，宽高比）的三元组。我们为不同的分辨率提供预定义的宽高比，涵盖了大多数常见的视频宽高比。在每个epoch之前，我们打乱数据集并将样本分配到不同的桶中，如图所示。我们将样本放入最大分辨率和帧长度小于视频的桶中。 + +考虑到我们的计算资源有限，我们进一步为每个（分辨率，num_frame）二元组引入keep_prob和batch_size两个属性，以降低计算成本并实现多阶段训练。具体来说，高清视频将以概率1-keep_prob下采样到较低分辨率的桶中，并且每个桶的样本数量是由batch_size属性决定的。这样，我们可以控制不同桶中的样本数量，并通过为每个桶搜索合适的数据量来平衡GPU负载。 + +有关训练中桶使用的详细说明，请参阅[配置文件](/docs/config.md#training-bucket-configs). + +## 使用Masked DiT作为图生视频/视频生视频模型 + +Transformer可以很容易地扩展到支持图生图和视频生视频的任务。我们提出了一种蒙版策略来支持图像和视频的调节。蒙版策略如下图所示。 + +![mask strategy](/assets/readme/report_mask.png) + +在将图像或视频转换成另一个视频的过程中，我们通常会选择出需要作为条件的帧并取消其掩码（unmask）。在使用ST-DiT模型进行前向传播时，被选择取消掩码（unmask）的帧将被赋予时间步长0，而其他帧则保持它们原有的时间步长t。我们发现，如果直接将这种策略应用到训练好的模型上，会得到较差的结果，因为扩散模型在训练过程中并未学会如何处理一个样本中具有不同时间步长的帧。 + +受[UL2](https://arxiv.org/abs/2205.05131)的启发，我们在训练期间引入了随机掩码策略。具体来说，我们在训练期间随机取消掩码帧，包括取消掩码第一帧，前k帧，最后k帧，最后k帧，第一和最后k帧，随机帧等。基于Open-Sora 1.0模型，以50%的概率应用掩码策略，我们发现模型能够在10,000步的训练中学会处理图像条件（而30%的概率会导致处理能力变差），同时文本到视频的性能略有下降。因此，在Open-Sora 1.1版本中，我们从头开始预训练模型，并采用了掩码策略。 + +下图给出了用于推理的掩码策略配置的说明。五数字元组在定义掩码策略方面提供了极大的灵活性。 + +![mask strategy config](/assets/readme/report_mask_config.png) + +掩码策略用法的详细说明可在[配置文件](/docs/config.md#advanced-inference-config)中查看. + + +## 数据收集和流程 + +正如我们在Sora1.0版本中看见的那样，数据数量和质量对于训练一个好的模型至关重要，因此，我们努力扩展数据集。首先，我们创建了一个遵循[SVD](https://arxiv.org/abs/2311.15127)的自动流水线，包括场景切割、字幕、各种评分和过滤以及数据集管理脚本和通用惯例。 + +![pipeline](/assets/readme/report_data_pipeline.png) + +我们计划使用[panda-70M](https://snap-research.github.io/Panda-70M/)和其他数据来训练模型，大约包含3000万条数据。然而，我们发现磁盘输入输出（disk IO）在同时进行训练和数据处理时成为了一个瓶颈。因此，我们只能准备一个包含1000万条数据的数据集，并且没有完成我们构建的所有处理流程。最终，我们使用了包含970万视频和260万图像的数据集进行预训练，以及560,000视频和160万图像的数据集进行微调。预训练数据集的统计信息如下所示。 + +图像文本标记 (使用T5分词器)： +![image text tokens](/assets/readme/report_image_textlen.png) + +视频文本标记 (使用T5分词器)。我们直接使用Panda的短视频描述进行训练，并自己给其他数据集加视频描述。生成的字幕通常少于200个token。 +![video text tokens](/assets/readme/report_video_textlen.png) + +视频时长： +![video duration](/assets/readme/report_video_duration.png) + +## 训练详情 + +由于计算资源有限，我们必须仔细监控训练过程，并在推测模型学习不佳时更改训练策略，因为没有消融研究的计算。因此，Open-Sora1.1版本的训练包括多个更改，所以，指数移动平均（EMA）未被应用。 + +1. 首先，我们从`Pixart-alpha-1024`的模型checkpoint开始，使用不同分辨率的图像进行了6000步的微调。我们发现模型能够很容易地适应并生成不同分辨率的图像。为了加快扩散过程的训练，我们使用了[SpeeDiT](https://github.com/1zeryu/SpeeDiT)（iddpm-speed）技术。 +2. **[阶段一]** 然后，我们使用梯度检查点（gradient-checkpointing）技术对模型进行了**24,000**步的预训练，这个过程在64个H800 GPU上运行了**4天**。尽管模型看到的数据样本数量相同，我们发现与使用较小批量大小相比，模型的学习速度较慢。我们推测，在训练的早期阶段，步数的数量对于训练更为重要。大多数视频的分辨率是**240p**，预训练时使用的配置与[stage2.py](/configs/opensora-v1-1/train/stage2.py)相似。 +3. **[阶段一]** 为了增加训练步数，我们改用了更小的批量大小，并且没有使用梯度检查点技术。在这个阶段，我们还引入了帧率（fps）条件。模型训练了**40,000**步，持续了**2天**。训练中使用的视频大多数是**144p**分辨率，使用的配置文件是[stage1.py](/configs/opensora-v1-1/train/stage1.py)。我们使用较低的分辨率，因为我们在Open-Sora 1.0版本中发现模型可以以相对较低的分辨率学习时间知识。 +4. **[阶段一]** 我们发现模型不能很好地学习长视频，并在Open-Sora1.0训练中发现了一个噪声生成结果，推测是半精度问题。因此，我们采用QK-归一化来稳定训练。我们还将iddpm-speed切换成iddpm。我们训练了**17k**步**14小时**。大多数视频的分辨率是144p，预训练时使用的配置是[stage1.py](/configs/opensora-v1-1/train/stage1.py)。阶段1训练持续约一周，总步长**81k**。 +5. **[阶段二]** 我们切换到更高的分辨率，其中大多数视频是**240p和480p**分辨率（[stage2.py](/configs/opensora-v1-1/train/stage2.py)）。我们在所有预训练数据上训练了**22000**步，持续**一天**。 +6. **[阶段三]** 我们切换到更高的分辨率，大多数视频的分辨率是**480p和720p**（[stage3.py](/configs/opensora-v1-1/train/stage3.py)）。我们在高质量数据上训了**4000**步，用时**一天**。 + +## 结果和评价 + +## 不足和下一步计划 + +随着我们离Sora的复现又近了一步，我们发现当前模型存在许多不足，这些不足将在我们下阶段工作中得到改善。 + +- **噪音的生成和影响**：我们发现生成的模型，特别是长视频中，有时很多噪点，不流畅。我们认为问题在于没有使用时间VAE。由于[Pixart-Sigma](https://arxiv.org/abs/2403.04692)发现适应新VAE很容易，我们计划在下一个版本中为模型开发时间VAE。 +- **缺乏时间一致性**：我们发现模型无法生成具有高时间一致性的视频，我们认为问题是由于缺乏训练FLOPs，我们计划收集更多数据并继续训练模型以提高时间一致性。 +- **人像生成质量低**：我们发现模型无法生成高质量的人类视频，我们认为问题是由于缺乏人类数据，我们计划收集更多的人类数据，并继续训练模型以提高人类生成。 +- **美学得分低**：我们发现模型的美学得分不高。问题在于缺少美学得分过滤，由于IO瓶颈没我们没有进行这一步骤。我们计划通过美学得分和微调模型来过滤数据，以提高美学得分。 +- **长视频生成质量低**：我们发现，使用同样的提示词，视频越长，质量越差。这意味着图像质量不能同等地被不同长度的序列所适应。 + +> - **算法与加速实现**：Zangwei Zheng, Xiangyu Peng, Shenggui Li, Hongxing Liu, Yukun Zhou +> - **数据收集与处理**：Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Young, Junjie Wang, Chenfeng Yu From 2219fc87038604a0b5abbd9e2b11150fd9b5bd31 Mon Sep 17 00:00:00 2001 From: xyupeng <99191637+xyupeng@users.noreply.github.com> Date: Wed, 24 Apr 2024 15:52:07 +0800 Subject: [PATCH 2/3] Dev/pxy (#72) * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scene_cut * update scene_cut * update scene_cut[A * update scene_cut * update scene_cut * update scene_cut * update scene_cut * update scene_cut * update scene_cut * m * m * m * m * m * m * m * m * m * m * m * m * m * m * update readme * update readme * extract frames using opencv everywhere * extract frames using opencv everywhere * extract frames using opencv everywhere * filter panda10m * filter panda10m * m * m * m * m * m * m * m * m * m * m * m * m * m * m * m * m * m * ocr * add ocr * add main.sh * add ocr * add ocr * add ocr * add ocr * add ocr * add ocr * update scene_cut * update remove main.sh * update scoring * update scoring * update scoring * update README * update readme * update scene_cut * update readme * update scoring * update readme * update readme * update filter_panda10m * update readme * update readme * update launch.ipynb * update scene_cut * update scene_cut * update readme * update launch.ipynb --- README.md | 36 +- docs/data_processing.md | 65 ++- notebooks/launch.ipynb | 620 ++------------------------ tools/scene_cut/convert_id_to_path.py | 4 +- tools/scene_cut/cut.py | 14 +- tools/scene_cut/scene_detect.py | 2 + tools/scoring/README.md | 2 +- 7 files changed, 113 insertions(+), 630 deletions(-) diff --git a/README.md b/README.md index 1bc4f14..04cbe08 100644 --- a/README.md +++ b/README.md @@ -247,39 +247,11 @@ To lower the memory usage, set a smaller `vae.micro_batch_size` in the config (s ## Data Processing +High-quality data is crucial for training good generation models. +To this end, we establish a complete pipeline for data processing, which could seamlessly convert raw videos to high-quality video-text pairs. +The pipeline is shown below. For detailed information, please refer to [data processing](docs/data_processing.md). +Also check out the [datasets](docs/data_processing.md) we use. -Te be modified - -High-quality Data is the key to high-quality models. Our used datasets and data collection plan -is [here](/docs/datasets.md). We provide tools to process video data. Our data processing pipeline includes -the following steps: - -1. Manage datasets. [[docs](/tools/datasets/README.md)] -2. Scene detection and video splitting. [[docs](/tools/scene_cut/README.md)] -3. Score and filter videos. [[docs](/tools/scoring/README.md)] -4. Generate video captions. [[docs](/tools/caption/README.md)] - -Below is an example workflow to process data. However, we recommend you to read the detailed documentation for each tool, and decide which tools to use based on your needs. This pipeline applies to both image and video data. Full pipeline is available in [datasets.md](/tools/datasets/README.md#data-process-pipeline). - -```bash -# Suppose videos and images under ~/dataset/ -# 1. Convert dataset to CSV (meta.csv) -python -m tools.datasets.convert video ~/dataset --output meta.csv -# 2. Get video information (meta_info_fmin1.csv) -python -m tools.datasets.datautil meta.csv --info --fmin 1 -# 3. Get caption information -torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava meta_info_fmin1.csv --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video -# merge generated results (meta_caption.csv) -python -m tools.datasets.datautil meta_info_fmin1_caption_part*.csv --output meta_caption.csv -# clean caption (meta_caption_processed.csv) -python -m tools.datasets.datautil meta_caption.csv --clean-caption --refine-llm-caption --remove-empty-caption --output meta_caption_processed.csv -# 4. Scoring (meta_caption_processed_aes.csv) -torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference meta_caption_processed.csv --bs 1024 --num_workers 16 -# Filter videos by aesthetic scores (meta_aes_aesmin5.csv) -python -m tools.datasets.csvutil meta_caption_processed_aes.csv --aesmin 5 --output meta_aes_aesmin5.csv -# 5. Additional filtering -python -m tools.datasets.csvutil ~/dataset_ready.csv --fmin 48 -``` ![Data Processing Pipeline](assets/readme/report_data_pipeline.png) diff --git a/docs/data_processing.md b/docs/data_processing.md index 21c55e3..f907194 100644 --- a/docs/data_processing.md +++ b/docs/data_processing.md @@ -13,8 +13,65 @@ conduct camera motion detection for the remaining samples. In summary, our pipeline produces video-text pairs which have high aesthetic quality, large video motion and strong semantic consistency. +Below is an example workflow to process videos. + +```bash +ROOT_VIDEO="/path/to/video/folder" +ROOT_CLIPS="/path/to/video/clips/folder" +ROOT_META="/path/to/meta/folder" + +# 1.1 Create a meta file from a video folder. This should output ${ROOT_META}/meta.csv +python -m tools.datasets.convert video ${ROOT_VIDEO} --output ${ROOT_META}/meta.csv + +# 1.2 Get video information and remove broken videos. This should output ${ROOT_META}/meta_info_fmin1.csv +python -m tools.datasets.datautil ${ROOT_META}/meta.csv --info --fmin 1 + +# 2.1 Detect scenes. This should output ${ROOT_META}/meta_info_fmin1_timestamp.csv +python -m tools.scene_cut.scene_detect ${ROOT_META}/meta_info_fmin1.csv + +# 2.2 Cut video into clips based on scenes. This should produce video clips under ${ROOT_CLIPS} +python -m tools.scene_cut.cut ${ROOT_META}/meta_info_fmin1_timestamp.csv --save_dir ${ROOT_CLIPS} + +# 2.3 Create a meta file for video clips. This should output ${ROOT_META}/meta_clips.csv +python -m tools.datasets.convert video ${ROOT_CLIPS} --output ${ROOT_META}/meta_clips.csv + +# 2.4 Get clips information and remove broken ones. This should output ${ROOT_META}/meta_clips_info_fmin1.csv +python -m tools.datasets.datautil ${ROOT_META}/meta_clips.csv --info --fmin 1 + +# 3.1 Predict aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_part*.csv +torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference \ + ${ROOT_META}/meta_clips_info_fmin1.csv \ + --bs 1024 \ + --num_workers 16 + +# 3.2 Merge files; This should output ${ROOT_META}/meta_clips_info_fmin1_aes.csv +python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes_part*.csv --output ${ROOT_META}/meta_clips_info_fmin1_aes.csv + +# 3.2 Filter by aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv +python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes.csv --aesmin 5 + +# 4.1 Generate caption. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv +torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava \ + ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv \ + --dp-size 8 \ + --tp-size 1 \ + --model-path /path/to/llava-v1.6-mistral-7b \ + --prompt video + +# 4.2 Merge caption results. This should output ${ROOT_META}/meta_clips_caption.csv +python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv --output ${ROOT_META}/meta_clips_caption.csv + +# 4.3 Clean caption. This should output ${ROOT_META}/meta_clips_caption_cleaned.csv +python -m tools.datasets.datautil \ + ${ROOT_META}/meta_clips_caption.csv \ + --clean-caption \ + --refine-llm-caption \ + --remove-empty-caption \ + --output ${ROOT_META}/meta_clips_caption_cleaned.csv +``` + For more information, please refer to: -- [Dataset Management](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/datasets/README.md) -- [Scene Detection and Video Splitting](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/scene_cut/README.md) -- [Scoring and Filtering](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/scoring/README.md) -- [Captioning](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/caption/README.md) \ No newline at end of file +- [Dataset Management](../tools/datasets/README.md) +- [Scene Detection and Video Splitting](../tools/scene_cut/README.md) +- [Scoring and Filtering](../tools/scoring/README.md) +- [Captioning](../tools/caption/README.md) \ No newline at end of file diff --git a/notebooks/launch.ipynb b/notebooks/launch.ipynb index 29dfafb..309d7e4 100644 --- a/notebooks/launch.ipynb +++ b/notebooks/launch.ipynb @@ -16,13 +16,14 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", - "OPEN_SORA_HOME = \"/home/zhaowangbo/zangwei/opensora/\"\n", + "# TODO: change to your own project path!!!\n", + "OPEN_SORA_HOME = \"/path/to/Open-Sora/\"\n", "\n", "\n", "def convert_dataset_cmd(input_dir, output_file, datatype=\"video\"):\n", @@ -123,6 +124,17 @@ " return \" && \".join(commands), output_file\n", "\n", "\n", + "def get_ocr(input_file):\n", + " commands = []\n", + " base, ext = os.path.splitext(input_file)\n", + " output_file = f\"{base}_match{ext}\"\n", + "\n", + " commands.append(f'echo \"Getting match score of {input_file} to {output_file}\"')\n", + " commands.append(f\"cd {OPEN_SORA_HOME}\")\n", + " commands.append(f\"torchrun --standalone --nproc_per_node 8 -m tools.scoring.ocr.inference {input_file}\")\n", + " return \" && \".join(commands), output_file\n", + "\n", + " \n", "def get_match_score(input_file):\n", " commands = []\n", " base, ext = os.path.splitext(input_file)\n", @@ -167,18 +179,18 @@ "source": [ "### Remote Launch via Paramiko\n", "\n", - "First, you should add hosts in your ~/.ssh/config file" + "First, add hosts to `~/.ssh/config`" ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import paramiko\n", "\n", - "HOSTS = [\"h800-80\", \"h800-81\", \"h800-82\", \"h800-83\", \"h800-84\", \"h800-85\", \"h800-86\", \"h800-170\", \"h800-171\"]\n", + "HOSTS = [\"host-0\", \"host-1\", \"host-2\", \"host-3\", \"host-4\", \"host-5\", \"host-6\", \"host-7\"]\n", "\n", "# load from ~/.ssh/config\n", "ssh_config = paramiko.SSHConfig()\n", @@ -246,7 +258,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ @@ -259,9 +271,9 @@ "\n", "def nvitop(host=None):\n", " if host:\n", - " run_command(f\"/home/zhaowangbo/.local/bin/nvitop -1\", host)\n", + " run_command(f\"/home/user/.local/bin/nvitop -1\", host)\n", " else:\n", - " run_command_all_hosts(\"/home/zhaowangbo/.local/bin/nvitop -1\")\n", + " run_command_all_hosts(\"/home/user/.local/bin/nvitop -1\")\n", "\n", "\n", "def ps(host=None, interest=\"python|sleep|torchrun|colossal\", all=True):\n", @@ -290,389 +302,22 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Examples\n", + "### Example\n", "\n", - "The following is the pipeline for panda." - ] - }, - { - "cell_type": "code", - "execution_count": 49, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "echo \"Getting aesthetic score of /mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info.csv to /mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info_aes.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && torchrun --standalone --nproc_per_node 8 -m tools.scoring.aesthetic.inference /mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info.csv && python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info_part*.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info_aes.csv --format csv --sort aes && echo \"All Done!\"\n", - "/mnt/hdd/data/panda70m_by/raw/meta/split-16/meta_info_caption_info_aes.csv\n" - ] - } - ], - "source": [ - "# panda\n", - "host = \"h800-83\"\n", - "split = 16\n", - "input_dir = f\"/mnt/disk1/data-panda/{split}\"\n", - "log_file = os.path.join(OPEN_SORA_HOME, f\"logs/data-panda-{split}-split.log\")\n", - "output_file = f\"/mnt/hdd/data/panda70m_by/raw/meta/split-{split}/meta.csv\"\n", - "cmd, output_file = get_commands(\n", - " [\n", - " # {\n", - " # \"cmd\": convert_dataset_cmd,\n", - " # \"input_dir\": input_dir,\n", - " # \"output_file\": output_file,\n", - " # },\n", - " # {\n", - " # \"cmd\": get_caption_load,\n", - " # },\n", - " # {\n", - " # \"cmd\": get_video_info_torchvision,\n", - " # },\n", - " # {\n", - " # \"cmd\": get_aesthetic_score,\n", - " # },\n", - " # {\n", - " # \"cmd\": get_flow_score,\n", - " # },\n", - " # {\n", - " # \"cmd\": get_match_score,\n", - " # },\n", - " # {\n", - " # \"cmd\": get_cmotion_score,\n", - " # },\n", - " ]\n", - ")\n", - "print(cmd)\n", - "print(output_file)" - ] - }, - { - "cell_type": "code", - "execution_count": 45, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "echo \"Getting info of /mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean.csv to /mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1 && echo \"All Done!\"\n", - "/mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean_info.csv\n" - ] - } - ], - "source": [ - "# panda\n", - "host = \"h800-82\"\n", - "split = 7\n", - "log_file = os.path.join(OPEN_SORA_HOME, f\"logs/data-panda-{split}-split.log\")\n", - "cmd, output_file = get_commands(\n", - " [\n", - " {\n", - " \"cmd\": get_video_info_torchvision,\n", - " \"input_file\": f\"/mnt/hdd/data/panda70m_by/raw/meta/split-7/meta_loadjson_noempty_clean.csv\",\n", - " },\n", - " ]\n", - ")\n", - "print(cmd)\n", - "print(output_file)" - ] - }, - { - "cell_type": "code", - "execution_count": 83, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "echo \"Getting info of /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted.csv to /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && conda activate llava2 && torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted.csv --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video && conda activate opensora && python -m tools.datasets.datautil /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted_caption_part*.csv --output /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted_info.csv --format csv --intersection /home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted.csv --clean-caption --refine-llm-caption --remove-empty-caption && echo \"All Done!\"\n", - "/home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted_info.csv\n" - ] - } - ], - "source": [ - "# v2text\n", - "host = \"h800-86\"\n", - "log_file = os.path.join(OPEN_SORA_HOME, f\"logs/data-v2text-18.log\")\n", - "input_file = \"/home/zhaowangbo/data/v2text/raw/meta/split-18/meta_remove_corrupted.csv\"\n", - "cmd, output_file = get_commands(\n", - " [\n", - " {\n", - " \"cmd\": get_caption_llava7b_video,\n", - " \"input_file\": input_file,\n", - " },\n", - " ]\n", - ")\n", - "print(cmd)\n", - "print(output_file)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ "Remote launch via paramiko." ] }, { "cell_type": "code", - "execution_count": 67, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "HOST: h800-80\n", - "COMMAND: nohup bash -ic 'echo \"Getting info of /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv to /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && conda activate llava2 && torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video && conda activate opensora && python -m tools.datasets.datautil /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_part*.csv --output /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_info.csv --format csv --intersection /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv --clean-caption --refine-llm-caption --remove-empty-caption && echo \"All Done!\"' >> /home/zhaowangbo/zangwei/opensora/logs/data-v2text-12.log 2>&1 &\n", - "HOST: h800-80\n", - "COMMAND: bash -ic 'ps ux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " zhaowan+ 3482147 2.0 0.0 8496 5244 ? S 13:24 0:00 bash -ic echo \"Getting info of /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv to /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && conda activate llava2 && torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video && conda activate opensora && python -m tools.datasets.datautil /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_part*.csv --output /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted_info.csv --format csv --intersection /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv --clean-caption --refine-llm-caption --remove-empty-caption && echo \"All Done!\"\n", - "zhaowan+ 3482641 10.0 0.0 8488 5088 ? Ss 13:24 0:00 bash -ic ps ux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "zhaowan+ 3482792 42.0 0.0 2933116 227692 ? R 13:24 0:00 /home/zhaowangbo/.conda/envs/llava2/bin/python /home/zhaowangbo/.conda/envs/llava2/bin/torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava /home/zhaowangbo/data/v2text/raw/meta/split-12/meta_remove_corrupted.csv --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video\n", - "zhaowan+ 3482808 0.0 0.0 6412 728 ? S 13:24 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "\n" - ] - } - ], + "outputs": [], "source": [ "sleep = None\n", "run_command(cmd, host, log_file=log_file, nohup=True, sleep=sleep)\n", "ps(host)" ] }, - { - "cell_type": "code", - "execution_count": 81, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "HOST: h800-84\n", - "COMMAND: bash -ic 'ps ux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " zhaowan+ 697488 0.8 0.0 21302928 982860 ? Sl 07:54 3:19 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 756910 2.3 0.0 28226540 982328 ? Sl 07:55 9:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757066 22.0 0.1 56222740 4023872 ? Il 07:55 86:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757068 21.2 0.2 62354592 4400760 ? Il 07:55 83:44 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757115 20.6 0.3 61031756 8265648 ? Il 07:55 81:23 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757132 22.6 0.2 68694032 4909496 ? Il 07:55 89:27 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757134 26.8 0.2 63051476 5462788 ? Rl 07:55 105:47 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757181 23.8 0.2 49112896 4392012 ? Il 07:55 94:00 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757183 23.2 0.2 66357972 5091892 ? Rl 07:55 91:41 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757186 22.2 0.2 73577628 5263940 ? Il 07:55 87:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757194 21.2 0.2 61154288 4409388 ? Il 07:55 83:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757220 22.5 0.2 56297716 5071064 ? Il 07:55 88:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757222 21.0 0.4 80284000 8926020 ? Rl 07:55 83:07 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757224 21.3 0.2 61690852 4616640 ? Il 07:55 84:16 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757300 21.4 0.2 63578668 4423544 ? Il 07:55 84:35 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757348 21.7 0.2 79282760 5193352 ? Il 07:55 85:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757350 26.9 0.2 60714784 4810316 ? Rl 07:55 106:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757367 22.2 0.2 70532632 5095456 ? Il 07:55 87:39 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757399 22.4 0.2 65492460 4838288 ? Il 07:55 88:23 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757401 24.0 0.2 53068672 5885536 ? Rl 07:55 94:41 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757418 21.8 0.2 61157180 4504976 ? Il 07:55 86:16 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757435 22.8 0.2 62984764 5448604 ? Il 07:55 90:09 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757437 22.0 0.2 67874668 5040264 ? Rl 07:55 86:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757439 22.7 0.2 68867624 4337728 ? Il 07:55 89:29 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757456 20.2 0.2 68795064 4935428 ? Il 07:55 79:45 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757458 21.7 0.2 61065776 4931596 ? Il 07:55 85:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757475 22.3 0.2 58261592 4732108 ? Il 07:55 88:08 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757673 20.5 0.1 56322500 4223948 ? Il 07:55 80:59 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757722 22.2 0.4 72053868 10389480 ? Il 07:55 87:41 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757737 21.2 0.2 60275360 4627408 ? Il 07:55 83:54 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757769 20.7 0.2 64496484 4712216 ? Il 07:55 81:50 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757816 19.5 0.2 49375580 5353560 ? Rl 07:55 76:57 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757848 20.8 0.2 67125764 4799684 ? Dl 07:55 82:05 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757865 20.1 0.1 68462464 3823092 ? Il 07:55 79:35 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757897 21.4 0.2 72993788 5109220 ? Il 07:55 84:22 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757914 21.0 0.2 58775368 4358980 ? Rl 07:55 82:48 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757946 27.0 0.2 65172932 4311496 ? Rl 07:55 106:46 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757963 22.8 0.1 60634860 2854188 ? Il 07:55 90:02 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 757980 27.8 0.2 70003380 4786480 ? Il 07:55 109:48 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758027 21.4 0.2 61093344 5021592 ? Dl 07:55 84:40 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758055 21.4 0.2 63173892 4465772 ? Il 07:55 84:38 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758076 27.6 0.2 63197068 4837008 ? Il 07:55 109:08 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758078 19.0 0.2 57450172 4515412 ? Il 07:55 74:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758095 22.8 0.2 63550712 5040656 ? Il 07:55 90:09 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758112 22.3 0.1 60771048 3967276 ? Il 07:55 88:11 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758129 20.9 0.2 76323044 5139840 ? Il 07:55 82:25 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758161 22.9 0.1 63293068 4105016 ? Il 07:55 90:38 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758238 20.7 0.1 58782420 3733868 ? Il 07:55 81:41 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758240 23.6 0.2 60366724 4870712 ? Il 07:55 93:15 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758287 20.4 0.2 67594208 4885716 ? Il 07:55 80:37 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758334 20.9 0.2 60524216 5200828 ? Il 07:55 82:47 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758420 21.9 0.2 61651496 4922668 ? Il 07:55 86:25 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758461 20.0 0.2 58606524 4798940 ? Il 07:55 78:58 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758490 25.6 0.2 57541972 4511752 ? Il 07:55 101:00 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758522 21.4 0.2 57452244 4556340 ? Il 07:55 84:30 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758539 21.8 0.2 64637932 4421864 ? Il 07:55 86:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758541 21.4 0.2 63791100 5004104 ? Il 07:55 84:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758543 21.1 0.2 72491116 4859764 ? Il 07:55 83:26 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758575 21.7 0.2 54628964 4677276 ? Il 07:55 85:45 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758622 20.1 0.2 54905520 4660216 ? Il 07:55 79:34 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758654 20.6 0.2 65326772 4237344 ? Il 07:55 81:24 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758671 21.3 0.2 67287820 4826480 ? Rl 07:55 84:01 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758688 23.9 0.2 66189768 5231632 ? Rl 07:55 94:17 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758735 23.6 0.1 59340972 3407908 ? Il 07:55 93:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758767 21.1 0.2 59026076 4922724 ? Dl 07:55 83:19 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758799 24.9 0.2 51343480 4485852 ? Il 07:55 98:15 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758831 20.7 0.2 73143608 5161436 ? Rl 07:55 81:42 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758833 21.3 0.2 49991612 4309764 ? Il 07:55 84:07 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758836 21.4 0.2 63950844 4979736 ? Il 07:55 84:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758853 24.8 0.2 58920328 4975560 ? Rl 07:55 97:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758885 22.6 0.2 66055404 5366452 ? Il 07:55 89:13 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758932 21.1 0.2 63554972 4502792 ? Il 07:55 83:23 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758949 29.8 0.1 61384460 3507884 ? Il 07:55 117:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758981 22.2 0.1 69580512 4075980 ? Il 07:55 87:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 758983 22.9 0.2 60777712 5068084 ? Rl 07:55 90:38 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759000 23.8 0.2 64954152 4806056 ? Il 07:55 93:50 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759032 26.3 0.2 58059496 4858848 ? Il 07:55 104:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759094 25.8 0.2 65760368 5394312 ? Rl 07:55 101:49 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759126 26.5 0.2 59127380 4639020 ? Il 07:55 104:40 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759128 21.3 0.2 60868916 4568500 ? Il 07:55 84:19 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759130 20.5 0.2 61144416 4353776 ? Il 07:55 81:11 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759147 22.2 0.2 59077984 4776584 ? Dl 07:55 87:40 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759164 22.1 0.2 65963804 4400904 ? Il 07:55 87:09 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759166 21.1 0.2 60764900 5435900 ? Il 07:55 83:15 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759168 25.9 0.2 62305136 5418788 ? Rl 07:55 102:11 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759200 22.1 0.2 63089284 4373652 ? Il 07:55 87:29 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759202 19.9 0.2 49378748 4509304 ? Il 07:55 78:48 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759219 21.2 0.2 73910200 4413528 ? Il 07:55 83:35 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759311 19.9 0.2 58333480 4383732 ? Il 07:55 78:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759328 20.1 0.2 51891348 4394716 ? Il 07:55 79:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759345 21.3 0.2 62480824 4778464 ? Il 07:55 84:13 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759392 21.8 0.2 66537592 5815980 ? Il 07:55 86:02 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759409 21.0 0.2 61670672 4647044 ? Rl 07:55 83:07 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759426 21.0 0.2 67450788 4598964 ? Il 07:55 83:06 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759443 24.9 0.2 60838980 4473136 ? Il 07:55 98:25 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759460 22.9 0.1 64643588 3131576 ? Il 07:55 90:30 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759462 22.5 0.2 65706064 4940932 ? Dl 07:55 88:52 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 759494 20.4 0.2 56295000 4496504 ? Il 07:55 80:43 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 1262361 0.0 0.0 14180 5296 ? S 01:54 0:00 bash -ic sleep 3h; echo \"Getting info of /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv to /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1 && echo \"All Done!\"\n", - "zhaowan+ 1354819 0.0 0.0 14180 5232 ? S 01:54 0:00 bash -ic sleep 6h; echo \"Getting info of /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv to /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv\" && cd /home/zhaowangbo/zangwei/opensora/ && python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-4/meta_loadjson_noempty_clean_info_info.csv --format csv --video-info --fmin 1 && echo \"All Done!\"\n", - "zhaowan+ 1984585 0.8 0.0 21234260 914656 ? Sl 04:54 4:44 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2064791 2.5 0.0 28001092 915132 ? Sl 04:55 14:24 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2064871 18.9 0.1 23901996 3583608 ? Il 04:55 108:52 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2064873 20.1 0.2 24702060 4261168 ? Il 04:55 115:25 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2064965 20.1 0.1 24774688 4017516 ? Sl 04:55 115:42 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065099 20.2 0.1 24040840 3599188 ? Il 04:55 116:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065101 19.3 0.1 23808764 3612580 ? Dl 04:55 111:09 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065118 19.0 0.1 23696028 3377124 ? Il 04:55 109:05 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065135 19.8 0.1 23934804 3577844 ? Sl 04:55 114:08 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065152 21.2 0.1 23751700 3432956 ? Il 04:55 121:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065184 20.4 0.1 23980124 3410420 ? Sl 04:55 117:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065186 20.9 0.1 23809520 3244896 ? Il 04:55 120:01 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065203 20.1 0.1 24171548 3601216 ? Sl 04:55 115:43 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065205 21.6 0.1 23632596 3062812 ? Sl 04:55 124:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065222 20.1 0.1 23794996 3315016 ? Sl 04:55 115:58 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065239 20.2 0.1 24125720 3522592 ? Sl 04:55 116:07 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065241 19.8 0.1 24244016 3764500 ? Sl 04:55 113:46 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065243 21.5 0.1 24222260 3530048 ? Sl 04:55 123:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065245 21.1 0.1 24215696 3735876 ? Sl 04:55 121:39 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065247 21.5 0.1 23730660 3535380 ? Il 04:55 123:58 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065264 21.2 0.1 23773068 3454216 ? Il 04:55 121:45 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065266 20.9 0.3 27030228 6811788 ? Rl 04:55 120:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065283 20.7 0.1 23630864 3312760 ? Il 04:55 119:16 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065300 23.4 0.1 24095968 3621376 ? Il 04:55 134:47 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065317 20.9 0.1 23896548 3454832 ? Il 04:55 120:02 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065319 20.4 0.1 23852780 3495144 ? Sl 04:55 117:08 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065321 22.3 0.2 24970292 4325780 ? Rl 04:55 128:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065545 25.8 0.1 24964468 4157640 ? Rl 04:55 148:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065562 22.1 0.1 23664124 3216992 ? Sl 04:55 126:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065564 23.3 0.2 25803564 5036196 ? Rl 04:55 133:46 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065566 23.3 0.1 23749396 3553960 ? Il 04:55 134:16 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065598 21.0 0.1 23867720 3516672 ? Il 04:55 120:35 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065615 21.7 0.1 24266168 3508132 ? Sl 04:55 124:50 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065627 22.0 0.1 23496084 3144832 ? Il 04:55 126:49 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065649 23.1 0.1 24012732 3694248 ? Il 04:55 132:51 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065780 20.5 0.1 24022680 3826792 ? Il 04:55 118:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065877 21.0 0.1 24020120 3507712 ? Sl 04:55 120:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065879 20.7 0.1 23247420 3051268 ? Il 04:55 119:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065881 24.8 0.1 23613316 3172020 ? Il 04:55 142:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065883 21.1 0.2 25951160 5594956 ? Sl 04:55 121:12 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065915 21.3 0.1 23865148 3295056 ? Sl 04:55 122:50 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065917 19.8 0.1 23812588 3299900 ? Sl 04:55 114:06 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065934 20.3 0.1 24185736 3616268 ? Sl 04:55 116:55 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065936 25.8 0.1 24135688 3816780 ? Il 04:55 148:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2065938 21.8 0.1 23936204 3617428 ? Il 04:55 125:40 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066168 20.6 0.1 24040208 3560608 ? Sl 04:55 118:44 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066254 21.7 0.1 24113416 3671912 ? Il 04:55 124:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066463 20.1 0.1 24137384 3567148 ? Sl 04:55 115:56 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066465 21.0 0.3 27174728 6728004 ? Sl 04:55 120:59 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066467 21.9 0.1 24238792 3674556 ? Il 04:55 126:17 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066469 21.7 0.2 26543524 6101716 ? Rl 04:55 124:40 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066471 20.3 0.3 27072852 6721744 ? Il 04:55 116:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066503 23.5 0.1 24265704 3700984 ? Il 04:55 135:04 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066535 20.4 0.1 24308408 4047644 ? Dl 04:55 117:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066537 23.8 0.1 23862900 3666564 ? Il 04:55 136:49 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066554 25.6 0.1 24714016 4075292 ? Rl 04:55 147:22 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066571 24.8 0.1 24000324 3681940 ? Il 04:55 142:25 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066573 21.4 0.2 25996736 5555128 ? Il 04:55 123:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066575 23.7 0.1 24301656 3949984 ? Il 04:55 136:22 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066577 20.1 0.3 26725368 6406652 ? Rl 04:55 115:54 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066579 24.9 0.1 23801220 3360128 ? Il 04:55 143:30 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066581 21.8 0.1 24127376 3554688 ? Il 04:55 125:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066583 19.5 0.4 30593056 10112812 ? Sl 04:55 112:16 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066585 19.6 0.1 23536080 3089880 ? Sl 04:55 113:05 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066602 19.7 0.1 24258636 3688524 ? Sl 04:55 113:24 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066604 21.4 0.1 23373520 3177488 ? Il 04:55 123:12 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066606 21.3 0.1 23965420 3273080 ? Sl 04:55 122:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066608 23.0 0.1 24060464 3741516 ? Il 04:55 132:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066625 23.1 0.1 24998568 3895104 ? Rl 04:55 132:42 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066627 22.6 0.1 24209284 3768280 ? Il 04:55 129:55 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066629 23.0 0.1 24071816 3721036 ? Rl 04:55 132:31 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066646 23.7 0.1 23595984 3277364 ? Il 04:55 136:21 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066648 21.9 0.3 26924956 6450372 ? Dl 04:55 125:53 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066650 23.3 0.1 23828300 3445160 ? Il 04:55 133:48 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066652 20.5 0.1 24249456 3561936 ? Rl 04:55 118:11 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066654 21.0 0.1 23778072 3427236 ? Il 04:55 120:37 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066656 26.0 0.1 24788272 4198708 ? Rl 04:55 149:23 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066658 20.9 0.1 23684832 3489420 ? Il 04:55 120:12 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066660 20.4 0.1 24054808 3580084 ? Il 04:55 117:17 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066662 24.6 0.2 24910136 4403304 ? Rl 04:55 141:44 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066664 20.4 0.1 24376528 4025124 ? Dl 04:55 117:21 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066666 22.4 0.1 23970000 3523596 ? Sl 04:55 128:52 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066733 19.9 0.1 23976636 3496212 ? Sl 04:55 114:45 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066877 23.6 0.2 24892000 4431224 ? Rl 04:55 136:02 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066879 20.1 0.1 23928356 3358612 ? Sl 04:55 115:27 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066911 21.0 0.1 23792432 3596812 ? Il 04:55 120:44 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066913 20.7 0.1 24435236 3628644 ? Sl 04:55 119:18 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066930 23.7 0.1 23690284 3371840 ? Il 04:55 136:12 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2066947 24.4 0.1 23663732 3517224 ? Il 04:55 140:38 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2067141 21.1 0.1 24832424 4074672 ? Sl 04:55 121:12 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2067143 19.4 0.1 24615160 3921920 ? Sl 04:55 111:36 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2067160 26.5 0.1 23807964 3456564 ? Il 04:55 152:21 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2067162 23.2 0.1 24068296 3503972 ? Il 04:55 133:17 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2067164 25.2 0.1 23698596 3346972 ? Il 04:55 144:42 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2067166 19.8 0.1 23898984 3456920 ? Il 04:55 114:14 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2067168 21.6 0.1 23833948 3637788 ? Il 04:55 124:05 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2067170 23.6 0.1 23952140 3511024 ? Rl 04:55 135:54 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 2067172 24.3 0.2 25158908 4555396 ? Rl 04:55 139:57 python -m tools.datasets.datautil /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean.csv --output /mnt/hdd/data/panda70m_by/raw/meta/split-2/meta_remove_corrupted_loadjson_noempty_clean_info.csv --format csv --video-info --fmin 1\n", - "zhaowan+ 3380048 10.0 0.0 14176 5312 ? Ss 14:29 0:00 bash -ic ps ux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "zhaowan+ 3381150 0.0 0.0 12124 2496 ? S 14:29 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "\n" - ] - } - ], - "source": [ - "# pkill(\"split-6\", \"h800-84\")\n", - "ps(\"h800-84\")" - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -682,170 +327,9 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "HOST: h800-80\n", - "COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " root 4838 0.0 0.0 29820 18308 ? Ss Apr08 0:01 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n", - "lisheng+ 1551124 0.0 0.0 3090356 185552 pts/9 Sl 11:48 0:02 /home/lishenggui/.conda/envs/opensora/bin/python /home/lishenggui/.conda/envs/opensora/bin/torchrun --master_addr 10.20.1.80 --master_port 29550 --nproc_per_node 8 --nnodes 8 --node_rank 0 /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 1565730 101 0.1 71931860 4155240 ? Ssl 11:48 377:44 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 1565747 101 0.4 71657092 8687376 ? Ssl 11:48 377:51 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 1565766 101 0.4 71851652 8879768 ? Rsl 11:48 377:36 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 1565775 101 0.4 71543296 8769772 ? Ssl 11:48 377:49 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 1565776 101 0.3 71417920 8448272 ? Ssl 11:48 377:38 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 1565777 101 0.3 71220924 8251132 ? Ssl 11:48 377:37 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 1565778 101 0.3 71397324 8431376 ? Ssl 11:48 377:47 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 1565779 101 0.4 71677852 8709120 ? Ssl 11:48 377:36 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "root 1569630 0.0 0.0 11152 580 ? S 18:01 0:00 sleep 30\n", - "zhaowan+ 1571020 4.5 0.0 8488 5184 ? Ss 18:01 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "zhaowan+ 1571252 0.0 0.0 6412 2356 ? S 18:01 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "root 2401123 0.0 0.0 193404 64564 ? S Apr10 0:00 /usr/bin/python3 /usr/local/bin/jupyter-lab --allow-root --ServerApp.base_url=lab\n", - "\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "\n", - "HOST: h800-81\n", - "COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " root 4828 0.0 0.0 29820 17552 ? Ss Mar19 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n", - "zhaowan+ 320182 0.2 0.0 1255960 460164 ? Sl 14:08 0:34 /home/zhaowangbo/.vscode-server/bin/1e790d77f81672c49be070e04474901747115651/node /home/zhaowangbo/.vscode-server/extensions/ms-python.vscode-pylance-2024.4.1/dist/server.bundle.js --cancellationReceive=file:ff42b41eb2644cbe0dab526c305f3ec2a5e3ca1694 --node-ipc --clientProcessId=294384\n", - "tom 382879 0.0 0.0 683308 47204 ? Sl Apr12 0:35 /home/zhaowangbo/.conda/envs/opensora/bin/python -m ipykernel_launcher --f=/home/tom/.local/share/jupyter/runtime/kernel-v2-362345kP0b0TEeW8vP.json\n", - "lisheng+ 672111 0.0 0.0 3016604 185384 ? S 11:48 0:01 /home/lishenggui/.conda/envs/opensora/bin/python /home/lishenggui/.conda/envs/opensora/bin/torchrun --master_addr 10.20.1.80 --master_port 29550 --nproc_per_node 8 --nnodes 8 --node_rank 1 /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 686018 101 0.3 70429568 7713964 ? Ssl 11:48 377:47 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 686019 101 0.3 70867092 7934756 ? Rsl 11:48 377:36 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 686026 101 0.3 70857736 7925428 ? Ssl 11:48 378:07 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 686037 101 0.4 71659836 8688480 ? Ssl 11:48 377:47 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 686038 101 0.4 71799792 8828624 ? Ssl 11:48 377:38 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 686039 101 0.4 71458224 8488680 ? Ssl 11:48 377:34 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 686042 101 0.4 71650732 8680932 ? Rsl 11:48 377:40 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "lisheng+ 686050 101 0.4 71747956 8778104 ? Ssl 11:48 377:39 /home/lishenggui/.conda/envs/opensora/bin/python -u /home/lishenggui/projects/sora/Open-Sora-dev/scripts/train.py configs/opensora-v1-1/train/video.py --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --wandb True --load /mnt/hdd/zangwei/opensora/outputs/789-STDiT2-XL-2/epoch1-global_step6500\n", - "zhaowan+ 3388150 0.4 0.0 129732 45044 ? Sl 18:00 0:00 /usr/bin/python3 /home/zhaowangbo/.vscode-server/extensions/ms-python.black-formatter-2024.2.0/bundled/tool/lsp_server.py --stdio\n", - "zhaowan+ 3388671 5.0 0.0 8492 5064 ? Ss 18:01 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "zhaowan+ 3388767 0.0 0.0 6412 2264 ? S 18:01 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "\n", - "HOST: h800-82\n", - "COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " root 4798 0.0 0.0 30076 18704 ? Ss Mar18 0:02 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n", - "zhaowan+ 1606637 5.0 0.0 8488 5292 ? Ss 18:01 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "zhaowan+ 1606730 0.0 0.0 6412 2360 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "\n", - "HOST: h800-83\n", - "COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " root 4804 0.0 0.0 35356 14248 ? Ss Mar21 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n", - "lisheng+ 1171411 2.2 0.0 1289796 482508 ? Sl 17:43 0:25 /home/lishenggui/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node /home/lishenggui/.vscode-server/extensions/ms-python.vscode-pylance-2024.4.1/dist/server.bundle.js --cancellationReceive=file:5946790779c12b9e336d5758f1b93c58461b5cde95 --node-ipc --clientProcessId=1169435\n", - "zhaowan+ 1206902 10.0 0.0 14176 5304 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "zhaowan+ 1207036 0.0 0.0 12124 2548 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "lisheng+ 3761623 0.1 0.0 1219948 387940 ? Sl 11:48 0:25 /home/lishenggui/.vscode-server/cli/servers/Stable-5c3e652f63e798a5ac2f31ffd0d863669328dc4c/server/node /home/lishenggui/.vscode-server/extensions/ms-python.vscode-pylance-2024.4.1/dist/server.bundle.js --cancellationReceive=file:cac8d8fe58263fb931aacdabe459d3cf363759b6a6 --node-ipc --clientProcessId=3735637\n", - "\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "\n", - "HOST: h800-84\n", - "COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " root 4731 0.0 0.0 35612 15176 ? Ss Mar19 0:02 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n", - "zhaowan+ 2207851 5.0 0.0 14176 5360 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "zhaowan+ 2207944 0.0 0.0 12124 2672 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "\n", - "HOST: h800-85\n", - "COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " root 4901 0.0 0.0 29820 18840 ? Ss Apr04 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n", - "zhaowan+ 2201547 0.0 0.0 8700 3260 pts/2 T Apr13 0:00 bash -ic nohup sleep 3m\n", - "root 2388479 0.0 0.0 5613836 24020 pts/5 Sl+ Apr13 1:29 docker run -it --gpus all --entrypoint=bash image.luchentech.com/base/opensora:1.0.0-ubuntu20.04-python3.10-torch2.2.1-cuda121\n", - "litianyi 3016261 6.7 0.0 275912 41260 pts/28 Sl+ 12:01 24:23 /home/share/software/miniconda3/build/bin/python /home/litianyi/.local/bin/nvitop\n", - "litianyi 3169865 0.2 0.0 5781128 292448 pts/26 Sl+ 15:17 0:23 /home/litianyi/.conda/envs/opensorawb/bin/python /home/litianyi/.conda/envs/opensorawb/bin/torchrun --standalone --nproc_per_node 1 scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3169959 216 0.5 136100876 12633772 ? Rsl 15:17 357:05 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3170857 1.5 0.1 80501332 2850552 ? Sl 15:18 2:32 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3170920 1.5 0.1 80689012 3038212 ? Sl 15:18 2:30 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3170983 1.5 0.1 80685000 3034264 ? Sl 15:18 2:29 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171046 1.5 0.1 80600804 2950032 ? Sl 15:18 2:31 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171109 1.4 0.1 80714872 3063928 ? Sl 15:18 2:26 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171172 1.5 0.1 80746208 3095252 ? Sl 15:18 2:29 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171235 1.5 0.1 80663276 3012272 ? Sl 15:18 2:31 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171298 1.5 0.1 80682696 3031880 ? Sl 15:18 2:30 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171361 1.5 0.1 80556108 2905152 ? Sl 15:18 2:33 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171424 1.5 0.1 80678164 3026948 ? Sl 15:18 2:28 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171487 1.5 0.1 80644056 2993252 ? Sl 15:18 2:29 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171550 1.5 0.1 80768108 3117204 ? Sl 15:18 2:28 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171613 1.5 0.1 80692264 3041440 ? Sl 15:18 2:30 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171676 1.5 0.1 80629512 2978456 ? Sl 15:18 2:33 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171739 1.5 0.1 80599356 2948260 ? Sl 15:18 2:32 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3171802 1.5 0.1 80815668 3164776 ? Sl 15:18 2:29 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora-v1-1/train/image_rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3317198 1.7 0.0 5782152 310412 pts/34 Sl+ 17:52 0:09 /home/litianyi/.conda/envs/opensorawb/bin/python /home/litianyi/.conda/envs/opensorawb/bin/torchrun --standalone --nproc_per_node 1 scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3317292 442 0.3 84751060 6609672 ? Ssl 17:52 42:48 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3318034 3.4 0.2 52835540 4631824 ? Sl 17:53 0:18 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3318097 3.4 0.2 52835552 4630380 ? Sl 17:53 0:18 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3318160 3.4 0.2 52835564 4629260 ? Sl 17:53 0:18 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3318223 3.3 0.2 52835576 4630124 ? Sl 17:53 0:17 /home/litianyi/.conda/envs/opensorawb/bin/python -u scripts/train.py configs/opensora/train/16x256x256-spee-rflow.py --data-path /home/zhaowangbo/data/csv/image_test.csv\n", - "litianyi 3323247 0.0 0.0 5476 580 ? S 17:59 0:00 sleep 180\n", - "zhaowan+ 3325310 10.0 0.0 8496 5284 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "litianyi 3325318 0.0 0.0 5476 516 ? S 18:02 0:00 sleep 1\n", - "litianyi 3325354 0.0 0.0 5476 520 ? S 18:02 0:00 sleep 1\n", - "zhaowan+ 3325418 0.0 0.0 6444 2620 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "\n", - "HOST: h800-86\n", - "COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " root 4893 0.0 0.0 35360 18904 ? Ss Apr02 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n", - "zhaowan+ 25146 10.0 0.0 14600 5740 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "zhaowan+ 25327 0.0 0.0 12148 2536 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "\n", - "==== STDERR ====\n", - " bash: 无法设定终端进程组(-1): 对设备不适当的 ioctl 操作\n", - "bash: 此 shell 中无任务控制\n", - "\n", - "HOST: h800-170\n", - "COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " root 4085 0.0 0.0 35768 19104 ? Ss Mar20 0:03 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n", - "zhaowan+ 2338486 5.0 0.0 14176 5428 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "zhaowan+ 2338580 0.0 0.0 12124 2708 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "\n", - "HOST: h800-171\n", - "COMMAND: bash -ic 'ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"'\n", - "==== STDOUT ====\n", - " root 4813 0.0 0.0 29676 18824 ? Ss Mar19 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers\n", - "zhaowan+ 838735 10.0 0.0 8496 5208 ? Ss 18:02 0:00 bash -ic ps aux | cat | grep --color=never -E \"python|sleep|torchrun|colossal\"\n", - "zhaowan+ 838828 0.0 0.0 6444 2612 ? S 18:02 0:00 grep --color=auto --color=never -E python|sleep|torchrun|colossal\n", - "\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "\n" - ] - } - ], + "outputs": [], "source": [ "ps()" ] @@ -877,7 +361,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ @@ -909,23 +393,14 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "cd /home/zhaowangbo/zangwei/opensora/ && colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py configs/opensora-v1-1/train/video.py --wandb True --data-path /home/zhaowangbo/data/csv/video_image_test_2.csv --load-path outputs/764-STDiT2-XL-2/epoch1-global_step6000\n" - ] - } - ], + "outputs": [], "source": [ - "host = \"h800-80\"\n", - "log_file = os.path.join(OPEN_SORA_HOME, \"logs/train_02.log\")\n", - "data_path = \"/home/zhaowangbo/data/csv/video_image_test_2.csv\"\n", - "ckpt_path = \"outputs/764-STDiT2-XL-2/epoch1-global_step6000\"\n", - "cmd = colossal_run(data_path, ckpt_path)\n", + "host = \"host-0\"\n", + "log_file = os.path.join(OPEN_SORA_HOME, \"logs/train.log\")\n", + "data_path = \"/path/to/meta.csv\"\n", + "cmd = colossal_run(data_path)\n", "print(cmd)" ] }, @@ -940,39 +415,18 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "HOST: h800-80\n", - "COMMAND: bash -ic 'cd /home/zhaowangbo/zangwei/opensora/ && cat hostfile | xargs -I \"{}\" ssh \"{}\" pkill -9 python'\n", - "==== STDERR ====\n", - " bash: cannot set terminal process group (-1): Inappropriate ioctl for device\n", - "bash: no job control in this shell\n", - "pkill: killing pid 382879 failed: Operation not permitted\n", - "\n" - ] - } - ], + "outputs": [], "source": [ "cmd = kill_all()\n", "run_command(cmd, host)" ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -986,9 +440,9 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.13" + "version": "3.9.18" } }, "nbformat": 4, - "nbformat_minor": 2 + "nbformat_minor": 4 } diff --git a/tools/scene_cut/convert_id_to_path.py b/tools/scene_cut/convert_id_to_path.py index 025cb7b..eb7b1cb 100644 --- a/tools/scene_cut/convert_id_to_path.py +++ b/tools/scene_cut/convert_id_to_path.py @@ -7,10 +7,12 @@ from functools import partial import numpy as np import pandas as pd from pandarallel import pandarallel -from tqdm import tqdm import cv2 from mmengine.logging import print_log from moviepy.editor import VideoFileClip +from tqdm import tqdm + +tqdm.pandas() def is_intact_video(video_path, mode="moviepy", verbose=False, logger=None): diff --git a/tools/scene_cut/cut.py b/tools/scene_cut/cut.py index 353e392..b3ecbe0 100644 --- a/tools/scene_cut/cut.py +++ b/tools/scene_cut/cut.py @@ -9,6 +9,9 @@ from imageio_ffmpeg import get_ffmpeg_exe from mmengine.logging import MMLogger, print_log from pandarallel import pandarallel from scenedetect import FrameTimecode +from tqdm import tqdm + +tqdm.pandas() def process_single_row(row, args, log_name=None): @@ -92,25 +95,18 @@ def split_video( # for the remaining calls. # cmd += ['-v', 'error'] + # clip to cut # -ss after -i is very slow; put -ss before -i - # input path - # cmd += ["-i", video_path] - - # clip to cut - # cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-t", str(duration.get_seconds())] - - # clip to cut cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-i", video_path, "-t", str(duration.get_seconds())] # target fps - # cmd += ['-vf', 'select=mod(n\,2)'] if target_fps is not None: cmd += ["-r", f"{target_fps}"] # aspect ratio if shorter_size is not None: cmd += ["-vf", f"scale='if(gt(iw,ih),-2,{shorter_size})':'if(gt(iw,ih),{shorter_size},-2)'"] - # cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"] + # cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"] cmd += ["-map", "0", save_path] diff --git a/tools/scene_cut/scene_detect.py b/tools/scene_cut/scene_detect.py index f3a6548..eb7b003 100644 --- a/tools/scene_cut/scene_detect.py +++ b/tools/scene_cut/scene_detect.py @@ -7,6 +7,8 @@ from pandarallel import pandarallel from scenedetect import AdaptiveDetector, detect from tqdm import tqdm +tqdm.pandas() + def process_single_row(row): # windows diff --git a/tools/scoring/README.md b/tools/scoring/README.md index 06ddd17..a944d5c 100644 --- a/tools/scoring/README.md +++ b/tools/scoring/README.md @@ -30,7 +30,7 @@ Then, run the following command. **Make sure** the meta file has column `path` ( ```bash torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference /path/to/meta.csv --bs 1024 --num_workers 16 ``` -This will generate multiple part files. Run `python -m tools.datasets.datautil /path/to/meta_part1.csv /path/to/meta_part2.csv` to merge these part files. +This will generate multiple part files, each corresponding to a node . Run `python -m tools.datasets.datautil /path/to/meta_aes_part*.csv --output /path/to/meta_aes.csv` to merge them. ## Optical Flow Score From e49affab242a55e96102dafe5025e71ba8cc6edf Mon Sep 17 00:00:00 2001 From: xyupeng <99191637+xyupeng@users.noreply.github.com> Date: Wed, 24 Apr 2024 15:56:22 +0800 Subject: [PATCH 3/3] Dev/pxy (#73) * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scoring/matching * update scene_cut * update scene_cut * update scene_cut[A * update scene_cut * update scene_cut * update scene_cut * update scene_cut * update scene_cut * update scene_cut * m * m * m * m * m * m * m * m * m * m * m * m * m * m * update readme * update readme * extract frames using opencv everywhere * extract frames using opencv everywhere * extract frames using opencv everywhere * filter panda10m * filter panda10m * m * m * m * m * m * m * m * m * m * m * m * m * m * m * m * m * m * ocr * add ocr * add main.sh * add ocr * add ocr * add ocr * add ocr * add ocr * add ocr * update scene_cut * update remove main.sh * update scoring * update scoring * update scoring * update README * update readme * update scene_cut * update readme * update scoring * update readme * update readme * update filter_panda10m * update readme * update readme * update launch.ipynb * update scene_cut * update scene_cut * update readme * update launch.ipynb * update readme --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 04cbe08..ab00ee4 100644 --- a/README.md +++ b/README.md @@ -250,7 +250,7 @@ To lower the memory usage, set a smaller `vae.micro_batch_size` in the config (s High-quality data is crucial for training good generation models. To this end, we establish a complete pipeline for data processing, which could seamlessly convert raw videos to high-quality video-text pairs. The pipeline is shown below. For detailed information, please refer to [data processing](docs/data_processing.md). -Also check out the [datasets](docs/data_processing.md) we use. +Also check out the [datasets](docs/datasets.md) we use. ![Data Processing Pipeline](assets/readme/report_data_pipeline.png)