nicolabs/Open-Sora
1
0
Fork 0
mirror of https://github.com/hpcaitech/Open-Sora.git synced 2026-05-21 11:59:01 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

update llama3 readme

Browse source
This commit is contained in:
Tom Young 2024-06-17 06:31:34 +00:00
parent 69b9dd477e
commit 14988e070c
1 changed files with 14 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
tools/caption/README.md
View file

@ -198,3 +198,17 @@ Each video is classified according to 8 categories:
static,
unclassified`.
Categories of `tilt`, `pan` and `zoom` can overlap with each other.
## Tagging with Llama3
To understand the overall category distribution of our training dataset, we use Llama3 to generate tags based on the video captions.
After obtaining Llama3 usage permission from huggingface/meta, you may generate tags based on the captions using Llama3 like this:
```bash
torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llama3 meta.csv --key objects --output_prefix meta
```
This will generate tags based on the `text` column of `meta.csv` and put the results to `output_prefix + key.csv`. Currently the prompts for `objects` and `actions` are supported.