hf-internal-testing
/

llama3-tokenizer-tiktoken

Model card Files Files and versions

llama3-tokenizer-tiktoken

2.19 MB

1 contributor

History: 4 commits

itazap's picture

itazap HF Staff

Delete tokenizer.model

a1d81e8 verified 3 days ago

.gitattributes

1.52 kB

initial commit 3 days ago
README.md

845 Bytes

Add tiktoken tokenizer files from Meta-Llama-3-8B-Instruct/original 3 days ago
tiktoken.model

2.18 MB
xet

Add tiktoken tokenizer files from Meta-Llama-3-8B-Instruct/original 3 days ago