README.md

metadata

tags:
  - test
  - tiktoken
  - tokenizer

Llama 3 TikToken Tokenizer (Test Model)

This is a test model for hf-internal-testing containing only the tiktoken tokenizer files from meta-llama/Meta-Llama-3-8B-Instruct.

This model contains the tiktoken format tokenizer.model file for testing purposes.

Files

• tiktoken.model - TikToken format tokenizer (renamed from tokenizer.model)
• tokenizer_config.json - Tokenizer configuration
• special_tokens_map.json - Special tokens mapping

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama3-tokenizer-tiktoken")

Source

This is a subset of files from meta-llama/Meta-Llama-3-8B-Instruct, specifically from the original/ subfolder.