---
tags:
- test
- tiktoken
- tokenizer
---

# Llama 3 TikToken Tokenizer (Test Model)

This is a test model for `hf-internal-testing` containing only the tiktoken tokenizer files from `meta-llama/Meta-Llama-3-8B-Instruct`.

This model contains the tiktoken format tokenizer.model file for testing purposes.

## Files

- `tiktoken.model` - TikToken format tokenizer (renamed from tokenizer.model)
- `tokenizer_config.json` - Tokenizer configuration
- `special_tokens_map.json` - Special tokens mapping

## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama3-tokenizer-tiktoken")
```

## Source

This is a subset of files from [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), specifically from the `original/` subfolder.