Instructions to use TheBloke/BigTranslate-13B-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/BigTranslate-13B-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TheBloke/BigTranslate-13B-GPTQ")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("TheBloke/BigTranslate-13B-GPTQ")
model = AutoModelForCausalLM.from_pretrained("TheBloke/BigTranslate-13B-GPTQ", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use TheBloke/BigTranslate-13B-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBloke/BigTranslate-13B-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/BigTranslate-13B-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TheBloke/BigTranslate-13B-GPTQ

SGLang

How to use TheBloke/BigTranslate-13B-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheBloke/BigTranslate-13B-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/BigTranslate-13B-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheBloke/BigTranslate-13B-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/BigTranslate-13B-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use TheBloke/BigTranslate-13B-GPTQ with Docker Model Runner:
```
docker model run hf.co/TheBloke/BigTranslate-13B-GPTQ
```

Weird responses in Oobabooga

by Satya93 - opened Aug 1, 2023

Discussion

Satya93

Aug 1, 2023

Using the Alpaca template, like in the model card, but this is providing odd output.

Input: "Translate the following from Russian into English: Дабы апнуть форум , Оживить народ, Надеюсь морально поддержать админов И так Первое видело Наверное многие сталкивались с этим WARNING: Detected AP<....>"

Output: "Translate the following from German into English: Warum die Menschen so aufgeregt sind braucht ich nicht zu"

Another instance it provided English, but nothing resembling the input of Russian.

Lolkid654

Aug 1, 2023

Yhea this model was giving me issues in Oobabooga too.

vractal

Aug 3, 2023

Hey did anyone manage to get this working? It gives weird output, either this one on ooba or the ggml in kobold, same issue

TheBloke

Owner Aug 5, 2023

Ah OK, maybe it doesn't work well. I only tested it very briefly from a Python script, and it seemed to translate English to French and French to English OK. But I didn't test it much

vractal

Aug 9, 2023

Hmm actually it seems the main issue is that instruction template must be in Chinese, with proper template I managed to get more useful responses out of it

vractal

Aug 9, 2023

•

edited Aug 9, 2023

Something like this:

以下是一个描述任务的指令，请写一个完成该指令的适当回复
### 指令:
{prompt}
### 回复

There are some specific prompts for translating to lang a -> b from what I saw in the original repo inference code, but this one seems to help

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment