Instructions to use stockmark/stockmark-13b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use stockmark/stockmark-13b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="stockmark/stockmark-13b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("stockmark/stockmark-13b")
model = AutoModelForCausalLM.from_pretrained("stockmark/stockmark-13b")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use stockmark/stockmark-13b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "stockmark/stockmark-13b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stockmark/stockmark-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/stockmark/stockmark-13b

SGLang

How to use stockmark/stockmark-13b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "stockmark/stockmark-13b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stockmark/stockmark-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "stockmark/stockmark-13b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stockmark/stockmark-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use stockmark/stockmark-13b with Docker Model Runner:
```
docker model run hf.co/stockmark/stockmark-13b
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

stockmark/stockmark-13b

Stockmark-13b is a 13 billion parameter LLM pretrained from scratch based on Japanese corpus of about 220B tokens. This model is developed by Stockmark Inc.

Please see our blog for more details.

This project is supported by AWS LLM development support program.

We also provide stockmark-13b-instruct, which is the instruction tuned version of stockmark-13b.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# For A100 or H100 GPU
model = AutoModelForCausalLM.from_pretrained("stockmark/stockmark-13b", device_map="auto", torch_dtype=torch.bfloat16)

# If you use a T4 or V100 GPU, please load a model in 8 bit with the below code.
# To do so, you need to install `bitsandbytes` via `pip install bitsandbytes`.
# model = AutoModelForCausalLM.from_pretrained("stockmark/stockmark-13b", device_map={"": 0}, load_in_8bit=True)

tokenizer = AutoTokenizer.from_pretrained("stockmark/stockmark-13b")

inputs = tokenizer("自然言語処理とは", return_tensors="pt").to(model.device)
with torch.no_grad():
    tokens = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.7
    )
    
output = tokenizer.decode(tokens[0], skip_special_tokens=True)
print(output)

Examples:

LoRA tuning: https://huggingface.co/stockmark/stockmark-13b/blob/main/notebooks/LoRA.ipynb

Training dataset

We have used Japanese corpus of total of about 220 billion tokens.

corpus	tokens after preprocessing
Stockmark Web Corpus (This dataset will not be released)	9.1 billion
Patent	34.8 billion
Wikipedia	1.0 billion
CC100	10.9 billion
mC4	53.2 billion
CommonCrawl (snapshot: 2023-23, 2022-49, 2022-21, 2021-21)	112.9 billion