Instructions to use sirmyrrh/Kyllima-34B-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sirmyrrh/Kyllima-34B-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="sirmyrrh/Kyllima-34B-v1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("sirmyrrh/Kyllima-34B-v1")
model = AutoModelForCausalLM.from_pretrained("sirmyrrh/Kyllima-34B-v1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sirmyrrh/Kyllima-34B-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sirmyrrh/Kyllima-34B-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sirmyrrh/Kyllima-34B-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/sirmyrrh/Kyllima-34B-v1

SGLang

How to use sirmyrrh/Kyllima-34B-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sirmyrrh/Kyllima-34B-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sirmyrrh/Kyllima-34B-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sirmyrrh/Kyllima-34B-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sirmyrrh/Kyllima-34B-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use sirmyrrh/Kyllima-34B-v1 with Docker Model Runner:
```
docker model run hf.co/sirmyrrh/Kyllima-34B-v1
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Kyllima 34B v1

Model Details

This is a simple 50/50 merge of two of my favorite Yi 34B-based models for roleplay and creative writing, created using mergekit on Arcee.ai. There's a good amount of Nous Capybara 34B in here, some Bagel DPO, Lima RP v3, and other goodness. Less sloppy thanks to Kyllene. 200k context. Uncensored.

Use with the metadata prompt format, Alpaca-LimaRP, or Vicuna.

Recommended settings: 0.8-1 temp, 1.1-1.2 rep pen, 1 top P, 0.05 min P, 40 top K.

Add </s> to stop strings, and /n {{user}} or [INST] if necessary.

I use a slightly modified version of RisuAI's default system prompt with good results. I suggest adding a couple lines to the system prompt telling the model to write in complete sentences, and NOT to write prompts to itself.

It's sensitive to small changes in settings and to the style/format of your own writing.

The original upload had a broken tokenizer. If you downloaded before 10/2/24, please re-download.

Static GGUF available here or here. Imatrix GGUF available here. With thanks to mradermacher.

Merge Method

This model was merged using the TIES merge method using TeeZee/Kyllene-34B-v1.1 as a base.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

base_model: TeeZee/Kyllene-34B-v1.1
chat_template: auto
dtype: float16
merge_method: ties
models:
- model: TeeZee/Kyllene-34B-v1.1
  parameters:
    density: 0.5
    weight: 0.5
- model: Doctor-Shotgun/Nous-Capybara-limarpv3-34B
  parameters:
    density: 0.5
    weight: 0.5
parameters:
  embed_slerp: true
  int8_mask: true
  normalize: false
tokenizer_source: base

Downloads last month: 13

Safetensors

Model size

34B params

Tensor type

F16

Model tree for sirmyrrh/Kyllima-34B-v1

DS-Archive/Nous-Capybara-limarpv3-34B

TeeZee/Kyllene-34B-v1.1

Merge model

this model

Quantizations

3 models

Paper for sirmyrrh/Kyllima-34B-v1

Resolving Interference When Merging Models

Paper • 2306.01708 • Published Jun 2, 2023 • 19