Instructions to use Nexusflow/Starling-LM-7B-beta with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Nexusflow/Starling-LM-7B-beta with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Nexusflow/Starling-LM-7B-beta")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Nexusflow/Starling-LM-7B-beta")
model = AutoModelForCausalLM.from_pretrained("Nexusflow/Starling-LM-7B-beta")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Nexusflow/Starling-LM-7B-beta with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nexusflow/Starling-LM-7B-beta"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nexusflow/Starling-LM-7B-beta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Nexusflow/Starling-LM-7B-beta

SGLang

How to use Nexusflow/Starling-LM-7B-beta with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nexusflow/Starling-LM-7B-beta" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nexusflow/Starling-LM-7B-beta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nexusflow/Starling-LM-7B-beta" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nexusflow/Starling-LM-7B-beta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Nexusflow/Starling-LM-7B-beta with Docker Model Runner:
```
docker model run hf.co/Nexusflow/Starling-LM-7B-beta
```

Lack of a System Prompt

#11

by Metricon - opened Apr 1, 2024

Discussion

Metricon

Apr 1, 2024

This model shows remarkable reasoning abilities for such a small parameter count. However, the lack of a usable System Prompt hurts it's performance as it does not strongly follow direction. A temporary work around is a one-shot entry and pseudo reply as a first message, but it's effectiveness is mild at best. In future iterations, I would recommend adding a formal System Prompt with additional adherence training.

banghua

Nexusflow org Apr 2, 2024

Thank you for the suggestion! We will try to include the system prompt in the future version.

John0007

Apr 4, 2024

Model is a beast in terms of perfomance and creativity but big problem is like Metricon suggested System Prompt following and ultra censored nature.Its really hard to do even basic stuff because model would continuosly bugging you with ethics and morals.
Same as Chat GPT4 thats why i dont like to use it.
If you deal with censoring but with the same capatibility it would top any other models currently availabe even 70B ones,great job!

banghua

Nexusflow org Apr 4, 2024

Thank you! This is really helpful feedback. We will try to incorporate those in the future!

karthik-ganesan-nexusflow

Apr 4, 2024

Thank you for the feedback @John0007 , could you share cases where its performance is very good and cases where its trying to act censored ? This will be very helpful for us to bake in these

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment