Instructions to use microsoft/Phi-3.5-MoE-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Phi-3.5-MoE-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="microsoft/Phi-3.5-MoE-instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3.5-MoE-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3.5-MoE-instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use microsoft/Phi-3.5-MoE-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Phi-3.5-MoE-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3.5-MoE-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/microsoft/Phi-3.5-MoE-instruct

SGLang

How to use microsoft/Phi-3.5-MoE-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Phi-3.5-MoE-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3.5-MoE-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Phi-3.5-MoE-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3.5-MoE-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use microsoft/Phi-3.5-MoE-instruct with Docker Model Runner:
```
docker model run hf.co/microsoft/Phi-3.5-MoE-instruct
```

The model 'PhiMoEForCausalLM' is not supported for text-generation. Supported models are ['BartForCausalLM', 'BertLMHeadModel', .......

#34

by xxbadarxx - opened Aug 27, 2024

Discussion

xxbadarxx

Aug 27, 2024

I have installed transformers 4.43.0 according to your requirement

ykim362

Microsoft org Aug 27, 2024

•

edited Aug 27, 2024

Hi Thanks for your interest. Have you added 'trust_remote_code=True' in your load code?

e. g.

model = AutoModelForCausalLM.from_pretrained(
"microsoft/Phi-3.5-MoE-instruct",
device_map="cuda",
torch_dtype="auto",
trust_remote_code=True,
)

xxbadarxx

Aug 28, 2024

This comment has been hidden

xxbadarxx

Aug 28, 2024

yes I have added it .The difference is that I loaded the model from a local path (Does this have any impact?)

Anyway, it works, but there's the warning output "The model 'PhiMoEForCausalLM' is not supported for text-generation. ". I'm not sure if it will have any impact.

gargamit

Microsoft org Aug 28, 2024

@xxbadarxx Loading the model from a local path is perfectly fine and has no impact. As for the text-generation warning, I couldn’t repro it. It doesn’t affect the quality, so you can disregard the warning.

goodasdgood

Aug 29, 2024

Can you convert it to gguf?

goodasdgood

Aug 29, 2024

1]:
import torch
from transformers import AutoModel For CausalLM, AutoTokenizer, pipeline
torch.random.manual_seed(0)
model AutoModel For CausalLM.from_pretrained(
"/ /Phi-3.5-MoE-instruct",
device_map="cuda",
torch_dtype="auto",
trust_remote_code=True,
)
tokenizer AutoTokenizer.from_pretrained("/_
/Phi-3.5-MoE-instruct")

goodasdgood

Aug 29, 2024

config.json: 100%
4.53k/4.53k [00:00<00:00, 160kB/s]
configuration_phimoe.py: 100%
12.3k/12.3k [00:00<00:00, 575kB/s]
A new version of the following files was downloaded from https://huggingface.co/microsoft/Phi-3.5-MoE-instruct:
- configuration_phimoe.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
modeling_phimoe.py: 100%
80.5k/80.5k [00:00<00:00, 3.19MB/s]

ImportError Traceback (most recent call last)
in <cell line: 6>()
4 torch.random.manual_seed(0)
5
----> 6 model = AutoModelForCausalLM.from_pretrained(
7 "microsoft/Phi-3.5-MoE-instruct",
8 device_map="cuda",

3 frames
/usr/local/lib/python3.10/dist-packages/transformers/dynamic_module_utils.py in check_imports(filename)
180
181 if len(missing_packages) > 0:
--> 182 raise ImportError(
183 "This modeling file requires the following packages that were not found in your environment: "
184 f"{', '.join(missing_packages)}. Run pip install {' '.join(missing_packages)}"

ImportError: This modeling file requires the following packages that were not found in your environment: flash_attn. Run pip install flash_attn

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.

goodasdgood

Aug 29, 2024

I could never download the flash_attn and I could not download the form

colab tpu

Software
PyTorch
Transformers
Flash-Attention
Hardware
Note that by default, the Phi-3.5-MoE-instruct model uses flash attention, which requires certain types of GPU hardware to run. We have tested on the following GPU types:

NVIDIA A100
NVIDIA A6000
NVIDIA H100

Se-Hun

Aug 29, 2024

This comment has been hidden

xxbadarxx

Aug 30, 2024

@xxbadarxx Loading the model from a local path is perfectly fine and has no impact. As for the text-generation warning, I couldn’t repro it. It doesn’t affect the quality, so you can disregard the warning.

OK, Thank you for your reply! 😊

ArchSid

Sep 16, 2024

•

edited Sep 16, 2024

I could never download the flash_attn and I could not download the form

colab tpu

Software
PyTorch
Transformers
Flash-Attention
Hardware
Note that by default, the Phi-3.5-MoE-instruct model uses flash attention, which requires certain types of GPU hardware to run. We have tested on the following GPU types:

NVIDIA A100
NVIDIA A6000
NVIDIA H100

I have the same issue. Is any solution found? i tested with

NVIDIA A40
NVIDIA RTX A5000

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

The model 'PhiMoEForCausalLM' is not supported for text-generation. Supported models are ['BartForCausalLM', 'BertLMHeadModel', .......

To view examples of installing some common dependencies, click the"Open Examples" button below.

To view examples of installing some common dependencies, click the
"Open Examples" button below.