Lightweight multilingual LLM powering AI applications in latency bound, memory/compute constrained environments
Description | Phi-3.5-mini is a lightweight, state-of-the-art open model built upon datasets used for Phi-3 - synthetic data and filtered publicly available websites - with a focus on very high-quality, reasoning dense data. The model belongs to the Phi-3 model family and supports 128K token context length. The model underwent a rigorous enhancement process, incorporating both supervised fine-tuning, proximal policy optimization, and direct preference optimization to ensure precise instruction adherence and robust safety measures. This model is ready for commercial and research use. |
Release date | August 20, 2024 |
License | MIT |
Architecture | Phi-3.5-mini has 3.8B parameters and is a dense decoder-only Transformer model using the same tokenizer as Phi-3 Mini. |
Inputs | Text. It is best suited for prompts using chat format. |
Context length | 128K tokens |
Outputs | Generated text (String) in response to the input |
Status | This is a static model trained on an offline dataset with cutoff date October 2023 for publicly available data. Future versions of the tuned models may be released as we improve models. |
Supported languages | Arabic, Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Hebrew, Hungarian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish, Ukrainian |
Nothing contained in this Model Card should be interpreted as or deemed a restriction or modification to the license the model is released under.
This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party’s requirements for this application and use case.
🏡 Phi-3 Portal
📰 Phi-3 Microsoft Blog
📖 Phi-3 Technical Report
🛠️ Phi-3 on Azure AI Studio
👩🍳 Phi-3 Cookbook
This is an update over the June 2024 instruction-tuned Phi-3-mini release based on valuable user feedback. The model used better post-training techniques and additional data leading to substantial gains on multilingual, multi-turn conversation quality, and reasoning capability. We believe most use cases will benefit from this release, but we encourage users to test in their particular AI applications. We appreciate the enthusiastic adoption of the Phi-3 model family, and continue to welcome all feedback from the community.
The model is intended for broad commercial and research use in multiple languages. The model provides uses for general purpose AI systems and applications which require:
Our model is designed to accelerate research on language and multimodal models, for use as a building block for generative AI powered features.
Our models are not specifically designed or evaluated for all downstream purposes. Developers should consider common limitations of language models, as well as performance difference across languages, as they select use cases, and evaluate and mitigate for accuracy, safety, and fairness before using within a specific downstream use case, particularly for high-risk scenarios. Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case.
Given the nature of the training data, the Phi-3.5-mini-instruct model is best suited for prompts using the chat format as follows:
<|system|> You are a helpful assistant.<|end|> <|user|> How to explain Internet for a medieval knight?<|end|> <|assistant|>
After obtaining the Phi-3.5-mini-instruct model checkpoints, users can use this sample code for inference.
import torch from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline torch.random.manual_seed(0) model = AutoModelForCausalLM.from_pretrained( "microsoft/Phi-3.5-mini-instruct", device_map="cuda", torch_dtype="auto", trust_remote_code=True, ) tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3.5-mini-instruct") messages = [ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"}, {"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."}, {"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"}, ] pipe = pipeline( "text-generation", model=model, tokenizer=tokenizer, ) generation_args = { "max_new_tokens": 500, "return_full_text": False, "temperature": 0.0, "do_sample": False, } output = pipe(messages, **generation_args) print(output[0]['generated_text'])
[Use unique identifier for identifying the model in the future- this may be part of our internal naming, specifying variation like "pruned," "unpruned," "trained," or "deployable" or "quantized" where necessary and including a versioning number like vX.X along with short description differentiating if multiple versions are available]
Dates | Trained between June and August 2024 |
Training time | 10 days |
Training data | 3.4T tokens |
Our training data includes a wide variety of sources, totaling 3.4 trillion tokens, and is a combination of
We are focusing on the quality of data that could potentially improve the reasoning ability for the model, and we filter the publicly available documents to contain the correct level of knowledge. As an example, the result of a game in premier league in a particular day might be good training data for frontier models, but we need to remove such information to leave more model capacity for reasoning for the small size models. More details about data can be found in the Phi-3 Technical Report.
The table below highlights multilingual capability of Phi-3.5-mini on multilingual MMLU, MEGA, and multilingual MMLU-pro datasets. Overall, we observed that even with just 3.8B active parameters, the model is very competitive on multilingual tasks in comparison to other models with a much bigger active parameters.
Benchmark | Phi-3.5-mini-instruct | Phi-3.0-mini-128k-instruct | Mistral-7B-Instruct-v0.3 | Mistral-Nemo-12B-Ins-2407 | Llama-3.1-8B-Ins | Gemma-2-9B-Ins | Gemini-1.5-Flash | GPT-4o-mini-2024-07-18 (Chat) |
---|---|---|---|---|---|---|---|---|
Multilingual MMLU | 55.4 | 51.08 | 47.4 | 58.9 | 56.2 | 63.8 | 77.2 | 72.9 |
Multilingual MMLU-Pro | 30.9 | 30.21 | 15.0 | 34.0 | 21.4 | 43.0 | 57.9 | 53.2 |
MGSM | 47.9 | 41.56 | 31.8 | 63.3 | 56.7 | 75.1 | 75.8 | 81.7 |
MEGA MLQA | 61.7 | 55.5 | 43.9 | 61.2 | 45.2 | 54.4 | 61.6 | 70.0 |
MEGA TyDi QA | 62.2 | 55.9 | 54.0 | 63.7 | 54.5 | 65.6 | 63.6 | 81.8 |
MEGA UDPOS | 46.5 | 48.1 | 57.2 | 58.2 | 54.1 | 56.6 | 62.4 | 66.0 |
MEGA XCOPA | 63.1 | 62.4 | 58.8 | 10.8 | 21.1 | 31.2 | 95.0 | 90.3 |
MEGA XStoryCloze | 73.5 | 73.6 | 75.5 | 92.3 | 71.0 | 87.0 | 20.7 | 96.6 |
Average | 55.2 | 52.3 | 47.9 | 55.3 | 47.5 | 59.6 | 64.3 | 76.6 |
Phi-3.5-mini supports 128K context length, therefore the model is capable of several long context tasks including long document/meeting summarization, long document QA, long document information retrieval. Phi-3.5-mini outperforms Gemma-2 family which only supports 8K context length and is competitive with other much larger open-weight models such as Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Mistral-Nemo-12B-Instruct-2407.
Benchmark | Phi-3.5-mini-instruct | Llama-3.1-8B-instruct | Mistral-7B-instruct-v0.3 | Mistral-Nemo-12B-instruct-2407 | Gemini-1.5-Flash | GPT-4o-mini-2024-07-18 (Chat) |
---|---|---|---|---|---|---|
GovReport | 25.9 | 25.1 | 26.0 | 25.6 | 27.8 | 24.8 |
QMSum | 21.3 | 21.6 | 21.3 | 22.1 | 24.0 | 21.7 |
Qasper | 41.9 | 37.2 | 31.4 | 30.7 | 43.5 | 39.8 |
SQuALITY | 25.3 | 26.2 | 25.9 | 25.8 | 23.5 | 23.8 |
SummScreenFD | 16.0 | 17.6 | 17.5 | 18.2 | 16.3 | 17.0 |
Average | 26.1 | 25.5 | 24.4 | 24.5 | 27.0 | 25.4 |
RULER: a retrieval-based benchmark for long context understanding
Model | 4K | 8K | 16K | 32K | 64K | 128K | Average |
---|---|---|---|---|---|---|---|
Phi-3.5-mini-instruct | 94.3 | 91.1 | 90.7 | 87.1 | 78.0 | 63.6 | 84.1 |
Llama-3.1-8B-instruct | 95.5 | 93.8 | 91.6 | 87.4 | 84.7 | 77.0 | 88.3 |
Mistral-Nemo-12B-instruct-2407 | 87.8 | 87.2 | 87.7 | 69.0 | 46.8 | 19.0 | 66.2 |
RepoQA: a benchmark for long context code understanding
Model | Python | C++ | Rust | Java | TypeScript | Average |
---|---|---|---|---|---|---|
Phi-3.5-mini-instruct | 86 | 67 | 73 | 77 | 82 | 77 |
Llama-3.1-8B-instruct | 80 | 65 | 73 | 76 | 63 | 71 |
Mistral-7B-instruct-v0.3 | 61 | 57 | 51 | 61 | 80 | 62 |
Engine: Tensor(RT)
Test Hardware [Name the specific test hardware model]:
Like other models, the Phi family of models can potentially behave in ways that are unfair, unreliable, or offensive. Some of the limiting behaviors to be aware of include:
Developers should apply responsible AI best practices and are responsible for ensuring that a specific use case complies with relevant laws and regulations (e.g. privacy, trade, etc.). Important areas for consideration include:
Please report security vulnerabilities or NVIDIA AI Concerns here.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft’s Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies.