Microsoft has launched phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5.
According to the technical report released by Microsoft, the Phi-3 model, despite being small enough to be deployed on a phone,. The model is also further aligned for robustness, safety, and chat format.
Microsoft has also introduced Phi-3-Small and Phi-3-Medium models, both significantly more capable than Phi-3-Mini. Phi-3-Small, with 7 billion parameters, utilizes the tiktoken tokenizer for improved multilingual tokenization. It boasts a vocabulary size of 100,352 and a default context length of 8K.
Also Read: Microsoft AI Video App Makes Mona Lisa Painting Sing; Video Gets Over 7 Million Views
What are the distinguished features of Microsoft’s Phi-3 mini model?
- Phi-3-Mini achieves 69% on the MMLU benchmark and 8.38 on the MT-bench, making it suitable for deployment on mobile phones.
- Phi-3-Mini’s small size allows it to be quantized to 4 bits, occupying approximately 1.8GB of memory. Microsoft tested the quantized model by deploying Phi-3-Mini on an iPhone 14 with an A16 Bionic chip, running natively on the device and fully offline, achieving more than 12 tokens per second.
- The innovation behind Phi-3-Mini lies in its training dataset, an expanded version of the one used for its predecessor, Phi-2. This dataset comprises heavily filtered web data and synthetic data. The model has also been optimized for robustness, safety, and chat format.
- The Phi-3-small 7 billion parameter model achieves an MMLU score of 75.3 and outperforms Meta’s recently launched Llama 3 8B Instruct with a score of 66.
- The model follows the standard decoder architecture of a 7B model class, featuring 32 layers and a hidden size of 4096. To minimize the KV cache footprint, Phi-3-Small employs grouped-query attention, with four queries sharing one key.
- Phi-3 mini utilizes alternative layers of dense attention and a novel block sparse attention to optimize KV cache savings while maintaining long context retrieval performance.
Also Read: Microsoft’s $2.9B Boost for Japan’s AI and Cloud Capabilities
Is Phi-3 Mini Safe to Use?
Phi-3-mini was developed following Microsoft’s responsible AI principles. The overall approach consisted of safety alignment in post-training, red-teaming, automated testing, and evaluations across dozens of RAI harm categories. Helpfulness and harmlessness preference datasets, along with modifications and multiple in-house generated datasets, were leveraged to address the RAI harm categories in safety post-training.
An independent red team at Microsoft iteratively examined phi-3-mini further to identify areas of improvement during the post-training process. Based on their feedback, Microsoft curated additional datasets tailored to address their insights, thereby refining the post-training dataset. This process resulted in a significant decrease in harmful response rates
Also Read: Mustafa Suleyman announces new Microsoft AI hub in London
What are the limitations of the Phi-3 Mini Model?
Phi-3 mini demonstrates a similar level of language understanding and reasoning ability as much larger models, but its size fundamentally limits it for certain tasks. For example, it cannot store extensive “factual knowledge,” resulting in lower performance on tasks such as TriviaQA.
Microsoft believes such weaknesses can be addressed by augmenting the model with a search engine. Additionally, the model’s language capabilities are mostly restricted to English, highlighting the need to explore multilingual capabilities for Small Language Models.
As per the reports, Microsoft said, Phi-3-mini will be available immediately on Microsoft cloud service platform Azure’s AI model catalog, machine learning model platform Hugging Face, and Ollama, a framework for running models on a local machine.
