Microsoft Unveils Phi-3.5 AI Model Family: Advanced MoE, Vision, and Mini Models Now Available

On Tuesday, Microsoft unveiled the Phi-3.5 artificial intelligence (AI) model family, which is the follow-up to the Phi-3 models unveiled in April. The Phi-3.5 Mixture of Experts (MoE), Phi-3.5 Vision, and Phi-3.5 Mini models are included in the latest edition. Because these are instructed models, users will need to provide explicit instructions for them to function as regular conversational AI and produce the necessary results. You can download the open-source AI models from the tech giant’s Hugging Face listings.

Weizhu Chen, a Microsoft executive, revealed the availability of the new AI models in a post on X (previously known as Twitter). The Phi-3.5 models are more capable than their predecessors, although they still use the same architecture, dataset, and training techniques. The AI model family now includes the MoE and Vision models in addition to the revised Mini model with language capability.

Also Read: Microsoft AI Tour 2024: Who and Why Should Attend? Find your city for Speakers and Sessions

Technical details: there are 3.8 billion parameters on the Phi-3.5 Mini. It makes use of a dense decoder-only transformer and the same tokenizer, a tool for segmenting text into smaller pieces. The model accepts a context window of 1,28,000 tokens and only accepts text as input. According to the business, between June and August, 3.4 trillion tokens were used to train it. The knowledge cut-off date is October 2023.

Arabic, Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Hebrew, Hungarian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish, and Ukrainian are just a few of the many new languages that this model now supports.

With 4.2 billion parameters, the Phi-3.5 Vision AI model can process information contained in images thanks to an image encoder. It takes images and text with the same context length as the Mini model as input. Its text knowledge cutoff date is March, and it was trained on 500 billion tokens of data between July and August.

Lastly, there are 16×3.8 billion parameters in the Phi-3.5 MoE AI model. But when two experts are used, just 6.6 billion of them become active parameters. Notably, MoE is a technique that enhances the accuracy and efficiency of the model by training multiple models (experts) individually and then combining them. This model has an October 2023 knowledge cutoff date and was trained on 4.9 trillion tokens of data between April and August.

Microsoft released the benchmark scores for each model. The Phi-3.5 MoE beats the Gemini 1.5 Flash and GPT-4o small in the SQuALITY benchmark, which evaluates readability and accuracy in summarizing lengthy text passages. This evaluates the AI model’s extended context window.

It should be noted, though, that this is not a fair comparison because MoE models have a different design and need more advanced technology and storage to function. In addition, the Phi-3.5 Mini and Vision models have surpassed pertinent rival AI models in the same market sector in a few different measures.

Also Read: Microsoft Designer App: How to Use the AI Image Generator Tool for Editing and Creation?

Hugging Face listings provide access to the Phi-3.5 AI models for those who would like to test them out. According to Microsoft, consumers running these devices will need to use sophisticated GPUs because they use flash attention. Utilizing GPUs from Nvidia A100, A6000, and H100, the company has evaluated them.