Meta AI develops compact language model for mobile devices


We want to hear from you! Take our quick AI survey and share your insights on the current state of AI, how you’re implementing it, and what you expect to see in the future. Learn More


Meta AI researchers have unveiled MobileLLM, a new approach to creating efficient language models designed for smartphones and other resource-constrained devices. Published on June 27, 2024, this work challenges assumptions about the necessary size of effective AI models.

The research team, comprising members from Meta Reality Labs, PyTorch and Meta AI Research (FAIR), focused on optimizing models with fewer than 1 billion parameters. This is a fraction of the size of models like GPT-4, which are estimated to have over a trillion parameters.

Yann LeCun, Meta’s Chief AI Scientist, highlighted key aspects of the research on X (formerly known as Twitter):

AD 4nXdLDIY g x884cPT67KAknSyiXnL6yTau 7jmY4uuE4wYPz2YIxv2Lb6Ksoo4GnyfXHExfFZgvtWv9mYTroCx9WIKGtvt8CqUDNnNXudaaGm6bSBP6bVqwai SnwLeF PV3duXSRq8LDF1vmguLhcqJzBDS?key=srF 1aFd

Countdown to VB Transform 2024

Join enterprise leaders in San Francisco from July 9 to 11 for our flagship AI event. Connect with peers, explore the opportunities and challenges of Generative AI, and learn how to integrate AI applications into your industry. Register Now


Key innovations in MobileLLM include:

  1. Prioritizing model depth over width
  2. Implementing embedding sharing and grouped-query attention
  3. Utilizing a novel immediate block-wise weight-sharing technique

These design choices allowed MobileLLM to outperform previous models of similar size by 2.7% to 4.3% on common benchmark tasks. While these single-digit improvements may seem small, they represent meaningful progress in the highly competitive field of language model development.

Notably, the 350 million parameter version of MobileLLM demonstrated comparable accuracy to the much larger 7 billion parameter LLaMA-2 model on certain API calling tasks. This suggests that for some specific applications, more compact models might offer similar functionality while using significantly fewer computational resources.

image 2a1cde
Credit: “MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases”, Zechun Liu et. al, Meta

MobileLLM’s development aligns with a growing interest in more efficient AI models. As progress in very large language models shows signs of slowing, researchers are increasingly exploring the potential of more compact, specialized designs. The focus on efficiency and on-device deployment puts MobileLLM in a similar category to what some researchers call Small Language Models (SLMs), despite the “LLM” in its name.

While MobileLLM is not yet available for public use, Meta has open-sourced the pre-training code, allowing other researchers to build on their work. As this technology develops, it may enable more advanced AI features on personal devices, though the timeline and exact capabilities remain uncertain.

The development of MobileLLM represents an important step in making advanced AI more accessible and sustainable. It challenges the notion that effective language models must be enormous, potentially opening new avenues for AI applications on personal devices.



Source link

About The Author

Scroll to Top