dotAI 2024 - Armand Joulin - Improving open language models at a practical size

Filmed at dotAI on October 18, 2024 in Paris. More about the conference on https://www.dotai.io Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranges in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture and train the 2B and 9B models with an alternative to next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3× bigger. All models are released to the community. Who is Armand Joulin? Armand is a Research Director at Google DeepMind, in charge of the open version of Gemini called Gemma. Prior to this, he was a Research Director at Meta in charge of EMEA, where he supervised several key open projects such as LLaMA, DINO or fasttext.