dotAI 2024 - Yann Léger - Serverless inferencing: an infrastructure point of view

Filmed at dotAI on October 17, 2024 in Paris. More about the conference on https://www.dotai.io Today, AI Infrastructure doesn’t rhyme with efficiency. Massive investments are made in GPUs sold by a single vendor and these GPUs end up underused due to poor software solutions. This is not a fatality and we, as an industry, are working on increasing average utilization and increasing diversity of accelerators. I’ll walk you through the different technical solutions to implement Serverless Inferencing and the trade-offs, from the chips to the virtualization software through the storage layers. Who is Yann Léger? Yann Léger is co-founder of Koyeb, a serverless platform for AI workloads, and spent the last 12 years building large-scale cloud service providers from scratch. Passionate about cloud computing, he has a deep understanding of the underlying infrastructure, from data centers to the software stack running on hypervisors. After building Scaleway, originally with bare metal ARM servers, Yann decided to go serverless with alternative chips for AI.