Description:

To address the inefficiencies in memory management for large language models (LLMs), which lead to wasted resources and limited batch sizes, the authors introduce PagedAttention, an attention algorithm inspired by virtual memory and paging techniques. They also develop vLLM, a distributed LLM serving engine that achieves near-zero waste in key-value cache memory through block-level memory management and preemptive request scheduling co-designed with PagedAttention. Through evaluations on various models and workloads, vLLM demonstrates significant throughput improvements of 2-4 times compared to state-of-the-art systems without compromising model accuracy. This solution not only optimizes memory usage but also supports popular LLMs of varying sizes, exceeding the memory capacity of a single GPU, making it a groundbreaking advancement in high-throughput LLM serving. The main goal is to build the fastest, cost effective and easiest-to-use open-source LLM inference & serving engine for Enterprise-grade LLM applications.

Talk Outline:

Challenges in serving LLMs [5 Min]

• Overview of large language models (LLMs) and their applications
• Challenges in serving LLMs, including memory management inefficiencies
• Importance of high-throughput serving for LLMs
• Overview of Transformer-based LLMs and their generation and serving procedures
• Iteration-level scheduling used in LLM serving
• Challenges in memory allocation and fragmentation in existing systems

PagedAttention Algorithm [5 Min]

• Description of the PagedAttention algorithm inspired by virtual memory and paging in operating systems
• Key and value vectors stored in non-contiguous blocks in memory
• Block-wise computation for attention scores and output derivation

vLLM System Overview and vLLM in action [Quick Live Demo] [8 Min]

• Architecture of vLLM, a distributed LLM serving engine built on top of PagedAttention
• Centralized scheduler and distributed GPU workers
• KV cache manager for efficient memory management
• Description of the PagedAttention algorithm and its implementation
• Design of the KV cache manager and its role in facilitating PagedAttention
• Handling of variable length input and output sequences

Evaluation of vLLM with other LLM Inference Libraries [7 Min]

• Performance comparisons with state-of-the-art systems such as FasterTransformer and Orca
• Results on various models and workloads, including chatbot and translation tasks
• Memory savings and throughput improvements achieved by vLLM
• Summary of the challenges and solutions in high-throughput serving of LLMs
• Advantages of vLLM and its potential applications in various domains

Q/A [5 Min]

Takeaways:

This talk introduces a groundbreaking solution to address memory management inefficiencies in large language models (LLMs). By leveraging PagedAttention, inspired by virtual memory and paging techniques, and developing the vLLM serving engine, the authors achieve near-zero waste in key-value cache memory through innovative block-level management and preemptive scheduling. Evaluations demonstrate that vLLM significantly boosts throughput by 2-4 times compared to existing systems, without sacrificing model accuracy. This solution optimizes memory usage, supports popular LLMs exceeding single GPU capacity, and aims to be the fastest, most cost-effective, and user-friendly open-source LLM inference and serving engine for enterprise-grade applications.

Prerequisites:

Basic Python Programming, Basic Deep Learning/Transformers and Basic knowledge of LLM

Speaker Info:

Mr. Saikumar Dandla is currently working as AI Research Analyst/Engineer II in Amazon Research and Data Science team, Amazon India. He has 5+ years of experience in research and development in the artificial intelligence domain and software development domain. He has served as an AI Researcher in DRDO Young Scientist Lab Cognitive Technology, Chennai from Oct 2020 to Dec 2021, during which he developed two major algorithms using deep learning in radar which improve and outperform results and one radar standalone software . He has served as software engineer in Infor (Product based company), During this period he successfully delivered three projects. His research interests span the areas of Deep Learning, Multimodal Multilingual Processing, Natural Language Generation, Natural Language Processing, Computer Vision. He won the tech innovation award for LLM Models and was the recipient of the ML University Champion Award in ML (2022), CV (2022) and LLM (2023). He delivered guest lectures in NITW, SRM, JNTUH, SNIT and worked as TA [Head] at IITM Research Park for AI for Engineer Course with Timothy Aloysius Gonsalves

Speaker Links:

LinkedIn: https://www.linkedin.com/in/saidsp19/

Github: https://github.com/Saidsp19