Framework-agnostic Generative AI with New Keras Ecosystem

Anshuman Mishra (~anshuman2) | 01 Jul, 2023

4

Votes

Description:

Generative AI is all the rage nowadays and has taken the world by storm. Want to generate images? Feed a prompt to stable diffusion models and generate a high-quality image by simply describing it. Want to generate meeting notes? Feed your conversation to an LLM, and generate a short summary of the conversation!

This is where the KerasCV and KerasNLP libraries come into the picture. KerasCV and KerasNLP are recent offerings by the Keras team at Google for Computer Vision (CV) and Natural Language Processing (NLP), respectively. Both these libraries keep the simplicity, ease of use, and performance of the core Keras API intact.

The release of Keras Core (new multi-backend Keras) aims to unify the Machine Learning community distributed across PyTorch and Tensorflow. It allows you to take advantage of simplicity of Keras API with powers of PyTorch, TensorFlow and JAX as backend. You can write a fine-tuning or generation script once, with KerasCV and KerasNLP, and switch from TensorFlow, to PyTorch to Jax with zero code changes. Alternately, you can choose one backend and write a custom training loop, without updating your checkpoints or model definition.

The focus of this workshop is four-fold:

Showcase how in a few lines of code, you can generate images using stable diffusion, generate summaries of conversations using GPT, etc.
Efficient Fine-tuning of LLMs using a technique called Low-rank Adaptation of Matrices (LoRA). LoRA greatly reduces memory usage during training and allows the training of big models on consumer GPUs.
Improved Text Generation Performance with Accelerated Linear Algebra (XLA), which speeds up generation (upto 10x-20x)!
Demonstrate how one can seamlessly switch frameworks without changing your training code and take advantage of the unique strengths of TensorFlow, PyTorch and JAX!

Outline

Brief Introduction to Transformers, Decoder-only Models, Encoder-Decoder Models, etc.
Generate images using Stable Diffusion
Efficiently Fine-tune GPT-2/OPT on Reddit data using LoRA
Demonstrate Speed-up in Text Generation with XLA and key/value Caching

Prerequisites:

A basic understanding of the Keras API and of Transformer models

Content URLs:

This is a talk that Anshuman gave on Multi-backend Keras ecosystem Link
A member of Keras team created following materials for Scipy con : https://github.com/chenmoneygithub/scipy-2023-keras , we intend to create a repository with support to multi-backend Keras as it was released after the scipy con.
Transformer Pretraining with KerasNLP ( Anshuman has ported this guide to multi backend)
PeFT with LORA (Abheesht is co-author of this guide)
Text Generation with KerasNLP (Anshuman is currently porting it to new multi backend Keras)
Generating Images with Stable Diffusion
Finetuning Stable Diffusion

Presentations : We don't have an exact presentation ready for the PyCon, however we can provide you with the presentation which we have given on KerasNLP in the last couple of months.

LoRA : https://docs.google.com/presentation/d/1EETynnZJnx0AEt71pAgeFz5uKI1r4u8lkCEgXba6fWQ/edit?usp=sharing
https://docs.google.com/presentation/d/1ibIxQ7B45o9-UndarTD9G3NU67sV916imQOgjmIE-54/edit?usp=sharing
https://docs.google.com/presentation/d/1D6yJXrvFtrEoW2duZDqC36QpjaZ2JZPxKn0WjDlISGA/edit?usp=sharing
https://docs.google.com/presentation/d/1uH3bG0o46GfGmGeTrGJB3p-SbQ9vOUJ63xp9S4I8JxE/

Speaker Info:

Abheesht is an ML Research Scientist at Amazon, where he works on detecting advertisement bots. Abheesht contributes extensively to ML open-source software and has been involved in developing KerasNLP.

Anshuman is a recent graduate from the National Institute of Technology Warangal. He is a Google Summer of Code mentee at TensorFlow and is contributing to KerasNLP. He is a 2xKaggle Master and has interned at Amazon as a Software Engineer and Bank of New York as a Data Scientist in the past.

Both Anshuman and Abheesht are long-time contributors to KerasNLP and are currently helping the Keras team in further developing a multi-backend Keras ecosystem.