Introduction to data-driven AI applications and multi-modal search

anthonypjshaw


2

Votes

Description:

Abstract

Want to learn more about AI but don't know where to start? Confused about terms like vectors, embeddings, Generative AI, and models? This workshop is for you! Experts at Microsoft are presenting an overview of building AI applications in Python using the latest technology and models. We'll build out features to a web application in Python that leverage AI to enrich and improve the UX. We’ll use data stored in Azure Cosmos DB to personalize these AI models for our application scenarios. We'll also cover foundational concepts and build up to practical advice for scaling and deploying AI applications to productions based on our experience with Global clients.

Overview

In this workshop, we will explore how to use AI to improve an online web store written in Python. This workshop is aimed at providing an accessible introduction to concepts in AI-powered applications such as embeddings, models, vector databases, and similarity. We will be using the latest multi-modal GPT-4 models to demonstrate how to take a feed of user pictures for an online store and add context to a database.

Outline

Act 1: Do I need an AI?

Presentation

  • Introduction to the speakers
  • When and where can AI add value to your application? Do users want a chat-bot dialog in your website or can AI be used to enrich the experience?

Demo

  • Showing the application we're going to build

Act 2: AI is not just LLMs and GPT

Presentation

  • Types of AI
  • Chat completions, RAG and embeddings - breaking down AI technical terms
  • Text embeddings
  • Use cases for embedding models
  • Multi-modal embeddings- using embedding models for images, video and audio
  • Storing embeddings as vectors
  • Adding a vectorized embeddings to a NoSQL database with Cosmos DB
  • Searching vectors with similarity search

    Hands-on exercises

  • We will use a Jupyter notebook to experiment with embeddings and doing similarity searches

Act 3: Building AI applications in Python

In this section we will take a basic FastAPI-powered application that has a web store and add functionality where a photographer adds pictures of products to a storage account. Using serverless triggers, these images are then converted into embeddings using an AI model and stored in a vector database. We will demonstrate how to then query that vector store to find similar pictures and match pictures with descriptions of existing products.

Hands-on exercises

The goal is to show a search feature in the website that users can type a query and it will locate the best product based on the name, description or picture. Users can then upload a picture of an outfit they like and it will find the best and most relevant product that is similar.

  • Setting up a storage container
  • Configuring a serverless function in Python to run whenever pictures are added to the storage container
  • Converting an image into an embedding
  • Setting up a vector database
  • Storing embeddings in a vector database
  • Creating search queries and similarity functions
  • Improving our search page
  • Adding a user-provided image upload to the site
  • Doing text-image and image-text relevance queries
  • Filtering data in queries

Act 4: Considerations for production

  • How do you assess how good your AI implementation is?
  • How do you measure new models and versions? Using relevance libraries
  • Localizing indexes and search - testing both English and Hindi. We will experiment with using Hindi queries on an English index
  • Thinking about scaling and performance
  • Writing tests for AI applications

Hands-on exercises

  • We will use a relevance model to measure and write test cases for our AI.

Act 5: Wrapping up

  • Where to go from here?
  • What about hybrid search?
  • Next steps and resources

Prerequisites:

We don't expect participants in this workshop to have any formal experience with AI. We will be breaking down concepts and talking about how AI converts text, video, images into a format (embedding) that can be used for predictive completion, search and comparisons. The application we will build in this workshop will be a serverless, cloud-based application. We will use Jupyter notebooks throughout for testing and exploration of concepts in AI. We will cover practical aspects of adding AI to applications, such as localization, testing, performance and scaling. Participants will be given credits to run and experiment with a deployed application in the cloud after the workshop. The technology shown in the workshop is Python, using open libraries and SDKs.

Video URL:

https://youtu.be/p8qC-pGqxs0

Speaker Info:

Ramya Oruganti is a Principal Product Manager at Microsoft with a strong history in Databases and Systems. Ramya has received technical accolades from Microsoft, Oracle and IBM. Ramya trained at Jawaharlal Nehru Technological University in Hyderabad.

Anthony Shaw is from Sydney, Australia and is a contributor to many open-source communities. Anthony maintains several popular open-source tools for DevOps, Security, Automation and Code Quality. He has been recognized for his contribution to open source, including as Fellow of the Python Software Foundation and member of the Apache Software Foundation. Anthony runs a Python blog and YouTube channel and authored the book "CPython Internals". Anthony is also a Fellow of Macquarie University and publishes articles on Python performance and compiler optimisation.

Speaker Links:

Anthony Shaw's home page includes links to talks, blog posts, GitHub projects, and publications.

Section: Artificial Intelligence and Machine Learning
Type: Workshops
Target Audience: Intermediate
Last Updated: