Python and the CI Factory: Crafting Effective Integration Pipelines

Dollar Dhingra (~dollar)


2

Votes

Description:

What to expect from this talk?: 1. What is proper continuous integration? 2. Why would you need one? 3. How to implement an efficient CI pipeline? 4. Some super useful python libraries for CI along the way

Objective: To provide attendees with a comprehensive understanding of the importance and nuances of Continuous Integration (CI) in Python. Through best practices, the talk aims to empower developers to design effective CI pipelines that enhance development workflows, increase system quality, and facilitate quick and high-quality deliveries – ultimately driving business success.

Details:

At Tata 1mg, we designed the continuous integration pipeline for our Python backend services. This process provided us with invaluable insights about the importance of implementing an effective CI/CD, which I'll share in the talk.

  • The talk will begin by defining proper continuous integration. I'll propose a set of questions that act as a litmus test, confirming if developers are genuinely practicing appropriate continuous integration in their systems.
  • We will then delve into the necessity of proper continuous integration, introducing concepts such as quicker feedback loops, smaller development iterations, the power of automation in reducing manual interventions, and improving developer productivity, with a special emphasis on automated tests and checks.
  • The audience will learn about the structure of an effective CI pipeline, popular tools and libraries used in Python projects
  • Additionally we will walk through the best practices for an ideal CI pipeline such as keeping the build fast, parallelizing tests, maintaining a prod like env for CI, keep track of key metrics, with some examples of python libraries helping in this process and more
  • We will also examine two different scenarios for building a CI pipelines:
    • Designing a CI pipeline from scratch.
    • Optimizing an existing CI pipeline.
  • Lastly, we will cover the long term benefits of CI for developers, the organization, and the business outcomes

Audience: Beginner to intermediate

Structure of the talk:

  • Introduction to CI/CD, and what is proper CI (3 mins)
  • What to expect from this talk? (1 mins)
  • Benefits for an effective CI pipeline for tech & business (3 mins)
  • Best Practices & success factors for CI pipeline (3 mins)
  • Tools in CI pipeline (1 min)
  • Scenarios for crafting an effective CI pipeline
    • Designing a CI pipeline from scratch (5 mins)
    • Optimizing an existing CI pipeline (7 mins)
  • Conclusion (1 min)

Prerequisites:

Although this talk is for beginners, there are some prerequisites which will help the audience better grasp the content

  • Interest in improving developer experience
  • Basic understanding of concepts like version control and code quality tools
  • Exposure to automated testing like unit and integration testing
  • Some familiarity with CI/CD

Content URLs:

[WIP] https://docs.google.com/presentation/d/1Gt9okXfR-Kw1IPTvFSyp2hHt4Q1H9Li41uafIG6Eh60/edit#slide=id.g27473a8238e_0_124

Speaker Info:

Dollar is working as an Engineering Manager at Tata 1mg, where he brings together his extensive experience with Python backend development, including Django and Sanic, and a genuine interest in core Python and concurrent programming. His work involves designing high-traffic, scalable, and fault-tolerant systems, which has shaped his comprehensive approach to backend engineering. A firm believer in fostering a positive work environment, Dollar is committed to creating spaces where every developer has the opportunity to reach their full potential. Building on these principles, he views CI/CD not just as a process, but as a natural progression to enhance developer experiences, streamline system quality, and ensure quick, high-quality deliveries which are crucial for business success.

Speaker Links:

https://dollardhingra.com/blog/python-code-quality-tools/

The below blogs may not be related to the CI/CD, but I just wanted to highlight blogs on which I worked hard to come up with a helpful content: https://dollardhingra.com/blog/python-abstract-base-class/ https://dollardhingra.com/blog/bus-lottery-factor-software-engineering/

Section: Developer tools and automation
Type: Talks
Target Audience: Beginner
Last Updated: