Cheaper Experimentation on an existing ecosystem using Airflow and Jupyter notebooks
Shivangi (~angiesk) |
Covid changed a lot for us at Grofers. After the first lockdown was announced, there was a sustained demand surge. Our supply chain was strained and this started to reflect in delivery times for our customers. Covid made us realise disruptions are synonymous with the current pandemic situation and continuous experimentation is the need of the hour. We had to innovate rapidly and come up with various solutions that could help us handle these disruptions. We started several experiments in a short span of time.
The major challenges we were faced during the disruption management were:
- Rapid Tech Prototyping
- Quicker Dev to Deployment Cycle
- External Orchestration over existing microservices layer
- Scale Faster
This talk will focus on how Jupyter and Airflow allowed us to solve these problems and why this workflow worked better than more conventional(commit + redeploy service and crons) based deployments.
As hypotheses were formed, some experiments were required to be made live in production with a turnaround time of less than half a day. Since these experiments had extremely small dev to deployment cycles, there were frequent changes in the code and logic. The developers and analysts had to make changes multiple times a day to power these experiments.
In addition, Airflow allowed us to develop complex workflows and tie them into existing services with very little overhead. And once the logic works(™), it scales.
What we got in return:
- Scaling on demand
- Multiple Solutions to Handle Future Disruptions
- Flexible business logic that can take advantage of existing microservices
- What's Experimentation? (2mins)
- What’s the Need for Cheaper Experimentation? (5mins)
- Need for Cheaper Experimentation?
- What’s a Disruption?
- Current Challenges due to Covid Disruptions
- How was Grofers impacted by Covid Disruptions (3 min)
- What did we do to overcome it? (2 mins)
- What's Airflow and python notebooks (5 min)
- Why did we choose them (3 mins)
- Why not a new service (2mins)
- Key Takeaways (1 min)
General knowledge of computer applications
If you are a developer, software engineer, data scientists, or data analyst
Anyone who knows the basics of Python
I'm Shivangi, an SDE 2 at Grofers India an e-commerce platform trying to revolutionize grocery deliveries for upcoming India. I currently help with building hypotheses and run experiments at the Innovation Hub Team. I like solving problems and coming up with hacky solutions to solve them quickly and run iterative experiments. I graduated in Aug 2019 with a Bachelor's Degree in CSE from PES University. I'm a cybersecurity and data science enthusiast along with being a full stack developer. I am a former GHCI 2018 student scholar. I'm also an active volunteer for WomenWhoCodePython and IOSF. I have previously volunteered at Pyjamas Conf 2020 and this is the link to the playlist.
Personal Website: https://my-portfolio-iota-ten.vercel.app/
WomenWhoCodePython Talk: https://www.youtube.com/watch?v=VYTzkvNihhk
Pyjamas Conf Recording : https://www.youtube.com/watch?v=uv8Gbt3ewhE&list=PLiwrSoq9tHkE_dwdkKcXQBeFnhTJJgosj&index=5&t=1951s