How to build/revamp a production pipeline
Varun Kochar (~varun58) |
What is a production pipeline?
Imagine being able to pin-point every trivial details of your product from the point of developing from scratch to deployment. Yes, this is what actually the concept of production pipeline generally mean. A product is basically, a series of sub-products and one should devise an optimal solution to inculcate all of them together. Here, instead of discussing the philosophical part of the pipeline building process, we will deep dive into the technical aspects of it.
Any company’s main goal would be to have their products in the growth stage, which is the key for establishing product's position in the market. In a product development team, tens of people work together, some push changes, some add bug fixes and some focus on new features. How can someone maintain every bits and pieces of the product technically?
To make that happen, we need an efficient, modular, well structured and well documented workflow, which takes us to the next step, that is understanding each and every component necessary for a production pipeline from a technical perspective:
- Foundation, Start with Abstract Base classes & Decorators? (BaseProcessor) (4 mins)
- Sub-products, Define a dedicated processor for each component (2 mins)
- Components Dependencies, How to encounter components dependencies (2 mins)
- Testing, tests components (2 mins)
- Clean code & Documentation, pep8 & git hooks (2 mins)
- Logging & Profiling (2 mins)
- Single Yaml to define the complete pipeline (5 mins)
- Core, which connects everything (4 mins)
- Case studies to test our above pipeline (5 mins)
What will you learn?
- Able to build your own production pipeline
- Save time & money by not dealing with unstructured and complicated workflows.
Basic Understanding of Python & git
Varun is Data Scientist III at Episource. He has 2.5+ years experience on Natural Language Processing problems and building data-driven products that are scalable for healthcare companies and hospitals. He has completed his graduation in electronics and communication from LNMIIT, jaipur, where he had worked on several python & deep learning projects. He has previously spoken in PyDATA Warsaw, Pie & AI conference conducted by DeepLearning.AI, Bangalore and PyDATA Mumbai.