Description:

Problem

In today's world, where data processing is a critical aspect of many business operations, the ability to handle large volumes of data efficiently is paramount. This talk will focus on our experiences and challenges in dealing with optimization challenges in a high volume and large scale system using MongoDB and share insights into the best practices and the strategies that made a real difference in our production systems (that are primarily written in Python).

Solution & Outline

1. Introduction to the Performance Challenges with MongoDB (5 mins):

• Setting context 🌟
• Real-world problem explanation.

2. Schema Design – More Than Just a Blueprint (5 mins):

• Designing the right schema in MongoDB, even though it's NoSQL 📐
• How thinking in a new perspective can help to innovate and obtain better performance.

3. Indexing and Query Patterns – Finding the Right Equilibrium (5 mins):

• The effective use of indexing, with focus on query patterns ⚖️
• Achieving balance and optimization through well-structured strategies.

4. Read Replicas and Aggregation Pipelines – Enhancing Performance (5 mins):

• Implementing read replicas to increase read throughput 📈.
• Leveraging aggregation pipelines for data transformation and processing.

5. Advanced Concepts Beyond Practical Experience (5 mins):

• The use of profiling to gain insights into query execution 🔍
• Sharding and Scaling (Vertical & Horizontal): Insights into these advanced strategies that we have not yet implemented but are valuable to understand.

6. Q&A Session (5 mins)

• Answer audience questions and engage in constructive discourse❓

Takeaways

• The audience will leave with actionable insights and techniques that can be applied to their large-scale systems in real-world scenarios 🛠️.
• They will gain a deep understanding of the underlying concepts, providing a strong foundation for future explorations 📚.

Prerequisites:

• Target Audience: This talk is designed for intermediate-level developers, database administrators, and enthusiasts. However, all are welcome, as there are no hard prerequisites.
• Basic Understanding of Databases: Familiarity with general database concepts including tables, schema design, queries, and indexing is helpful but not required.
• Interest in Performance Optimization: The talk will focus on scaling and optimizing database performance; attendees with curiosity in these areas are encouraged.
• General Programming Experience: Familiarity with programming, particularly in languages that interface with MongoDB, would enhance comprehension but is not essential.
• Some Experience with MongoDB: Having worked with MongoDB or similar NoSQL databases will be beneficial, though not mandatory.

Speaker Info:

I'm Krishna, currently a tech lead at Certa (https://www.getcerta.com, a Silicon-valley startup) with hands-on experience in dealing with the complexities and demands of large-scale systems. My focus is on building robust systems across different verticals of the product that cater to varied use cases. I also spearhead the backend hiring efforts and have successfully scaled teams at startups, marrying technical acumen with team leadership. This multifaceted experience enables me to approach problems from various angles and develop innovative and effective solutions.

Beyond the world of technology, I embrace a holistic approach to life, enjoying pursuits like working out and exploring new philosophies. This unique blend of technical expertise and holistic thinking fuels my passion for sharing knowledge, inspiring others, and tackling complex problems in the tech industry.

Speaker Links:

• https://www.linkedin.com/in/krishnareddyk/
• https://github.com/krishna-certa