Creating performant and scalable microservices with gRPC and protobuf
Siddharth Goyal (~sid22) |
Description:
Services in python are extremely popular due to inherent advantages of Python language. However, most service architectures converge around web http based APIs. Micro-service architecture has led to more proliferation of micro web APIs orchestrated together in larger systems.
However most systems have a lot of micro-services that are internal and would have no need to by publically available. Communicating over HTTP JSON APIs is very inefficient. It escapes type checking, is slow and compute heavy. Protocol buffers are a cross platform open source data format to communicate serialized data across services.
Combining protocol buffers for direct remote procedure calls we can get many benefits
- Significantly reduce the latency and compute overhead to serialize/de serialize data communication at multiple ends.
- Strict type and data definition API contracts for all services.
- More structured data processing among services.
- Flexibility to expand data source in strict/non-strict formats.
- Full interoperability with services in different languages/frameworks while maintaining common data format checks.
Given the data intensive and highly connected service systems today. A shift to better data communication yields outsized improvements in overall system performance and cost with little regressions. It is also very maintainable approach as your services scale in complexity and count.
Prerequisites:
- Basic understanding of python
- General familiarity with services in python
Content URLs:
- https://protobuf.dev/
- https://grpc.io/
Speaker Info:
Siddharth is a software engineer at Rubrik where he has worked on multiple data intensive high performant, compute optimized services and packages. He has worked in multiple open source projects and is an active member of OWASP project. He has previous experience working at Innovaccer and some of his focus areas are cloud computing, few shot learning, reinforcement learning, mathematical proofing, compute optimized data intensive applications etc.