<
>
+1 -1 → +10
Vote on this proposal

Which messaging layer should you use if you want to build a loosely coupled distributed Python app?

by Narahari Allamraju (speaking)

Section
Software Development Tools
Technical level
Intermediate

Objective

Loosely coupled, distributed, even driven, asynchronous - all these are terms that you will hear when you want to build an application that is scalable and handles large amounts of work load as if nothing happened, no sweat. There are many ways of building these, but when you talk about an application distributed across servers and talking to each other, the obvious mechanism is messaging - whether you send your own message or even using simple sockets like with Twisted or if you use a full blown broker like RabbitMQ. Even if you use something like Celery, you will have to choose a messaging backend which you want to use. And this choice is the problem - how to evaluate the various options? You can expect to learn from this session about how you can evaluate the tools and match them to your long term strategy and current requirements and then take an informed decision.

Description

Loosely coupled, distributed, even driven, asynchronous - all these are terms that you will hear when you want to build an application that is scalable and handles large amounts of work load as if nothing happened, no sweat. There are many ways of building these, but when you talk about an application distributed across servers and talking to each other, the obvious mechanism is messaging - whether you send your own message or even using simple sockets like with Twisted or if you use a full blown broker like RabbitMQ. Even if you use something like Celery, you will have to choose a messaging backend which you want to use.

And this choice is the problem - how to evaluate the various options?

The choice is not governed by external factors, although things like licence, cost, support etc make an impact on your final decision. The choice is primarily driven by what your application is built to do, whether it wants to handle only the functionality part and wants the underlying messaging layer to provide guarantees about message delivery and caching, querying etc.

This talk will distill my experience with seeing, building and designing such messaging applications and provide a methodology that you can use.

The talk will broadly follow these points to flesh out the methodology

The basic needs
Determining whether the application will benefit by distributing it into components that communicate using a messaging layer
Determine of we would like to have one or more instances of each of these components
Determine the basic layout of how they will communicate and maybe a simple flow and dependency analysis for each
Identifying the underlying patterns that we need when we say we need messaging
Determining the the type of interface these components will provide- request-reply or publish subscribe or queuing
Do we need to separate out messages based on types
Do we expect certain components to have a higher throughput of messages and do we need to accommodate them in a separate environment
Understanding any additional requirements that we will need and any assumptions of features that we expect the messaging layer to provide for us rather than we build.
Do we expect the messaging system to provide durable messages, guaranteed delivery?
Do we expect messages to be persisted to disk for crash recovery?
Do we need good monitoring tools for the messaging layer?
How easy is it provision new queues or topics or new instances of the messaging layer?
Latency, Routing and Priority
Determining what kind of latency do we need for these messages?
Can we afford messages sitting on a queue for a few hours?
Do we need message expiry?
Do we need message priority so that some messages can be routed to be processed faster?
How easy is it to route messages on this platform?
Understanding some of these messaging tools and what they provide to us
RabbitMQ
ZeroMQ
JMS servers
TIBCO RV
Redis
A database
Coming up with a quick reference matrix that we can use to evaluate
A brief summary of all the above tools and their best features
Applying future growth considerations
Thinking about volume growth - will the messaging layer handle a simple 15% year on year growth?
How much more storage or how many more brokers do we need if we grww year on year by 50% or more?
No previous experience with messaging is required - basic assumption is that you are comfortable with an application where two standalone components doing separate tasks wish to talk to each other.

Requirements

Basic knowledge of Python and an idea about distributed applications

Speaker bio

Hari is a developer with over 9 years of experience in building large scale messaging and low latency applications using Java at various investment banks, and working on integration projects at large companies. Hari is interested in exploring and building knowledge in other technologies and likes to use Python where possible in daily tasks - primarily using Jython to script some Java test cases and utilities. Hari also likes to explore Django and attempt to build a web application ans is also interested in Celery and ZeroMQ for server side development similar to his experience.

Comments

Login with Twitter or Google to leave a comment →