Microservices for startups: an interview with Sam Stagg & Pawel Ledwoń

Our VP of Engineering Sam Stagg & Platform Lead Paweł Ledwoń were recently interviewed by Jake Lumetta for the new Microservices for Startups book.

The book offers practical advice on working with microservices including how to determine proper service boundaries, break apart your monolith, and much more.

Sam and Pawel discussed how microservices are used at Pusher, the motivation behind adopting microservices on the new platform, their engineering approach to it and ways to organise your team to get the best results from working with microservices.

Here’s an extract on them discussing issues they had with microservices:

Have you run into issues managing data consistency in a microservice architecture? If so, can you describe those issues and how you went about solving them?

There is one particular problem companies like Pusher face — developers need to implement each feature in at least a dozen SDKs. This is a work multiplier and the time to develop new features is proportional to the number of SDKs. We have to keep things for our clients simple by avoiding any sophisticated coordination logic on that level.

Lack of data consistency increases complexity. We keep our client libraries lean by ensuring APIs they use provide consistent information via simple endpoints.

One such problem is resuming interrupted subscriptions. For example, a client subscribed to a chat loses the connection for 5 minutes. When it comes back, the client needs to retrieve missed messages and re-establish the subscription. It seems like a trivial problem, but out-of-the-box solutions that work at Pusher’s scale are hard to find.

We found a pattern across subscription implementations — services combine a primary data store (e.g. a relational database) and a message bus (e.g. Redis) for real-time updates. Because most message buses offer limited buffering capabilities, clients sometimes need to contact the primary store before resuming the subscription. Because this design is so common, we implemented an abstraction we call “resumable subscription.”

Resumable subscriptions​ coordinate the primary store and the message bus to provide a simple, consistent API for clients. They prevent data loss and duplication arising from having two disjoint sources of data. Requirements for the primary store and the message bus are low enough for existing open-source projects or hosted solutions.

Read the rest of the interview here.