Why we ditched Django for Microservices
blog.hubblehq.com
blog.hubblehq.com
As per staying lean as stated in the article. I don't think that's staying lean at all. I must mention that I have no knowledge of the framework or codebase of the company in question, although rewriting a codebase is almost never a good idea. Well, it will require a lot of rewriting which is redundant, you would be ditching a lot of testing and bug fixing you have done ever since, and you will probably create a space a lot of new bugs.
I believe a better approach would be componentize your structure in a way that's still governed by django, every component split into apps. And if there are outlier apps that do not need to be within Django at all could be used by redirecting the requests to those services on the load balancer / reverse proxy level.
Also we split up the re-write over stages and haven't done it all in one go. I'm gonna write more about it in another blog post but essentially we still have our monolithic Django system.
It currently powers the majority of the services except we have changed the architecture so we can split things out overtime. This allowed us to achieve what we wanted to in 4 weeks yet lay the groundwork for the future.
That should be the simplest problem ever to solve. Use a requirements.txt, use vagrant, and maybe even Salt or Chef if you require more specific tools.
Microservices are great in theory, but every app ends up requiring a lot more interconnectedness as you grow. I'm all for splitting services off, but it's usually something to consider once you're a few iterations in.
I strongly believe that learning new languages help people become better coders overall as it increases their knowledge of different problems and solutions which can be applied across most stacks.
Also no one developer should ever be solely responsible for a particular service. But that should also be the case for apps within a Django project.
Limiting developers in which languages and technologies they can use is not going to solve that problem, it merely addresses the symptom. Developers make hundreds of little less obvious and visible decisions that have a long term impact on your business. Picking the tools should always remain a developer decision.
If you a) hire the right people and b) make sure your goals are aligned, your developers limit themselves to a few core technologies. If not, you're screwed anyway.
(And yes, that does mean mutually accepting and respecting the fact that your developers' interests in exploring new things and your business interests in may not be aligned indefinitely.)
Technically, one way of solving this is to make sure that the acceptance criteria for any single micro service are clear. For example, you have to expose these REST endpoints; given data set A, respond in this way to these requests, etc. To me, a micro services architecture implies that the development team is responsible for the entire life span of the service, from conception to replacement. This includes maintenance. Insert the devops buzz here as well. When the given solution can no longer be maintained because of failing knowledge (or even interest?) by other devs, it should be replaced. This should be no harder than refactoring a large method, except on the scale of an entire service.
Non-technical benefits include expanded responsibility and shiny architecture can possibly attract better developers/engineers. I know this has worked for us.
I think it would be reasonable to somewhat limit the available choices, especially at the start of a project.
Some reasons why decoupling your architecture can make sense from a business perspective:
- You have a complicated product, and wish to break it up into more digestible chunks, with allocated teams working on each abstract business concept (Amazon and Spotify do this).
- You want to avoid having a single point of failure- if one service fails, the rest of the system might well keep on working. (Ever noticed how sometimes a panel on Amazon doesn't load?). If you're running a continuous development process, stuff breaks, a lot. It's good if that's locked down.
- You have a business requirement to build something with a different technology (from experience- rendering javascript in a node service was much, much easier and more maintainable than with an existing Django app- we tried both.)
- You want to experiment with new technologies without changing your whole stack.
I'm pretty sure that the author wasn't talking about moving to microservices so they can blindly start building every aspect of the application in a different stack- but more that if they need to use a new technology for their business requirements, or want to give something a trial without committing, they can. And with the uncertainty and rate things change in modern software, and in a startup, that flexibility is invaluable.
I get that monolith to microservice is all the rage these days, I just don't understand (and the article doesn't really say) why you have to "ditch Django" to do that.
Maybe they are still using Django in some capacity and the title is just slightly overstated ... not sure, but would appreciate an author comment in this regard.
However overtime I would imagine we would move away from Django as with small enough services it would be overkill as a solution.
Saying that, if it makes sense at the time then why not. With this architecture you have the flexibility to do that.
An overstated headline isn't the worst crime on the planet (though, I would say that Google has already carried the headline far and wide already ... sigh).
I keep picturing the legendary missing Novell server that was buried behind drywall at the University of North Carolina:
I thought Django and "microservices" are orthogonal. Can't Django be used to create any service, even a micro service? I certainly used Django in the past just as a REST server, without any "web site" to speak of. It just reduces some boiler plate.
I don't recommend it, but you can use it
I would prefer to use Django mainly for CMS kind of things
Django was born as a CMS and it really excels there, but it's modeled as a complete end-to-end stack. That's great if you love every single component of that stack, but not so great if you'd prefer alternatives.
We haven't separated many of our services out but we actually have extended a lot on top of DRF. We created a way to do internal requests, so now instead of passing querysets to HTML views we actually do an internal API call. In the future, that code can be moved into it's own service.
Abstracting things out like this allow us to plan for microservices in the future while not having to deal with tons of different codebases and deploy strategies right now.
In the past year as we've scaled from 3 to 20 developers, I would say that DRF is the single-most crucial element that helped us scale the team without any massive issues.
We've been using DRF pretty seriously for over a year and have yet to find a feature that we want that isn't already implemented or easy to implement on top of it.
Saying that if it increases development speed and makes sense in the scenario then I wouldn't shy away from it.
Microservices create a boundary and that forces you to create a request/response protocol to enforce that boundary. You also have to create documentation for it to be useful.
There is nothing stopping you from creating your app as a bunch of small services without the added complexity of microservices/SOA. Each slice of your system could be an isolated library with a stronger enforced protocol and documentation and you'd get largely the same benefits without having to deal with the deployment headache.
The reason people don't do that is if they build isolated components in the same monolitic system, nobody takes the time to create a proper protocol and documentation.
The only way people seem to be willing to give separate services the appropriate design respect is to physically separate them and solve them as an isolated problem. The physical separation forces better behavior for microservices to even be useful.
It doesn't help that most languages have really weak capabilities to enforce strong protocols/boundaries beyond a simple type checker via complier. A language like JS, Python, PHP, Ruby, etc. don't even have a compiler so the ability/desire to create strong protocols in most code is almost nonexistent to most developers.
The developer in me kind of likes that - fun new toys! But from a business point of view, I'm not sure it sounds like such a good idea: lots of weird, unmaintainable stuff.
If the service proves useful, you can migrate it to a more appropriate language. By appropriate, I mean a language that fits your companies culture. Java Shoppe? Java. C# Shoppe? F#.
You can also get a better match for the domain. For example Python has some great math libraries. They are easy to pickup, pleasant to use and reasonably performant. Say you need such features. Boom! Python service doing what it does best while the rest of your shop can use the tools that work well for the domain.
I get the idea you're trying to convey here but in the case of a well-organised team/organisation that won't really fly as everyone will be involved at some point and choices are being made together.
"Oh we'll just RabbitMQ everything together", sure, great, until you discovered certain services needed a smaller latency, more bandwidth, etc
And RabbitMQ libraries are a pain and are not smart
Tried RabbitMQ, had huge issues that standard library barely functioning. Running direct code samples from support failed, finally supporting dev confessed, "I don't know java and the java guy is busy."
I was forced to move on.
I mean lets say I request a web page which depends on 5 microservices. What happens ? does rails or whatver block till all five have returned the output - or do you use something like ESI ... how are the microservices composed finally ?
A benefit of this is that people get a unified view of the system. You can pull parts as needed. You can survive partial outages better than a monolithic app.
Of course you can still call each of the services independently. The ESB is just another tool in your box.
The way you will tackle this will largely depend on many questions like what services are safe to be exposed publicly, what services are dependant on other services, do you have a public API?
For example, you could have a public facing API on each service, or endpoints that render HTML templates. The services may or may not talk amongst themselves, either via the same public API, or a private one. If you make a request to one of the services, and that service needs to make a request to another service to generate the response, you've got to make a call on whether you want a blocking or non-blocking approach. Consider:
1) Blocking until the other services have responded
2) Pinging the server to check when it's done
3) Using web sockets
4) Whatever the hell else you can think of
Alternatively, you might want not want to expose your services publicly, and choose to have a single service that talks to the services on the clients behalf, a middleman. Your client speaks to this service, and the service sends of various requests to the other services. But the problem still stands. To block, or not to block?
AFAIK some big names like Amazon, Spotify, (Twitter?) have multiple public services. You may sometimes notice individual components on their [web] applications failing without bringing down the whole site, since the failures are isolated to the different services. Neat, huh?
The result was that changing the title on the homepage took two weeks of manual regression testing and it took a new developer a minimum of two weeks to get a development instance up and running. I felt so sorry for the Ops guys, I can't even imagine how terrible it was for them. There was no one, ops or dev, that knew even 25% of how the whole system worked. It always felt like the leaning tower of pisa made toothpicks and popsicle sticks held together with bubble gum and boogers.
The cost to run that Rube Goldberg machine monstrosity ruined what could have been an amazingly profitable business and great place to work. I don't recommend it.
So, now you have thirty smaller services for each new hire to install and link together? I can all but guarantee that this particular problem is better solved with Vagrant and some orchestration scripts. Takes your install from a list of packages to a single "vagrant up" command.
Unix's tools and good microservices have the following things in common:
1. They are small 2. They are loosely coupled 3. Shared-nothing state 4. They enforce standards for consuming and emitting data 5. They enforce a single standard for piping data from one process to another.
That said, I strongly believe that this won't work for every class of problem. And while I may be putting words into Tom Watson's mouth here, I believe he would say the same. But hey, guess what? This WOULD work for quite a few problems!
Some of the sentiment in the comments here feel defensive and knee-jerk. And I don't understand why. Tom Watson isn't telling you that you should use microservices. He's only explaining why he's using it at Hubble. I'm pretty sure how it turns out is going to have no impact on anyone else here, except (obviously) his customers.
But I hope it succeeds. I think there's a lot we can learn by applying the lessons Unix taught us to web-scale utilities.
This can all be done via libraries with clear APIs, with none of the deployment headache of a microservice.
Microservices doesn't fix everything and you're right about Django encouraging this type of design. Each approach has it's pros and cons.
Good people make mistakes too. Having good tools to ensure code quality doesn't make it impossible to ship bad products, but it's definitely important.
This approach will probably change as we get larger but being a small development team means that it is fine for now.
I assume that REST or Thrift (or something similar) is the way most microservices talk to each other. What if the latency isn't acceptable?
I've thought about switching our architecture to microservices but these two potential issues are making me hesitate.
How important are these issues in practice?
As for monitoring there are plenty of tools to do that. You have bigger problems if you expect your services to go down often.
Yes, even in your monolithic app, you probably used some tooling to monitor it. But now that single service being monitored has multiplied into 10-20 services being monitored.
Nobody expects their services to go down, but if you don't plan for it, then that means you haven't thought about high availability or disaster recovery enough.
Being myself - just a few years ago - a long time strong proponent of tightly coupled monoliths, I can't believe how blind and foolish I was in my fundamentalist rejection of loosely coupled architectures.
Monoliths do just fine up to a certain level of structural complexity. Above that, asynchronous service-like architectures are the only viable way to go.
Microservices is an attempt to see if the services patterns work below that waterline, down to the function level. That's why I find microservices at least interesting.
At any rate, it's a cost/benefit balance game, not an ideology.
Edit: clarity.
What people here are talking is mostly dumb, since Microservices are better in a lot of ways especially when it comes to multiple Developers. Currently my company struggled a lot with Code that got changed by two guys at the same time, which was hard to fix.
We had less unit tests than now, since they were too complicated at a certain time.
Overall though, services should expose either a well/self-documented RESTful, zmq or similar API such that they can be rewritten in (your favorite whatever here). Being able to discover the API meta information automagically can be helpful for autogenerating clients.
The other aspect is that services have a risk budget: if the service is conventional and not as crucial, increase the experimentation. If the business/process model is risky/uncertain, use common, stable technologies.
(Gist: Pick from the toolbox, carefully... apply common-sense... Don't get stuck on perfect now or perfect waay later... Anticipate a little now to avoid pain of changing later.)
PS: Enterprise Rails is a great read. It's safe to ignore the Rails bits, because the architecture bits are universal and excellent.
Update: RESTful API metadata formats http://apiux.com/2013/04/09/rest-metadata-formats/ (and also https://en.wikipedia.org/wiki/HATEOAS )
Splitting your app into multiple apps that send each other HTTP requests or MQ messages seems likely to just make your app slower.
I'm just not compelled by the reasons the author gave for needing to switch from Django to microservices.
>We have a multitude of parts that each could easily be self contained as their own service. For example: Messaging, Search, Authentication, etc. These all have different requirements and focuses and therefore lend themselves better to different languages.
Yeah, not really. Python is not inherently less suitable for authentication than, say, Java. Both languages are designed for general purpose programming. The only real drawback to python is that it is relatively slow and this isn't even likely to be an issue at all if you are using django (you'll most likely bottleneck on I/O).
>Also, looking at our roadmap, we have lots of sprints and projects coming up that could be implemented in their own silos. By splitting these into services this allows us to quickly test and iterate without worrying about the rest of our product.
And splitting them into separate libraries is not sufficient because?
>Whereas in a monolithic system these services are bundled together and can create a spaghetti mess of workarounds and packages.
Newsflash: you're MORE likely to make a spaghetti mess of workarounds if you use a microservices architecture because of the impedance mismatch between API layers. You'll have to create a representation of, say, a user object on both sides of the microservice divide and write code to serialize it and deserialize it. Hello massive code repetition and obscure bugs.
>Having been a dev team of 1 for a long time I'd never experienced the headache of installing all the dependencies required by our Django system. It never really occurred to me that as I installed more packages, it was becoming incrementally larger and more complicated for new team members.
If you think your headache is bad with django (which has consistent opinions - e.g. on how templates should work - that makes it possible for modules to work together), get ready for a nightmare when you start using a hodgepodge of different systems written in different languages.
>We don't worry too much about code quality
This has officially moved this blog post from bad idea to outright parody.
>However, most of the cons are part and parcel with programming.
Here's a small laundry list of the exciting issues you'll now get to deal with:
* Writing serializers and deserializers for your objects so they can be passed from one service to another. And debugging them!
* Babysitting all the new services you will need. You'll need to keep them up and running, monitor them, have a backup plan for all of them going down and lots of lovely extra error conditions.
* The headache of trying to figure out how to scale your network of microservices - probably meaning lots of time configuring load balancers.
And much, much more.
If OP is more worried about maintenance of project, Django gives foundation to build upon with sane conventions.
Anyone who reads few pages of Django Docs should be able to pick up quickly.
PS. I happen to currently maintain project written by other people with no docs.
What?
Every single one of those is a solved problem that has bindings to most major languages. Why in the world would you switch languages for an Auth mechanism (for example)?
Yet, the dynamic language community constantly laughs at Java developers and their high overhead dependency management solutions, which just happen to have these kinds of problems solved since more than a decade.
Pathetic.
From the talk of "packages" I think the original was talking about code dependencies. But it's worth saying that you can manage external dependencies the same way, with something like puppet.
Of course you can use things like puppet, but the comment I was replying to was saying these were solved problems in the Java world for more than a decade which isn't exactly true.