Comment by Cthulhu_ | Hacker News Reader

Comment by Cthulhu_ | Hacker News Reader

Parent Full thread

There is definitely merit in your comment - basically, solve your problem first - but in practice, extracting functionality (such as a message queue), especially if the system is older, is very difficult and expensive, so it's put off so it becomes even more expensive.

I've seen a few projects where instead of committing to a large re-architecting of an existing application, or extracting specific components, they went for a full rewrite. I don't think they were very successful (maybe for the managers that pushed for a new department to be cobbled together from consultants and self-employed folk), because the problem was not well-understood and they over-engineered towards e.g. microservices and/or everything into an event bus before understanding the problem and the consequences of their decisions.

At one project, I'm convinced the decision to rewrite was political (the existing application was a C#/.NET application, the new manager Decided to have the new stuff made in Java on AWS. Oh and for every IT system to be migrated to AWS as well. Then the army of consultants and self-employed folk Decided to go for a microservices architecture linked together on an event bus (AWS Kinesis, for some reason), spent two years developing only to realize that hey, having a separate 'order' and 'inventory' microservice is kinda tricky when you actually have to check inventory to make an order. [/rant over])

>At one project, I'm convinced the decision to rewrite was political (the existing application was a C#/.NET application, the new manager Decided to have the new stuff made in Java on AWS.

When my startup was acquired, they eventually landed on rewriting our C#, AWS-hosted software so it would be in Java and ran on in-house data centers.

On the grand scale of things, I don't have that many years of work experience under my belt. What do you think would be the optimal evolution of a company from day 1 would be in this context? Obviously it's all speculation, but I would be interested to know how you think the message queue on database idea would go if implemented from the start.

TBH, that's the most obvious solution for eg. task management. I am sure majority of applications started at least 10 years ago started off their task management using the app database for queue management, before the proliferation of messaging queues.

The couple webapps I worked on moved to a rabbitmq-based solution relatively painlessly (though old BaseTasks tend to linger for a long while still).

> What do you think would be the optimal evolution of a company from day 1

optimal solution is which ever one that gets the product shipped and making profit. All else can come after. Because if you spend too much time planning for scale, even before you reach any scale (unless you specifically have pre-knowledge of such), you would not be successful.

Depends what you need the message queue for. I think it can be a sensible choice, it depends how much of the rest of the system you need to start building. If you're then working out retries, etc. start looking to see if there's a prebuilt solution.

I find a few things I tend to look for in solutions:

* Are other people using it for the same sort of scale of problem I have? This cuts both ways - some projects are complicated and work well for people who have much harder scaling issues than I may have, and some projects are too simple to deal with scale issues I have. Some backend systems may need to process billions of things, others sit and take 20-30 a week.

* Are there plenty of people using it? More people = more chance bugs are known about with workarounds / already fixed, more chance other people have hit the same problems I will.

* Have people been using it for a while now? Unless there's a good reason, a project a year or two old that's stable and solves your business requirements can be far better than a shiny new thing. Let other people hit the issues and upgrade next year.

* Can I debug it? This is what actually came to mind reading your comment. A database is nice when you can keep a full history of what's been going on.

* What exists where I have to build the absolute minimum amount of stuff? A nice example here is python-rq. For some of my problems I can easily set it up, it does what I need and probably more and happily chugs along. I need to build next to nothing to use it. Also, whatever you pick first likely reveals another unspecified business requirement and finding that out ASAP is important.

* Is there some industry standard? Company standard?

Don't be afraid to adjust the way you were planning to build your project to fit better with some industry standard approach. If you want to use package X (say, luigi or airflow for data processing) what philosophy does it go by? Can you tweak the way you're working to fit in with that? There are often good reasons for certain choices, and even when it's less clear it'll at least avoid you fighting your systems so much.

Not too precise I'm afraid as it all comes under "it depends" but picking something:

Job queues / processing, try for example

* Amazon batch * Luigi (+aws batch potentially) * Python RQ (with redis) * Just files on S3 (an underutilised option imo) * SQLite or whatever database you're using now if you need something more custom

If you do only one thing then following the "functional core, imperative shell" idea of application development will set you up nicely for easy refactors in the future.

https://www.destroyallsoftware.com/screencasts/catalog/funct...

In short. Keep mutations to the edges of your application and keep your business logic purely functional.

Day 1 may be too soon to introduce queues etc, but having a clean separation between application logic and attached resources (like your storage layer) will greatly help with maintainability, testability and extensibility from the get go

I've been mostly functional throughout my career, so that method would have appealed to me before. The idea is to never mutate but always create a copy of data to prevent race conditions. Fundamentally though, you are mutating data. We have to accept this. Creating copies at each iteration does not change this. The question then is what abstraction allows mutation without allowing data races.

Recently I was looking into how I would implement a simple TCP protocol in C++. I think Boost's ASIO (asynchronous IO) provides the correct abstraction for this. It allows you to write your code as if it were single-threaded. Code which needs to be serialised can run on the same "strand". Strands simply serialise code, even if the code runs on separate threads.

I think this abstraction would work for the example in your link as well.

Just one data point, that might be of use to know:

We're currently running into an issue where we've used a table as a message queue in a database.

We gained a compliance requirement that we must log all access (including direct DB queries) to certain info in the database. One of the tables being used as a queue contains this data and is polled hundreds of times per minute making our logs ridiculously huge and we now have to move the queue somewhere else and/or split up the data.

Surely moving the data doesn't change the compliance requirement?

No, but in our case, moving the sensitive data to a table queried less frequently than one used as a queue shrinks our logs considerably.

Reply on news.ycombinator.com