Hacker News Reader

Comment by tonyjstark | Hacker News Reader

So much this. Same goes with all the Kafka usage, making development literally Kafkaesque. The data eventually flows through your microservice landscape, being eventually consistent and performance issues become a whole different nightmare. Yea, maybe I just didn't read enough of the documentation but maybe your 1500 users per month web-app doesn't need 7 microservices and Kafka as single source of truth, all hold together by 8 developers.

It's astonishing how many teams want to use cool tech instead of create a great tool for users or how many teams think that technology will solve bad management.

View on HN

So much this! As someone that regularly deals with data platforms and data engineering issues this nails it.

Today’s desire to make everything event driven and “real-time” boggles the mind. Solutions are contorted to fit the Kafka operating model.

The inevitable result is blog posts about how “we solved our self inflicted problems that would have never existed if we chose a sane design for our real problem. “

Oh and never forget that no senior stake holder ever really checks their info/dashboards in real time. Never mind actually making decisions in real time based on it.

Same! My dataengineering team wants everything to be moved from batch oriented to 'stream oriented'. We are replacing our batch workflows to flink/spark streaming. But no one has been able to answer me what business use cases need streaming or what if any problem is being solved by this added level of complexity.

Sometimes, streaming makes sense. When I was at a FAANG, we had a sampled streaming database with nice plots (kinda, ugly but functional).

This was incredibly useful as it allowed teams to figure out what had happened in prod without needing to wait for the next day's batch jobs.

OTOH, most companies don't need something this complicated, and I'd imagine you could do the same with MySQL/Postgres if you're not at FAANG scale.

I agree. When it is the right solution to the problem at hand there is nothing better than a good streaming solution.

It goes for pretty much all technology, that you need to choose the right tool for the job. For some reason the marketing just gets most people all worked up and ready to fit problems into technology that was never designed for their problems.

I migrated a unmaintainable over engineered java spring application consisting of around 20 micro services using Kafka all over the place to one single Rust application. A relief for anyone in the company.

Just you alone? Do others in the company know Rust?

I really like the idea of Rust I just worry about how things shake out In The Real World(tm).

I have no doubt Rust is here to stay for a while so I think it’s pretty solid play when appropriate.

Are you saying you rewrote not just the Spring application but all of the micro services as well?

I understood it as a distributed spring application.

What happened to the people who worked on those 20 microservices. What do they do now?

A few walked away. They did not agree with the move. But the company was really in the need for a radical change since development was stuck in complexity and maintainance.

One reason this can be doable is that working in the 20 microservice environment is so unproductive that the original app probably did very little.

Exactly. Making sure those services work in production, and the constant troubleshooting in local environments as well would easily be most of the developers' days.

Reply on news.ycombinator.com