Ask HN: Replacing an Inefficient Back End | Hacker News Reader

Ask HN: Replacing an Inefficient Back End

4 pointsbloopernova7 comments

I would like some advice on a more efficient backend for an application:

The system is a business process, where tickets are assigned, completed, worked on, etc etc. The source of truth is SAP, the data is graph, and that data is synced from an intermediary graph DB.

Currently all the ticket data is synced to yet another database because the intermediary is far too slow. About 100GB of database has to be imported all at once due to some issue I don't understand that prevents incremental updates.

The CI/CD for this is horrible. Deployment of a new version takes 6+ hours. Those deployments happen several times a week because the source of truth data gets modified. These deployments take place at 2am and require multiple people to be involved.

Have any HN people improved a design or situation like this? I'd like to help this project but I'm not sure what to tackle to get the best return on effort spent.

Is there a way to cache/proxy a graph source and pre-populate that proxy layer? It feels like an understatement of the year to say that the database shouldn't be recreated multiple times a week!

Thank you for any advice given or questions asked. I apologize for asking for free consultation, I will try to pay it forward.

> due to some issue I don't understand

This may sound flippant, but... go understand that issue. If that is the major pain point, you need to understand not only what the technical problems are, but what are the business drivers for it to exist. In the course of gaining that understanding, you very well might find a quick and easy solution that doesn't require a major architecture project.

If the source of truth is SAP, why not just build a service that gets the data you need from SAP?

> has to be imported all at once due to some issue I don't understand that prevents incremental updates.

Has to? Seems like a straightforward problem to solve. Might be good to understand why this “isn’t possible”

> These deployments take place at 2am and require multiple people to be involved.

Big red flag. If it’s a computer process it can be automated. Deployment should be automated. It is totally stupid to have people awake at 2am doing deployments.

I agree with all of those points. I'm told that there are corporate "challenges" involved with all 3 :(

corporate "challenges" cannot be solved with software, unfortunately.

Could you set up streaming?

You shouldn't need to re-create the database more than one time (seeding).

Did something similar a long time ago dealing with a massive monolith and multiple systems. Made a new system that was the source of truth. We used kafka and outbox pattern, so that records in the outbox were put in kafka and sent to the right system such as the monolith or other services.

> the intermediary is far too slow.

target that, everything else is "patching"

is graph db common in sap

first time hearing it

could it cause performance issues?

Reply on news.ycombinator.com