Hacker News Reader

Comment by juangacovas | Hacker News Reader

Where exactly is your bottleneck right now? Database queries? I mean, where those >300ms go, foreach request or just some of them?

View on HN

Definitely the database. How would I begin speeding that up?

Two million rows in tables are not a problem even for old databases (I'm thinking of mysql 5.1)... Are you confident on your indexes, normalization, etc?

I definitely will look at the indices. I have several joins. Perhaps if I explain what I'm trying to build.

I am building a git analytics dashboard. I want to help my organization analyse developer productivity from bitbucket. So I am pulling in all the git commit logs, the diffs for all projects my org within the company has. We have 1500 repos across 20 projects. I am building dashboards atop of this. The data will grow daily. I would like to showcase project health in terms of commits, pull requests and whether there are duplications in work (surprisingly we have this. Actual repos being copies from here to there btw, insane shit). In doing this I realized I am struggling to scale it. The UI Takes 2 minutes to load. It is abysmal. The org is impressed but I am not, and it is for my own learning.

That's the problem. Try denormalizing your data. Use indexes more. If you can, pre fetch & cache your data.

Optimise your queries as well, complex queries with multiple many-to-many joins will add up quick to tons of rows to filter. Look at separating queries; to keep the overall in a single query, the db may have to churn through tens of thousands of rows. Where if you do a first select than a second based on the result of the first, you can bring it down to just hundreds or tens - with a quicker response than just one query.

Reply on news.ycombinator.com