Hacker News Reader

Comment by sytse | Hacker News Reader

Great to hear you notice the performance improvements. We're very happy with the improvements but we still have long way to go. We want the 99% latency under 1s and right now it is 2s: https://drive.google.com/file/d/0BzQDcBnEfNRZSTZYVXlyc1FGdEk...

There are some controller timings that are red https://drive.google.com/a/gitlab.com/file/d/0BzQDcBnEfNRZZV...

And the git access timings are a sea of red https://drive.google.com/a/gitlab.com/file/d/0BzQDcBnEfNRZTz...

Git access will be improved now that we have Gitaly as part of this release. In GitLab 9.1 we want to move git access from the application server to the file server. That should greatly reduce the latency.

View on HN

I'm glad this is a priority for you guys Sid, the one thing keeping me from using Gitlab is that it doesn't feel as snappy as Github. But as a point of reference, I'm also a guy that won't use Atom because it doesn't feel as snappy as Sublime. For some of us, speed comes first.

Speed is essential for a tool like GitLab that people use intensely. We've made great progress in having better monitoring to see what the cause is. For 9.1 we've scheduled many more performance improvements https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf... than we were able to complete in 9.0 https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf...

What about memory usage?

Is Sublime really an editor of choice if speed comes first?

It's all relative... what kind of computer are you using, the cpu speed, memory available, even OS and drive/ssd can make differences. Even on an i7 with 32gb and an SSD atom was too slow for a really long time... in fact after trying earlier releases of atom and brackets, I almost didn't even try VS Code. I'm really glad I did though.

In any case, it's just a matter of relative performance and perception in relation to what one runs on, and their needs.

Well, compared to Atom, yes. Two somewhat similar tools with trade offs between them, one of them being speed.

Note that Atom is written in Javascript, so the run time is extraordinarily CPU heavy. Even though many Electron (Javascript desktop framework) apps may have snappy UIs, they're typically even worse than Java for CPU (battery life, etc).

btw: Sublime Text is written in C++, so the performance might be even better than Java.

What's a similar editor that's faster? VS Code?

Yes.

VSCode faster than Sublime? Are you sure of that?

VSCode is certainly nice and fast, but I've been able to do choke vscode a lot faster than Sublime.

Plus, you can't even open files above what, 100(?)mb in vscode whereas Sublime happily opens that in a snap.

No, I meant that VS Code, (which is similar to Atom in both features, technology used and price), is (much) faster than Atom, (Electron vs Electron), it is obviously not faster than Sublime, (Electron vs Native).

I understood the OP's question as what is similar to Atom, but faster, is it VS Code?

If speed really comes first, vi is probably similar enough.

Well isn't vi a bit bloated rendering irrelevant lines, shouldn't we just use ed?

Taking a look at Gitaly & the conversations around it, I don't see much in the way of consensus. Are you using Redis for the distributed lock or is this still a centralized system?

My query being you can effectively use the native pre-receive hooks & wrap the git shell to allow consensus without needing to replace git with an RPC system like Gitaly.

Gitaly is still a centralized system. It shards the git repositories over multiple git servers in one datacenter.

We looked into Git Ketch https://www.infoq.com/news/2016/02/google-kick-starts-git-ke... for a distributed system based on hooks. But this requires a sync per repository (very slow with 100k+ repos per server) and it is no longer actively being worked on by Google.

Next time you think about distributing git, you should take a look at the combination of jgit and a distributed key value store.

A little while ago I spent a night and I was able to get a fully distributed git http server up and running. I used HBase and was able to get it most things working. That allows the only contention to be around refs. Everything else is well distributed and spread across as many machines as you need. Caching is handled on the key value store side, and the http server side. I didn't get to git gc work though.

> That allows the only contention to be around refs.

This is solved without modifying the git client using a distributed/shared lock wrapping the git-shell & git-hooks (pre-receive) . If you were to modify the git server behavior to say use epaxos, one could provide even better latency guaranties https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf .

I'm not sure about the GC operation though, did you use the DfsGarbageCollector that is part of Jgit? I thought that was provided as part of the DFS interface.

http://download.eclipse.org/jgit/docs/jgit-2.0.0.20120613090...

Thanks for the reply.

There are plenty of off-the-shelf consensus algorithms/services you could use in the git-hooks that don't have the git-ketch limitation. Zookeeper, etcd, consule, redis, etc. It really just matters your criteria and requirements, they all of trade-offs but I'm sure one fits.

To be frank I think your time is better spent looking into why one of these consensus packages isn't the right call for your needs. Since they could easily plug into the git-hooks & git-shell commands, it could avoid a massive investment of time from your team and you.

> you can effectively use the native pre-receive hooks & wrap the git shell

The way GitLab is organized (pre-Gitaly) makes it very hard to do things like that. Roughly speaking, when handling a push, it is a coordinated dance between GitLab components to make sure it all goes in, but none of these components has full control over the push from start to finish. One of the reasons we are creating Gitaly is to create a 'place' (namely Gitaly) where we _can_ exercise such control.

The other big reason for building it is not having to use Git repos on NFS.

(Gitaly maintainer)

Looks like those last two screenshots are Grafana - do you mind sharing the dashboard JSON for those views? Very cool!

Thanks. Most of our monitoring is public on http://monitor.gitlab.net/ the things I showed are not because we're still working to make Prometheus work with multi process applications. I'm not sure the Json is public.

There's also http://arewefastyet.gitlab.com

Can't access any of those links, just get a login prompt.

That is the internal one, Sid meant to link to the public monitoring, located at http://monitor.gitlab.net/. It doesn't have the daily overview dashboard, but the fleet overview might be an indicator too (http://monitor.gitlab.net/dashboard/db/fleet-overview).

Sorry about that, we've replaced the links with screenshots.

The problem is that we can't measure Unicorn timings with Prometheus since it is multi-process. Our Prometheus team is working to fix this for everyone.

Reply on news.ycombinator.com