Launch HN: Haystack (YC W21) – Engineering analytics that don’t suck | Hacker News Reader

Launch HN: Haystack (YC W21) – Engineering analytics that don’t suck

104 pointslovedev39 comments

Hey HN!

I'm Julian, co-founder of Haystack (https://usehaystack.io). We’re building one-click dashboards and alerts using Github data.

While managing teams from startups to more established companies like Cloudflare, my cofounder Kan and I were constantly trying to improve our team and process. But it was pretty tough to tell if our efforts were paying off. Even tougher to tell where we could improve.

We tried messing around with JIRA which gave us story points and tickets completed but it didn’t help us dig deeper on where we could improve. We found a few tools that integrated with Github measuring # of commits, lines of code, and even comparing engineers using these metrics! - but we didn’t like that approach.

We wanted to know (1) how quickly we deliver as a team (2) what bottlenecks tend to get in the way and (3) as we make adjustments, are they helping us improve?

We scoured the internet looking for every piece of research on the topic we could find, talked to >500 engineering leaders working everywhere from startups to FAANG, and started to learn which metrics helped answer our questions and which ones just sucked. Once we had a clear picture of what that looked like, we built Haystack.

Haystack analyzes pull requests on the team level, giving you “northstar” metrics like cycle time, deployment frequency, change failure rate and 20+ more to help you improve delivery. Teams use Haystack to quickly find bottlenecks like code review, experiment with changes like smaller pull requests or automated tests, and see the result. Using this feedback loop, the top 70% of Haystack users have increased production deployments by 58% and achieved 30% faster cycle times on average.

We’re lucky enough to work with some awesome teams at Microsoft, Robinhood, and The Economist. As we continue to build out our product, we’d love to hear any of your experiences with engineering metrics, your thoughts about how to actually get them right, and of course your disaster stories :)

I just signed up and I'm blown away by how great the entire experience was. Their product is fantastic (better than Screenful or LinearB), their Slack community is awesome, and they even made me a personalized 5min onboarding video as soon as the data was ready to be presented. I'm a huge fan!

Thanks for the kind words :)

Hey, this looks really cool.

In my line of work [1] I talk to a lot of developers about productivity, here are some questions about what you've built in the context of that experience:

1. I think (as you mentioned) "number of commits" or "number of pull requests" merged are proxies for productivity at best. Most teams that I've seen care more about "are the tasks for the epic being chipped down" - is that something your tool can help estimate as well? The progress bar for the epic? It'd be nice to get a table of the metrics and where they come from.

2. In the same vein, "scrum with epics" might not always be how teams decide on units of work, do you support other methodologies?

3. You mention deployment frequency a lot, but that seems tangent to developer productivity - Isn't getting things deployed quickly mostly for the benefit of the product folks? (getting feedback faster)

4. How does maintenance work fit into this? Does a developer who spends most of their time refactoring show up with "worse metrics" than someone pushing copy changes to the landing page?

[1] CEO @ https://layerci.com (YC S20)

Great question! I'll try to cover them :)

1. While we do use pull requests as a unit of measure, we don't actually consider # of commits and # of pull requests as great proxies for productivity. We mainly track Cycle Time, Deployment Frequency, Change Failure Rate, and Throughput which help encapsulate speed vs. quality of an engineering teams. It's important for engineering metrics to correlate to engineering work. Specifically, that the engineering team has full control over those metrics. When looking from an Epic, Story, or Task perspective - this is often controlled by other parties, making it more difficult to know if engineering itself is improving or not. In addition to this, Epics often don't reflect actual engineering work in the same way that pull requests in Github do making them problematic to focus on.

2. Because we focus on engineering work itself from Github. Haystack can be used with any methodology so long as the team is using Git!

3. From an engineering perspective, the main goal is '# of successful iterations'. Product's main goal is that those successful iterations are pointed in the right direction. Looking at # of successful iterations (aka speed, deployment frequency, and quality) allow us to see how often we deploy value to customers and what is the quality of those deployments. Focusing on these metrics allow engineering teams to improve 'how they deliver'.

4. We actively don't look at individual engineers and we especially don't compare them using any metric since we've seen that lead to some bad takeaways like the one you mentioned. We help measure at the team-level to simply highlight improvements and bottlenecks rather than cross-compare any team or individual which is like comparing apples to oranges

I don't mean this critically but your product alarms me. I can imagine these reports being gamed and used to stack rank teams.

I am trying to be more open minded about your work but I can't understand how developer productivity could be measured with any of these metrics. I'm not aware of any prior work or research that shows that either. Writing software is a complex task and quantising it in this way seems reductive and problematic.

I would welcome being wrong about this. What am I missing?

TBH I love the skepticism here. It's important to think about metrics critically in this way to make sure we're building truly useful applications of data.

We've found that evaluating individuals, stack ranking teams, and measuring an individual's 'productivity' to be an ultimately futile effort.

It's more important to focus on team delivery as described in many research studies. My favorite being the book Accelerate by nicole forsgren. If you haven't, I do encourage you to check it out!

As the founder of Haystack I can be seen as biased so its probably more helpful to checkout their research across thousands of teams on what makes an 'elite engineering team' and how elite engineering teams ultimately help drive successful organizations across profitability, market share, and customer satisfaction :)

Big fan of this - we use it at AdoptOpenJDK (>100 repos, >60 active contributors) - it really helps you improve the engineering culture!

Hey AdoptOpenJDK team! :)

Hey it is great that engineering analytics is getting some love. I have something similar and I've found that you can generate some interesting insights from commit history if you look at how things are changing from a historical point of view.

I'm not sure if my public server can withstand it, but you can play with the public version at https://public-001.gitsense.com

If that doesn't survive, you can install GitSense (my product) with a single docker command, which you can find at https://gitsense.com

Looks cool!

Would love to chat. You can contact me in my bio.

I have often considered our field a bit messy when it comes to actually measuring productivity. Obviously it is not strictly about line of codes, commits etc. However, any estimation we may have by analyzing the metrics could still yield useful insights which is indeed better than nothing. In that regard, I can see what you guys are trying to achieve and I'm interested seeing it as a mainstream product which would also allow us to see how useful it can get when it was used more commonly. I'll be keeping an eye out for GitLab integration and ask my team to use it.

Love it! In the meantime, you can check out one of our customers who was able decrease their delivery speeds by nearly 90% and improve predictability to >95%!

The webinar link is below - pretty cool to see how teams use metrics to really enhance the way they work. Even cooler to hear how happy the engineering team was after removing all those annoying friction points in the development process :)

https://www.youtube.com/watch?v=KrSc9EOrE_4

Heads up: site is not loading. iOS Safari & macOS Chrome.

Mixed Content: The page at 'https://usehaystack.io/' was loaded over HTTPS, but requested an insecure favicon 'http://www.usehaystack.io/favicon.ico'. This request has been blocked; the content must be served over HTTPS.

Edit: It appears to be an issue on the naked domain only.

Thanks for the heads up, looking into this now! :)

In the meantime, we've opened up our team Slack (link below) if you'd like more info on Haystack (or just want to jam about metrics with other eng leaders from companies like Microsoft and The Economist).

I believe this link should work: https://www.usehaystack.io/haystack-slack-community

And sorry for the technical difficulties. Should be resolved shortly.

Looks good now. Thanks!

Also won't load on Windows with Brave

Looks like it was an issue with our Webflow set up. Nice catch! It should be up and running now :)

Likewise not loading on MacOS + Firefox.

Where is this product used at Robinhood?

GitLab support in your future plans?

Definitely! We support Github at the moment with plans to expand into Bitbucket and Gitlab shortly :)

How is Haystack different from Pluralsight Flow (formerly GitPrime)?

https://www.pluralsight.com/product/flow

They published a blog post comparing themselves to GitPrime and other competitors: https://www.usehaystack.io/blog/top-alternatives-to-gitprime...

You beat me to it :)

Seems like a potentially useful product, but the branding looks like you're already preparing for an Atlassian acquisition.

Our main goal is to build eng analytics that are genuinely useful for teams :)

But I suppose we will be building a Bitbucket integration shortly?

It was just an observation. The logo typeface and the blue are very similar to Atlassian's branding.

Gitlab first, please.

I made something like this for my own use in previous company I am thinking may be i should opensource it.

Just out of interest really, I've done some work for a company called BlueOptima (https://blueoptima.com) which does this in the enterprise space.

github integration sounds good for getting up to speed quickly and if I was still a CTO I'd be trying it out.

What pricing? Can't find it anywhere on the site.

There's a free 14 day trial, then billed based on active devs (aka # of devs who create pull requests). Also, here's a link to the pricing section!

https://www.usehaystack.io/#Pricing

This is awesome! Gitlab support will be nice :)

Loving all the Gitlab requests! If you sign up and select "Gitlab" as your tool of choice, it will put you on our waitlist and help us prioritize that work :)

I absolutely love this. Congrats on building such an obviously missing problem. The efficiency/productivity obsession in me dances for joy at the idea of this.

The last company I worked at used the "Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations" book as their bible. It was a wonderful way to structure team rituals and review cycles.

Awesome! Every person on our team has Accelerate on our their desk :) highly recommended read and is a huge part of how we've shaped our product

Reply on news.ycombinator.com