Backstage: an open platform for building developer portals | Hacker News Reader

Backstage: an open platform for building developer portals

github.com

146 pointsJamesQuigley27 comments

Member of the Backstage team here. Sorry if the messaging is unclear; we'll work on it.

Allow me to tell you about Backstage in the form of a story. Assume you're at a medium-to-large-sized company, in a team tasked with boosting developer productivity. You've developed a kick-ass pipeline to calculate test flakiness based on historical build logs and some data from your internal userinfo service. Problem is, userinfo has suddenly started exhibiting p99 response times in the 10s of seconds. It's not feeling well.

You go to backstage.yourcompany.net and search for userinfo. You reach the overview page for the service. It shows a ton of information, some of it pertaining to who owns and operates the service. Apparently it's the devcore team, they're on #devcore-support on slack according to the page but they're in the US west coast timezone so they're asleep. Well you are using the PagerDuty plugin in Backstage so there's a big "page us" button right there.

So now you've got the issues solved and you need to expose the pipeline output somehow. Cool, you go to backstage.yourcompany.net and click "Create", and select one of the templates. It scaffolds a Go service in your GitHub org, according to your company's best practices. All the builds and deployments are already set up and hot. You add some code to your service and shortly it's up and running and serving the pipeline output with caching out of S3. You check backstage.yourcompany.net to see how all aspects of your service's runtime status is looking. All good. Now, how do you actually get that useful data into your company's other devs' hands?

Well you COULD make an entire microsite, or some CLI tool orrrr... Come to think of it, they are already all using backstage.yourcompany.net, and there's already an integration in there showing all builds for all services, and it has all of the utilities and auth and bells and whistles already in place for you. So you make an internal backstage plugin. You find that it's pretty much a matter of cloning your internal backstage repo, running a create-plugin script, and boom you are up and running. You add a data source feeding off of your new Go service, you add a column to the builds table ... and you have your own team review and merge it because it's covered by CODEOWNERS. And everybody rejoiced and told you how useful this new column was to them. You start getting a bit famous on your internal Slack, actually.

Now you are getting greedy. You feel that actually, this plugin could be useful to the general public - as long as the build logs are in a known format, you can eat them. You start open sourcing the pipeline, and the service - and, critically, also the backstage plugin.

This is how things are internally at Spotify today. What we released here today is not this entire story yet, but we are lifting more and more out into the open at a rapid pace. Check the repo and microsite for more info, and hold tight.

More is coming.

Predictions so far:

- "to me it seems like it's a way of abstracting all the different infrastructure tooling that a large-scale company has."

- "It looks like an onboarding tool, monitoring, and knowledge database."

- "So it's a frontend to put in front of your existing frontends so developers can infrastructure the infrastructure?"

- "Seems like a developer-oriented version of Confluence, if I'm reading it correctly, but I'm also a bit unsure."

- "Seems like a competitor for Datadog and other monitoring tools to me."

That message really couldn't be more unclear.

A sort of meta lesson here for bootstrappers and technical founders is that explaining in simple terms what your product does is actually extremely difficult. YC hammers that point a lot too.

Good example here is the “open, plugin etc.” feature being almost right the top. I don’t care if it’s open or that I can make a plug-in if I have no clue what it does.

Also, start with the problem. Get people engaged first, then offer a solution.

For the people here that are confused as to what it does, to me it seems like it's a way of abstracting all the different infrastructure tooling that a large-scale company has.

Consider a team that wants to deploy something to the cloud in a company. They are probably engineers who aren't necessarily well-versed in all cloud tooling, yet they are required to know and understand Terraform, GCloud/AWS/Azure CLIs, Gitlab CI, Prometheus, various monitoring and alerting tools, Kubernetes, Docker, etc. And this is just to deploy their application. Once other resources come into play (databases, queueing, etc), they need even MORE knowledge.

A tool like Backstage allows developers to get a uniform overview of all their resources, regardless of how and where they are running. I'm not sure if it also allows the creation of new resources, but I guess that would be a fairly obvious direction to go in.

It won't solve the need to at some point know a bit more about the technical details about all the underlying tools, but it can allow teams to gradually learn about all this new tooling instead of having to know about everything upfront.

Borg, Kubernetes, et al? Abstracting complexity away is great, but at a certain level you just can't hide it, I think. Would love to be proven wrong, though.

This is by Stefan Ålund from Spotify, who was (is still?) product manager for System-Z. System-Z is Spotify's internal service catalog and developer productivity suite. (I'm not sure if they still use the name System-Z internally).

Here's an old talk about it (2016): https://www.youtube.com/watch?v=7XDA044tl8k

and here's some slides from a newer talk (2017): https://microcph.dk/media/1024/conference-microcph-2017.pdf

In a nutshell, the aim is to make it easy for developers to find out info about other services as well as make changes to the services they own.

It looks like Backstage is about open sourcing big parts of System-Z so that others can build on it.

Disclosure: I'm CTO @ OpsLevel, where we make a SaaS-based service catalog product.

I just skimmed much of their website, and felt like I was left still wondering what Backstage actually does.

It's a god system. It tries to do everything, but in the end does nothing well.

as a user of it, I beg to differ.

The Github repo was helpful:

- Phase 1: Extensible frontend platform (now) - You will be able to easily create a single consistent UI layer for your internal infrastructure and tools. A set of reusable UX patterns and components help ensure a consistent experience between tools.

- Phase 2: Manage your stuff (next 2-3 months) - Manage anything from microservices to software components to infrastructure and your service catalog. Regardless of whether you want to create a new library, view service deployment status in Kubernetes, or check the test coverage for a website -- Backstage will provide all of those tools - and many more - in a single developer portal.

- Phase 3: Ecosystem (later) - Everyone's infrastructure stack is different. By fostering a vibrant community of contributors we hope to provide an ecosystem of Open Source plugins/integrations that allows you to pick the tools that match your stack.

It looks like an onboarding tool, monitoring, and knowledge database.

Is it me or is this really unclear? I get it is supposed to simplify the back-end for developers but for what exactly? Scheduling tasks? looking at metrics? I get I can install plugins but what plugins? Maybe this isn't for me, but it is unclear to me precisely what scenario this is solving for?

So it's a frontend to put in front of your existing frontends so developers can infrastructure the infrastructure?

Exactly. This isn't a developer tool, it's a developing the developers tool. ;-)

The first read I had on this (which sounds like it's not correct based on other posts) is that it's a simply-extensible frontend for creating portals from a series of disparate frontends of existing systems.

In our case, I might want to use that to create a frontend for QA to see relevant information summaries from GitLab, build information from Jenkins, and current test plan status from our integration testing system.

For managers, I might want a frontend that shows them activity on certain branches of certain repos, the number of open merge requests/change orders, and the % of progress towards a milestone.

I hope I'm right, because that could be super useful.

I agree with other comments here about unclear.

The juiciest part is the animated gif, but all I want is a lot of screenshots and bit of text explaining each feature.

(Now I need to take my own advice and revisit my projects. hahaha)

That's really useful for big companies having problems with documenting their ecosystem without relying on tons of Word documents.

Looks like a developer portal similar to the AWS[1] and Azure[2] open source offerings where a non-developer can execute API calls in the web browser.

[1] https://github.com/awslabs/aws-api-gateway-developer-portal

[2] https://github.com/Azure/api-management-developer-portal

Seems like a developer-oriented version of Confluence, if I'm reading it correctly, but I'm also a bit unsure.

Sounds like similar to the pitch for https://buoyant.io/2019/11/18/announcing-dive/, which is in closed beta and I haven’t tested yet.

Any other competitors / SRE dashboard/metric aggregators that folks like? I think this is a level of abstraction above raw datadog/grafana dashboards.

Gitlab seems to be moving into this space a little with their deploy boards and k8s monitoring tools.

New thing aims to help developers deal with the overhead of having too many things to worry about, uses plugins.

It needs clearer messaging, but I think this looks great! Like a turbo CMS built for developers.

Integration with GCP/AWS is also great, I know plenty of people on my team that struggle with their UIs - this would really help them out

Wow, so a dashboard + backend? Or a framework? Or both?

Seems like a competitor for Datadog and other monitoring tools to me.

its not a monitoring platform (though we have our own in-house opensource tool for that), its a portal for developers and engineers to mange...everything. create new components, manage capacity and deployments, see your CI builds, find documentation, etc etc etc. ties all the separate systems and tools together into a single site.

This text should be copy-pasted wholesale onto the landing page :)

So it's an extensible dashboard / intranet site? A lot of people here are having a hard time understanding what this project is for. I think the README needs work.

Reply on news.ycombinator.com