Spotify getting into enterprise IT tools | Hacker News Reader

Spotify getting into enterprise IT tools

techcrunch.com

117 pointsiamzycon63 comments

I’m not sure who the customer is for Backstage.

For a small startup, poking around in the gcp/aws console works well enough. Maintaining another console layer on top of that feels like one more thing that could break.

If you’re a big company with a team that could support developer platform tools, you probably have some sort of portal system hacked together and you’ll never be able to change because somebody has a critical dependency on the old system

The big company use case is the ideal backstage use. Backstage is just a framework. you can use it to interface with the old system behind the scenes and provide a nicer layer on top for your application devs.

Glad I'm not the only one confused. Even the product purpose seems vague...

> Libraries, websites, ML models — you name it, Backstage knows all about it

This is in reference to the software catalogue feature.

Backstage is great at slowly migrating dysfunctional organizations into slightly less dysfunctional organizations. I deal with 30 years of random tech debt in a company with immense shadow IT (domains randomly registered, vendors randomly hired to do X, often nobody knows things are still needed, our zonefiles are a mess, unclear responsibilities with our team of 4 being the default fallback), and Backstage at least puts some semblance of insight into that entire mess and provides a way forward to formalize such processes. Also, putting documentation where developers commit to every day instead of a Wiki where it's out of sight / out of mind is very underrated.

But doesn’t that mean you have a long migration period with two systems to maintain and reconcile?

Yes. I've seen this go well, and I've seen this go terribly. It works (in my experience) when the migration work is pushed down to the directly responsible teams - with, of course, affordance made for the time it will take them. (Sidenote: that has the benefit that they learn / re-learn their dependencies for themselves, and discover efficiencies they never would have otherwise.) An overarching "migration team", or whatever you choose to call it, only adds friction, confusion, and more kludges. Management has to be fully committed, and enforce deadlines all the way down the line. Running two systems can (usually) be done short-term, but not long.

(Another thought: "short" and "long" terms are variable for different teams and different types of services. Scheduling should take that into account.)

Admittedly, however, I've not been through it at a particularly large organization. I'm sure there is exponentially metastasizing complexity (human perhaps even more than technical) once the process involves more people than can fit in a room.

In such dysfunctional case the truth is that you do not have an old system to reconcile with - there's no real system of record

Companies grow - today you have 50 engineers and a few microservices, in 2 years you might have 200 engineers and tons of APIs and web applications. I think if old orgs could have bought off the shelf developer portals years ago, they would have. Now companies that grow have this option.

It might not be a massive market, but there are a few interesting companies building this kind of tools and they sell it

We were that big company (50,000 employees, probably about 10,000 or so in technical roles across the enterprise) that had a bunch of custom portals and internal sites for all our dev stuff, and also Stoplight (Spotlight?) for APIs etc..

We were early adopters of Backstage, back in 2020. We've migrated almost all of our portals and "meta-infrastructure" to Backstage, and it's certainly not perfect, and not everyone has moved over, but the majority of our teams are in one service catalogue and people can much more easily find service owners, dependencies and more.

It's been a great tool and a pretty big internal success story for us.

But this move from Spotify tells me that they are no longer stewards of an open source project, they now use open source as a loss-leader for their paid offerings.

Which is totally fine, but it will change how we look at that relationship.

The customers for Backstage aren't startups but mostly big corporations with hundreds of developer teams. Those corporations want to get rid of these hacky internal solutions and many don't even have one.

Yes, and it’s not just poking. You have to actually code lots of stuff from scratch and maintain it. Feels like you almost need a full time person maintaining it.

It is tough to keep two widely different businesses under one roof. One is a mature, public B2C, and the other is a dev tooling startup.

Spotify should spin off Backstage as a separate company, providing the initial IP and funding and being the first client.

Many companies, like Uber, created a lot of useful infra, though they failed to monetize it. They could take a cut in the Chronosphere (M3) and Temporal (Cadence).

Much better than laying off people, though origin companies would need to be smart, without being too greedy.

Though the model is unproven.

> One is a mature, public B2C, and the other is a dev tooling startup.

Amazon/AWS?

Amazon is an outlier, where each business runs its profit/loss, and higher-ups act as VCs. Wish more companies were like that, but you need to take a lot of bets to make that work.

AWS worked, but the Fire phone turned out to be a write-off.

Fire Phone was largely outsourced I’ve been told. Was big regrets all the way to the top about that decision. Lesson learned.

If anything, AWS looks like the outlier compared to their other projects. It only survived because of the sheer size of Amazon.com and was built to accommodate that traffic and deployment. If they set out to create AWS without the need for it, don’t think it would’ve made it this far. Even after all these years, AWS is the breadwinner, while the website it was made for can barely turn a profit and resorts to increase the price of prime every year to show revenue growth.

I think you are describing spin-in's. Made famous by Cisco. Didn't catch on much.

Many companies, like uber, use their other more profitable business units to subsidize other business units. It doesn’t always make sense to spin out a company / product

Backstage is an open source CNCF project. It needs a corporate sponsor to keep it alive.

Has anyone successfully implemented Backstage at their company? How'd it go?

We're looking at it for an internal platform serving ~50 engineers today, spanning mobile, backend, enterprise integrations, and ML teams. The hope is it'll make it easier to spin up new things "the right way" without e.g. ML PhDs having to learn Helm and K8s, plus other goodies like nice docs.

We use it, and apparently it’s great. It didn’t help with the silos though because I don’t have access.

Ive watched a couple of botched attempts at implementing it and just like with other spotify fads you need to be all in or not at all. Also im highly sceptical it is actually going to “break your silos” or whatever they are promising because silos usually come from the management and the culture not the type of idp management happened to cargo cult on you

It depends how you define successful. I work in a large org, and it work well for governance, and as a console, but it has its limitation. Pushing too many plugins on it makes management of the console hard

We're 30-40 developers, and we have just stopped using backstage. We started using it for APIM behind Okta SSO, but all the custom frontend and functionality tweaking needed to make it look nice was daunting.

If we start looking at it again, I'd try a simpler use-case, but definitely interested in trying it again.

We piloted two jobs ago and aborted. Wouldn't recommend at your size because it seemed to do effectively, you need a "Backstage Team" which is not Ops Team who occasionally does Backstage work. Probably why Spotify is considering it because sweet consulting money.

While I don't know your exact situation, I'm currently at company of your size engineers and stop torturing your Developers with Helm is off the shelf advice I would give. See if Kustomize would get the job done and be honest. I've found Devs can handle Kustomize a ton easier since kubectl kustomize > testoutput.yaml is alot easier for them to parse.

Helm is great if you have a bunch of different things to launch with a ton of different options but most companies don't do that. They need HPA/Deployment/Service/Ingress which is easily handled by Kustomize.

I'm somewhat new to helm, after a year I feel like I'm pretty good at it, but struggle to teach other ML folks what I've learned the hard way over the last year. Some of that is helm, plenty of it is Kubernetes. Most of our charts are ingress, service, deployment, service account, scaled object (keda), but there's plenty of small variation even across ML applications, plus the different needs of other teams.

Any good resources you can point me to on how Kustomize might be easier for us?

Documentation is best: https://kubectl.docs.kubernetes.io/references/kustomize/

However, biggest is it's just YAML patching. So you don't have this {{if such}} littered throughout your project that people can't easily pick up. Values is biggest reason I see people want Helm but if it's all env variables, kustomize lets you put in .env file it turns into config map and injects into deployments.

It's possible Kustomize doesn't fix your problem esp if Developers have decided it's not their problem. I just find Devs find Kustomize much easier.

Helm is what you should use as your cluster "package manager" to install all of your favorite off the shelf tools. Kustomize is probably a better choice for shipping your internal projects due to the variations you mentioned, but also because the learning curve isn't much steeper than learning about how to work with plain Kubernetes manifests.

we use it through roadie.so but it doesn't feel very polished compared to other developer portals...

I’ve introduced it at a company with ~3,000 engineers and it’s been quite successful, though we do not yet leverage all of its power and we do have a dedicated team to run it and build on top of it

We have a lot of in-house tooling and Backstage allows us to move big chunk of UIs into one place - it helps us keep UIs consistent, and makes it easier for everyone to find the tools we have. However with a lot of older, more mature in-house tooling it’s been a struggle to migrate to their data model, introduce their software templates etc.

For an org with ~50 engineers I’d recommend something more off-the-shelf like Cortex, Port, or OpsLevel

We are trialing it. Setting up VPN connections, firewall rules, automated secret rotation, API key creation for centralized external services, DNS entries for private endpoints, new team onboarding, new developer onboarding, anything time consuming repeatable task that usually takes our ops or sec teams used to deliver manually in support of the various tech teams to deliver their apps into our clusters. The various “plugins” create PR’s for the ops/sec team to double check and approve to keep the auditors happy. Servicing ~500 developers. It’s definitely a platform that needs a dedicated team though.

My company uses it. Our team is about the same size. I have mixed feelings. It has been useful for things like making sure a new repository is created with all of the CI goodies every project should have. You can also use it to automate things like opening PRs against your gitops repo to get apps deployed from a template.

Creating software templates can be a large investment. It would be great if there was a software template marketplace. The backstage contributors took a stab at this by creating a template repo that has not been updated in about a year. https://github.com/backstage/software-templates

If you do invest in creating a template, be sure to get plenty of stakeholder feedback so the thing you make is something that actually gets used.

Look into projen. It’s not as heavy and doesn’t require any services. But makes it easy to template and spin up projects. But the key is that templates are living and updatable. If you want an intro and a free consultation reach out via email in my profile.

The 'single pane of glass' is pretty seductive and there are a lot of tools that try to sell that vision. Backstage looks a pretty good swing with their modular/flexible approach, though I haven't used it.

The problem you end up having is that (for example) Google docs is really good for writing and commenting on docs, and a general purpose tool is never going to be as good as Google docs for docs. You could write a "Google Docs" plugin and just link out to a project's google doc but then you're just a link aggregator which offers limited usefulness.

  Backstage is built on a modular, plug-in based architecture that allows engineers to layer-up their developer portal to meet their own needs.

Frankly, this is just incorrect. Just read the configuration guide for "installing" a plugin to see what I mean. You have to modify the source code of your own deployment in several places. That's not a plugin based architecture.

https://backstage.io/docs/getting-started/configure-app-with...

Additionally, check out what it takes to stand up a standalone backstage server. You used to have to clone their repo if I remember right, but now they give you a starter template via npx and leave it up to you to put all of the pieces together. They don't even ship a docker image to get you started.

https://backstage.io/docs/getting-started/

Frankly, the architecture of Backstage is ill-conceived and a hassle to operate. If you have the budget their are vendors that will deal with all of this for you. I think the concept of Backstage is fine, but the execution leaves a lot to be desired.

I did not expect that. So Backstage asks you to manually insert plugin components into the app source code? Yeah... that really doesn't feel like a plugin architecture to me. More like patch the source code so it does what you need it to do. Nothing wrong with that, but not a plugin architecture.

The trouble with the way Backstage is designed is that it makes keeping everything up to date difficult. It is also harder for them to ship updates for their core and plugins due to the labor involved in incorporating any updates.

A plugin architecture like Wordpress or Grafana is much more manageable for an operator. In fact, a set of plugins for Wordpress could probably get you an equivalent solution, but it's PHP, so you know, ew. I'd love to see an alternative to backstage that gets it right.

The "Featured Article" at the top means it's a sponsored post, right?

and the account that submitted here has zero comments.

Somewhat related: the dev behind replyguy said he is working to add AI astroturfers to HN comments as well. I'm curious to know what type of mechanisms (if any) will be adopted to detect and neutralize them.

Making them buy immunity.

But how do we know you aren't just one of replyguy's bots being used to promote his service right now, eh?

To take this seriously, probably by account age, there just aren't that many 2007 accounts so eventually astroturfers will have to resort to newer and newer accounts.

people who are using DataDog, how does it compare to DataDog service catalogs? If I want to use Backstage what kind of value add it provides over observability tools with cataloging tools?

The scaffolder is the part Datadog won't get you, and in my experience it is the most useful feature.

I saw link to backstage some time ago, but I could not figure out what it's supposed to provide. Can someone explain what it does / what do you use it for?

In a past job of a fairly decent sized org (around 80 engineers, maybe 10 different active projects, a few dozen microservices, component library, template generators, and some custom CLTs) it was very useful as a way of documentation for your engineers.

I'm sure you used stuff like confluence but it was always a terrible way to search across code bases and projects to understand how they worked.

Backstage was interactive documentation. I can see what projects we have, I can click on one, see who is maintaining it, what their communication is, what their various perf/benchmark scores were. It was nice.

Apparently it's even grown beyond documentation needs from what I can tell and it provides platform tooling now.

By default it offers you a few tools to better understand and manage your software landscape:

- you have all your micro services catalogues in one place, with a fairly decent data model (services are grouped into systems, they are owned by teams, they have declared dependencies on each other)

- you have documentation for each service that you can easily find

- you have a template catalogue that allows you to easily create new services. You need a full-stack app? Here, recommend option is Java + React, use this template. You need a new ML system? Here’s what you should use, click “create” and we’ll bootstrap it for you

- plus a few more, like customizable home page

If you are looking for a way to start cataloguing and standardizing services across your org, the developer portal (whether Backstage or other solution) is a good thing to have

To me it looks like a SharePoint for engineering teams.

With plugins focused around projects/deployments/metrics.

Which sounds actually interesting.

I’ve always thought Backstage seemed like an interesting solution to existing god awful “solutions” many enterprise type companies subject their devs to. For example, PostHog, Swagger.

What's bad about PostHog? I haven't tried but I thought it looks good

What’s wrong with PostHog? Do you mean Postman?

I think you have the wrong product; PostHog is for startups, not enterprises.

try tell that to the people insisting we need it

I think one of the largest problems they will face is their consumer not having the same culture as them, and what worked wonderfully for them failing in other orgs.

Yes. The world needs another one of these...

My employer is considering using backstage for our data infrastructure tooling. This makes me nervous because Spotify seems unable to program a reliable shuffle feature on their player. Maybe it's because they put all the good devs on backstage?

Spotify's shuffle algorithm can't be changed easily because too many people rely on its existing quirks. Which means that Spotify's ability to keep it at a reliably stable level of unreliability for so long, through so many other platform changes, makes it a prime example of why Spotify is a perfect choice for enterprise! ;)

It's true; they make UI changes that no one wants (like pushing the queue off to a sidebar that can't be expanded to the way it used to be), constantly try to push features I don't want (some weird "DJ" feature that has an actual programmed voice between songs, "collaborative" playlists when I pretty much exclusively listen alone with headphones), and refuse to implement extraordinarily simple features that would be universally popular (like being able to mark certain songs/artists as "don't ever play this for me please"), and yet the few other players in the space all have similarly mediocre products that give me no real choice. This is peak enterprise UX.

Please don’t use “IDP”, we really already have that (for chrissakes we have like four of them in our “enterprise IT”). They can have SAML also, “Spotify: Acronym’s Mine, Losers”.

Reply on news.ycombinator.com