Ending support for self-hosted Gitpod and moving our source to AGPL | Hacker News Reader

Ending support for self-hosted Gitpod and moving our source to AGPL

github.com

102 pointsghuntley65 comments

I understand why they've done it, but I now consider this DOA. Open source without self hosting capability is just crowdsourcing your engineering team. Documented and supported or bust. The cloud is a prison.

I completely agree. It’s a full non-starter for us to run this on someone else’s infrastructure.

I was going to be looking hard at GitPod in 2023. That is now entirely off the table.

And you're going to use what instead?

If the answer is "nothing" then you don't really need it so it was never on the table.

With your infrastructure you mean actual hardware? Bc we can run Gitpod Dedicated in your cloud account or ours, in the region of your choice.

What about multi-region for anyone who has a remote team?

I'm a random developer that would be using it currently for local use.

I cannot/will not consider a solution that results in you directly managing or hosting my code/projects. I know many companies that would also take issue because of this.

Also I understand you're pivoting your customer base to enterprises and not the random hobbyist - but obviously I cannot afford enterprise pricing.

That's why many of us are sad to see this no longer be an option for us.

We've been working on platforms in sensitive environments that cannot run in public cloud due to compliance requirements, so we have been transitioning our traditional on-premises solutions to private clouds with modern capabilities where the goal is to provide a great developer experience. We had plans on using Gitpod's on-premises solution, but with this change it is no longer an available option for us.

Strict export control restrictions mean we have actually been unable to use ANY cloud provider so far.

If you did run it in "our" cloud account you would still be on the hook for very expensive audits and background checks for everyone involved.

It not even close to worth it :/

same here :-( we do everything on-prem. Really disappointing.

Maybe I am misunderstanding but I think they are ending commercial support for self-hosted and the source is still available and you can run it if you want to?

With a project like this, it doesn't matter if it's technically open source if they don't make it practical to self host intentionally. There are so many components and nuances in such a complicated system that figuring it out or documenting it yourself through reading source is again, free engineering work for them.

Not supporting self-hosting is a cost-based decision, saying "well I'll figure it out and do it anyway" is just encouraging companies with open source offerings to act like this. Open source is only have the equation to make something self-hostable in any practical sense, the other half being accessibility.

This seems like an eminently reasonable thing to do, with the difficulties of supporting self hosted for a product like this on the infinite variety of customer setups. I can't imagine the time sink it must have been trying to support them all, it can't have been profitable.

It sounds like the dedicated version essentially lets enterprises run in on their own accounts while letting gitpod manage everything, which seems like a reasonable middle ground to me. Also sounds like you can still self host, they just won't support it directly as a customer, which again, seems reasonable.

They really shouldn't have to do much support. You can do self-hosted a number of ways: ship a VM, ship a container, ship a static binary, ship some automation (ansible/terraform/etc). It doesn't matter where it's run or how after that point, because it's a commodity building block. It's up to the customer to figure out how to set it up.

But once it's running, troubleshooting it is the same everywhere. Can you perform network operation X? Y? Z? Is the process doing A? Is the VM in state B? Is the VM's OS up to date and on version C/D/E? It's all just basic networking and basic Linux systems administration.

Other vendors do the same thing. If you know what you're doing, self-hosted should be a small, well-understood feature of your overall offering. The fact that they're going full-cloud is more likely a factor of just not having a strong engineering organization, and seeing more money coming from their cloud offering from small players that don't do self-hosted. Later on they will want the whale money and go back to self-hosted, or they'll get their lunch eaten by the other players in this market.

I think you may have missed that they do indeed let you sort of self host, in that you can run it on your own AWS resources. They just need access to manage it for you. Which I'd think would cover most orgs requirements out there that would actually use gitpod, but I could be wrong.

I think in this particular case it's maybe not quite as easy as just shipping a container or whatever, because a huge amount of the value and effort is actually the orchestration of VMs/containers/however they do it under the hood. Again, I might just not know enough here, but it seems more like they've constrained their self hosted option a bit to keep it manageable rather than getting rid of it in practice. It also seems like you could just use the agpl licensed open source code if you want?

Finally, it seems unfair to characterize them as having a weak engineering culture when they've managed to create something rather impressive with, I think(?), quite a small team.

> It's all just basic networking and basic Linux systems administration.

This might be possible in the fast and loose world of scrappy startups, but the bigger (and more lucrative) customers will have a lot more homework to do.

It requires a team to manage databases and a team (or several) to administer on-prem or cloud ops. They're going to have to learn this new thing themselves or meet with whatever team owns this third party thing. And have vendor meetings.

It's work and upkeep. At enterprise scale, this is one FTE bare minimum (probably spread out across multiple people), plus an oncall rotation (this is a pretty important business function), and definitely involves maintenance and upgrades that need to be coordinated.

Plugins, peering with external IPs, vendor security evaluation, BeyondCorp, certs provisioning, service discovery, traffic routing ...

This is all more work than "just", which is to say, I understand why the vendor is throwing up their hands and going after easier money. It's hard to support this, as each company does all of these things differently [1]. It just doesn't scale as well as SaaS does.

[1] Future business opportunity.

I can confirm, it’s a much more complicated problem than it appears on the surface (or at least based on other commenters perceptions of how easy it should be to do this). Where I was even used Replicated to try and reduce the hassle but even that is just a series of different trade offs.

So while our cloud/SaaS offering had a shared codebase there was still a considerable engineering team responsible for the “on prem” packaging and support. Alongside a pretty extensive CI pipeline to try and catch all the nuanced customer setups that had bit us in the backside in the past.

My day job is working for big enterprises setting up self-hosted solutions like this. Yes, it's work and upkeep, and they do need in-house specialists, but you already know that if you're self-hosting... that's kind of the point... If they didn't have in-house specialists they would use the dedicated managed cloud hosting.

The only purpose to self-hosting is because you can't use the dedicated managed cloud, such as due to regulations, contractual stipulations, networking limitations. If you're already in that situation and you don't have the headcount to manage this technology, you're screwed.

So, again, for the vendor, providing self-hosted is as easy as "here is a binary, here's some docs, good luck to you". If the client can't hack it... they shouldn't be doing self-hosted. It's self hosted, not we'll help you host it.

I don't know much about gitpod, but from reading the github repo, it appears they were deploying into k8s clusters with a fairly complicated application.

I suspect that is the issue.

At $curjob, we offer self-hosted versions, but our product is a monolith (in java, if that matters) that needs a RDBMS and optionally a proxy and elasticsearch. That architectural simplicity lets us offer self hosted solutions in a variety (deb, rpm, zip, docker image, docker compose, kubernetes).

But we're also an application component (auth server) not a full featured application. A full fledged application would be tougher to support, but if I were to try, I'd definitely start with a monolith.

CEO from Gitpod here. Some background on why we moved to a managed enterprise cloud product: there are parts of Gitpod itself that are closer to a Kubelet then a Kubernetes application. We use much of the Kubernetes surface, interact with containerd, and use bleeding edge Linux features. The only way you make Cloud and Self Hosted co-exist is to have one codebase. What we deployed in SaaS we wanted to deploy in Self-Hosted. But not all Kubernetes are created equal (think GKE node label, EKS custom AMI to get Linux kernels but not other places). Today there are features in SaaS that are not available in Self-Hosted. It is just not possible given the limitations of the managed Kubernetes services. In SaaS, we can vertically integrate the stack down to the exact version of the Linux kernel. In Self-Hosted we’re limited to the common denominator across the various services. Cloud development environments are going to change the way developers work. In order to do that they have to be fast, opening a new terminal sort of fast. We can only deliver that if we have full control over the infrastructure.

Link to the actual webpage, not the GitHub source: https://www.gitpod.io/blog/introducing-gitpod-dedicated

Link to the community standup which occurred moments ago that explains rationale and reveals the archictecture of their Gitpod Dedicated offering. https://youtu.be/B83zRiP9uk8

We were currently evaluating using GitPod in conjunction with our on-premise GitLab instance. This move by GitPod rugpulled the idea from us since we are handling national (non US) classified data and aren't allowed to upload that data or anything related to it to Cloud companies.

Edit: Misunderstanding, we are only evaluating GitPod, we already have a on-premise Gitlab instance.

Does code-server[1] meet your needs? It's self hosted and has been meeting by (albeit) hobbyist needs well.

[1] https://github.com/coder/code-server

Personally i always opposed new projects like this, since we already have a big devops stack that is taking much manpower to operate and kept running reliably. So, yes, the rugpull is not in the interest of my company, but it is in my interest because i will have less operational responsibility.

Maintainer of code-server here (:wave:). A couple days ago we did a recap of the ecosystem of code-server and how people are deploying code-server to support multiple users. See https://coder.com/blog/code-server-multiple-users

I assume GitHub Enterprise Server would also meet the requirements for being entirely on-premise and not uploading data to the clouds?

It does meet these requirements, by force. The network topology does not allow our on-premise instance to phone home.

Have you considered Coder? We provision software development environments via Terraform on your own infrastructure for Linux, macOS, Windows, X86, ARM, and of course, Kubernetes.

Our customers include financial institutions, asset management firms, hedge funds, US government agencies, professional services and outsourcing firms, retailers, insurers, software providers, and more who also confirm what you are saying.

We are seeing a move to clouds but only when the cloud is under their control where infrastructure is locked down to meet regulatory and security requirements. One prospect we were speaking with recently signed a multi-year public cloud provider agreement, but shared it will take a couple years for their GitHub Enterprise solution to pass tech risk and information security reviews before developers can even access it...

Over at https://coder.com/blog/how-our-development-team-shares-one-g... explains how we build Coder with Coder.

If you got any questions lemme know. Our source code (AGPL) can be found at https://github.com/coder/coder

ps. We are we creators and maintainers of `code-server` btw.

> Our source code (MIT) can be found at https://github.com/coder/coder

https://github.com/coder/coder/blob/v0.13.1/LICENSE

> GNU AFFERO GENERAL PUBLIC LICENSE

Is there some licensing subtlety that I'm not seeing?

No, you aren't wrong. I fucked up whilst multi-tasking (see seperate post below). Coder/Coder is AGPL. Coder/Code-Server is MIT. Doh. Thanks for the correction have updated the post. :)

GitLab team member here.

Sorry to hear about your troubles. GitLab is working on adding remote development into the DevSecOps platform. I suggest reviewing the direction page [0] and implementation planning in the main epic [1] (with child epics and issues), and add your feedback and thoughts into the epic. (via the direction page in the 'how you can help' section [2])

[0] https://about.gitlab.com/direction/create/editor/remote_deve...

[1] https://gitlab.com/groups/gitlab-org/-/epics/7419

[2] https://about.gitlab.com/direction/create/editor/remote_deve...

Co-founder of Nimbus here (https://www.usenimbus.com/)

We also provide a remote dev environment solutions. I think in your use case is very special as you cannot leverage any cloud providers, which can potentially make it harder for you to integrate with certain solution.

We are looking for design partners. If you like to share more and have a discussion, I am very happy to learn more about use cases and make Nimbus work for your on-prem infra.

Interested to see if they can successfully pivot to full SaaS. It seems like with the recent AWS announcement of CodeCatalyst and Github CodeSpaces (both of which are free with an additional easy to use paid model), Gitpod has been backed into a corner. I hope they do well, but the odds are stacked against them as the enterprise selling machines that are Amazon and Microsoft are incredibly difficult to fight as a startup. With Gitlab also apparently working on their version of CodeSpaces, it seems like maybe the best position for Gitpod is an acquisition (possibly by Atlassian).

> It seems like with the recent AWS announcement of CodeCatalyst

I took Code Catalyst for a spin the other day and it is terrible.

https://twitter.com/GeoffreyHuntley/status/15997732931288637...

> It seems like maybe the best position for Gitpod is an acquisition (possibly by Atlassian)

I agree. I used to work there and think they need to slash circa 40 employees to shrink down to more sustainable burn rate numbers - https://ghuntley.com/tea

> but the odds are stacked against them

In more ways then one https://ghuntley.com/fracture documents Microsoft's strategy with Visual Studio Code which means 6 out of 10 top programming languages integrations only work on GitHub Codepaces. If you want .NET, C++, C, Python or Jupyter then the LSP integrations cannot be legally used on Gitpod.

GitHub Codespaces got one thing right - they chose IaaS as the brick of compute meaning they can do GPUs and run Kubernetes clusters. On Gitpod, you can't because Gitpod chose Kubernetes as the compute abstraction. Gitpod needs to ditch Kubernetes...

https://github.com/gitpod-io/gitpod/issues/4889 and https://github.com/gitpod-io/gitpod/issues/10650

How is Coder going to compete against GH/MSFT?

Hmm. We find that GH/MSFT customers come to us (especially Microsoft Azure DevOps customers which have been abandoned) and GH is categorically providing non-answers of "we are investigating offering on-prem" but they have done that for a long time now - ie. a classic sales tactic to keep MSFT aligned people hopeful vs saying no.

Microsoft has been retiring on-prem products as a business strategy for over 12 years now. Why would GitHub be the snowflake exception? Somewhere, in some CVP's mind GitHub ES is already EOL (look at GitHub.com/enterprise they are already muddling the meaning of "enterprise")

If you look at /.codespaces/bin or /workspaces/.codespaces/shared on GitHub Codespaces you'll understand how tightly coupled GitHub Codespaces is to Microsoft Azure thus on-prem isn't going to happen. The likely path they could do it would be like JetBrains Spaces does it "configure in our IAM/ARM key and we will provision Azure/AWS virtual machines" via GitHub Enterprise Server.

$ ghuntley /workspaces/.codespaces/shared $ grep sku * environment-variables.json: "skuName": "Standard_F2s_v2",

If you want further information check out https://coder.com/blog/github-codespaces-coder-and-enterpris...

MS have a VS code server which is in private preview now. it's very much like a self-hosted Codespace if you squint and write a tiny bit of automation around it.

license forbids using it to host a service for others, but you can host it for yourself.

> the LSP integrations cannot be legally used on Gitpod.

Can the user install them? How complicated these are? Is there some awareness about this in OSS communities? If yes, is there some kind of consensus? (Is it something like: okay, then each programming language project needs to provide its LSP, like Rust has rust-analyzer, Scala has Metals, etc..?)

> Can the user install them?

Doing so is not legal. Here's the standard boiler plate license for the LSPs

> You may install and use any number of copies of the software only with Microsoft Visual Studio, Visual Studio for Mac, Visual Studio Code, Azure DevOps, Team Foundation Server, and successor Microsoft products and services to develop and test your applications

https://marketplace.visualstudio.com/items/ms-vscode.cpptool...

The key here is VSCode (MIT) is NOT Microsoft Visual Studio (the product) and only the product can use the LSP's, tools such as GitHub Copilot or the Visual Studio Marketplace Ecosystem.

> Is there some awareness about this in OSS communities?

Yup

https://github.com/search?q=repo%3AVSCodium%2Fvscodium+pylan...

https://github.com/VSCodium/vscodium/issues/1020#issue-11802...

> is there some kind of consensus? (Is it something like: okay, then each programming language project needs to provide its LSP, like Rust has rust-analyzer, Scala has Metals, etc..?)

The consensus is that programming languages need to author their own LSP integrations and not allow Microsoft to build it. It's why rust-analyzer, go, metals are self-developed by their communities.

Again https://ghuntley.com/fracture recaps the problems w/VScode and how Microsoft is building a GitHub 365 vision through VSCode.

The actual repo seems MIT though https://github.com/microsoft/vscode-cpptools/blob/main/LICEN... ... what is missing from this?

CEO of Gitpod here - we just announced a $25m Series A last month (https://www.gitpod.io/blog/future-of-software-cdes), so not looking to get acquired anytime soon ¯\_(ツ)_/¯

On the market landscape: the whole market around CDEs (https://www.gitpod.io/cde) is in its infancy. Our largest competition is local development. We believe that unlike for other creative workflows the convenience threshold for development in the cloud has not (yet) been crossed. We are moving away from self hosted to execute faster on what we always wanted: from history-dependent cobbled local development environments to consistently reproducible, instant, ephemeral Cloud Development Environments (CDEs). You can think of CDEs (https://www.gitpod.io/cde) as our product north star.

Other links:

- How Redmonk looks at the market https://redmonk.com/jgovernor/2022/12/01/the-year-of-the-clo... - Kent Beck https://medium.com/@kentbeck_7670/cloud-development-environm... (full disclosure: Kent invested in the meantime into the company)

in my dreams they get acquired by Jetbrains :D

me too then the stock in Gitpod that I own might become liquid :)

Overall, I am glad that this shift in direction is happening for Gitpod. They are still leading the CDE movement (IMO) and this gives them a better chance at continuing to do that. I don't want this space to collapse into just use codespaces, it's mediocre, but whatever because innovators businesses couldn't keep their products alive long-term. Official support just doesn't feel like it was ever a money-maker, nor is it something that the target audience would really use IMO, at least not now.

100% agree.

Still self hosted alternative https://coder.com and Eclipse Che https://www.eclipse.org/che/.

Both of them allow you to host VSCode or Theia on your own infrastructure.

> The open-ended requirements to run on commoditised Kubernetes services (GKE, AKS, EKS) forced us to manage variance and prevented us from driving innovation and fully realizing the potential of Cloud

Really? Looks like they shot their own foot there

"On-prem" usually doesn't mean "running on AWS but managed by us". Sure, some people will want it, but not most of them.

I think Gitpod is a great product, and I've had a massive amount of use of their free tier. There's something pleasant about throwaway Ubuntu development environments. I can cover pretty much all of the major development use cases in it, even container work.

I just hope they survive long enough to take off as they're giving a lot away for free currently. Their dedicated instance setup looks like it'll cover the majority of businesses who'll likely have their own landing zones and cloud controls they can integrate into. Not being able to self host is a loss, but it might the right trade off.

I don't understand the AGPL bit well other than it was seen like poison by corporate risk types at my previous jobs.

Why doesn't this link to the main website? GitHub is not a blogging platform and this is a mess to read.

How is this "a mess"? Two first things (table, html/js) is a bit ugly, but the rest reads completely fine. Where is your limit for when something "becomes a mess"?

My limit is when the links don’t work. Which they don’t because they expect this to be on the website.

In addition to the links, <script context="module"> export const prerender = true; </script> <script> import Signup from "$lib/components/dedicated/signup.svelte" </script> is noisy

do any of the links from github.com/gitpod-io/website/blob/main/src/routes/blog/introducing-gitpod-dedicated.md work...?

Here's the real site link, I don't know why the poster here used the source markdown.

https://www.gitpod.io/blog/introducing-gitpod-dedicated

Great product, sounds like a reasonable move. Hope Gitpod can keep growing and improving even more

Yup, it is. I'm glad they have finally made their mind up if they are a on-prem company or a SaaS company. Who you hire and how software is built (and sold) is a fundamentally different approach to market. For example - it doesn't make sense to hire product managers from Slack if you are building on-prem (they don't have experience // product managers from RedHat should have been hired for an on-prem play)...

Is there enough of a community around Gitpod for it to be supported that way?

Unfortunately no. It is stupidly complex to be able to contribute to Gitpod as the build tooling (werft) automation is only accessible to employees thus a community never formed. During my time there I tried to get this resolved and implement some form of automation around community contribution w/CLA but both of these problems went no-where/were not prioritised.

Have opened https://github.com/gitpod-io/gitpod/issues/15254 for discussions re: if CLA is still needed after todays announcement.

I hope Gitpod can continue to be successful given Microsoft seems to be doing their best to work against them through the horrible licensing of many popular VSCode extensions.

Agreed. Gitpod is the best chance of unfucking the VSCode LSP ecosystem but it's going to not be cheap or easy to do. 'pod will need to reauthor any of the LSPs that Microsoft developed but even if they do it won't resolve the dependency graph issues of Jupyter Extension depending upon Pylance Extension. As Microsoft goes up the stack past LSPs building more IDE chrome it is just going to get harder as graph is going to get more tangled...

i dont understand. gitpod self hosted is gone and source is agpl so can someone continue a fork of gitpod self hosted?

I think the point is that they no longer support self-hosting. You can still self-host it, since it's AGPL after all, but they probably won't help you nor document how to self-host. This effectively pushes enterprises from self-hosting to their new Dedicated offering.

Correct.

> This effectively pushes enterprises from self-hosting to their new Dedicated offering.

Or, of course, into the arms of their competitors.

Maybe. But it sounds like their current enterprise customers were tired of managing the complex environment that the application requires. So with that, this seems like a good move by them. Probably not for everybody, of course. But this positions their enterprise business model from on-prem to SaaS, which from the blog post is exactly what they want.

Reply on news.ycombinator.com