Future of OSU Open Source Lab in Jeopardy | Hacker News Reader

Future of OSU Open Source Lab in Jeopardy

osuosl.org

289 pointsaendruk102 comments

I worked at the OSL as a student years ago, and it was one of the most impactful places I've ever worked at. I learned a lot, and I wouldn't be the engineer I am today without having worked there.

Since graduating, I've also hired, and worked with multiple alumni from the OSL and they're always top notch. Anyone looking for interns or new graduates with devops/SRE or SWE experience should be looking at the OSL for talent. It's not too often you can hire a new graduate with potentially multiple years of production experience, especially in devops.

In context of HN/Y Combinator, https://www.ycombinator.com/companies/coreos was a successful container/Kubernetes focused startup founded by two OSUOSL alumni, Alex Polvi and Brandon Philips, which was eventually acquired by Red Hat.

The OSL is something special.

For a list of projects the OSL helps host, check out https://osuosl.org/communities/. You might see a project you care about in that list! As an example: they provide aarch64 and powerpc VMs for a ton of projects to do their CI/builds on.

Same. I helped out/worked there out of high school(back around 2010).

One of the best experiences of my life.

I still prefer to use the OSL for my linux repos.

The OSL was transformative for my career as a budding CS student in Corvallis many years ago. I can’t say enough good things about the positive impact it has on the Open Source community and the students it employs.

In my experience, there isn’t a great on-ramp for learning to be a SysAdmin (or devop, etc) in a practical sense. Learning what it takes to support systems in “Production” with actual users, and all that entails, at some point requires a hands-on approach. Finding entry-level opportunities to do that isn’t easy until you have /some/ experience. The OSL provides that, and supports countless FOSS projects in the process. It’s really a great arrangement.

Obviously I’m biased, but the Open Source Lab should be viewed as one of the Crown Jewels of OSU.

A lot of the fun parts of the computing industry have, predictably, been hollowed out by the rent seeking model of cloud and *aaS. There is some grace as it's easier than ever to build some scalable web business.. but the most fun of my career was rabbit holing on computers for the sake of computers.. working on operating systems and device drivers and network stacks. And it did and still does matter to a lot of bottom lines, but corporates have a hard time connecting the dots or doing something other than what the flock is doing.

It's a little awkward because the AI datacenter boon is a little bit of a revival for physical and systems work but it is limited to that and I am skeptical of the longevity.

Those days of having fun working on network stacks, operating systems, setting up FOSS development labs and being a good steward of things.. harder and harder to do and even harder to get started.

When I was working on GHC many years ago OSUOSL helped us by providing us access to some nice POWER7 machines (courtesy of an IBM kernel hacker who recommended and endorsed us) and we used them for years to solve weird issues. I've always thought very highly of the Open Source Lab. I hope someone can help them make it through this.

I was always happily surprised to find that they were hosting what I needed when I needed it.

A great lab with a long history.

The Open Source Lab was a fundamental part of my college experience. I would not be the person I am now if not for the experience gained while employed there. It was such a great feeling to help hundreds of open source projects maintain infrastructure and services, especially some of the larger projects which have colocated hosts

A gap like cannot be closed without corporate and foundation gifts. That said, individual gifts can also contribute to the progress.

As is common with schools, parks districts, etc., the Open Source Lab partners with a 501(c)(3) organization, the Oregon State University Foundation, to accept tax-deductible donations.

For anyone who would like to directly support the Open Source Lab in staying open, please be sure to indicate "Open Source Lab Fund" on the Oregon State Foundation donation page [0]. Note that their form is *not* set up with any tracking to attribute your gift from your clickthrough, and that any general donations to the Foundation will likely *not* support the Lab in this effort to stay open.

[0] https://give.fororegonstate.org/PL1Uv3Fkug, or click through from the general donation page.

Oooo dang, the lack of an "opt out of future emails" checkbox really gives me pause there. Charitable donations are second only to vehicle purchases in the avalanche of shit they trigger.

Welp. Did it anyway. There's a "make my donation anonymous" checkbox on the second page, but that doesn't actually specify whether it opts out of emails, or merely hides my name from some list. Well, it's worth it either way.

Use the hyphenated email filter trick.

What is this trick?

https://danq.me/2017/09/26/gmail-plus/

I thought it was a plus sign?

I was with Mozilla when OSU's OSL was supporting the mirroring/serving of Firefox and Thunderbird globally. They were a key mirror/supporter during the early days of Mozilla and definitely contributed to the growth of Firefox.

Jensen is an OSU alum--it would be nice if this reached him.

OSUOSL and Lance specifically (the writer of this post) was extremely supportive of me during the early days of Vagrant and Packer. Lance tried many times to try to find a way for OSUOSL to help my projects but I don't think we ever formalized anything.

Regardless, they were always big users and big proponents of the OSS work I was doing. And I remember that. I think more than the OSS project support they do, the support and education they help provide for students is laudable.

I personally think corporate sponsors shouldn't blink twice at supporting OSU OSL, but I'm not surprised given the state of... things. And the individuals choosing to judge and criticize based only on a 4 bullet point budget are infuriating.

Well, I'll help. I've emailed to setup a donation.

Thanks for everything you've done Lance, OSUOSL. And thanks to anyone else who helps support them!

As someone who was a student at the OSL when Vagrant was hip, also thanks to Mitchell for creating Vagrant! We used it a ton for testing all our our configuration management.

Lance is a great guy and that lab for a decade plus has done great work and supported so much of community. Happily supported their start and will continue to.

They are part of gnu compile farm which donates compute to open source projects.

I used them a lot when I worked on OpenSCAD build system, there weren't a lot of places 12+ years ago you could go 'make -j 30' on a PowerPC or 'ctest' and have it run dozens of builds/tests in parallel. Really helped alot, that C++ template stuff would barely build at all on my personal machine.

Sorry to hear this

I feel like that shouldn’t be impossible to raise. I would be more than happy to donate $250, now we just need 999 more to do the same.

"I have reached out to our largest corporate sponsor and they are working to increase their support as we update our contract, but that still may not be enough."

Why wouldn't they name the sponsor? Seems like a great thing some company (presumably Google?[1]) is doing, but they don't get the credit they deserve.

[1] "Google sponsors the OSL through financial grants." (Noting that none of the other sponsors seem to provide financial grants) https://osuosl.org/sponsors/

They should reach out to IBM too, since POWER and Z hardware are mostly an IBM thing. ARM should be willing to help too, since they host AARCH64 stuff too.

"Below are groups who supported the Open Source Lab through annual contributions of $25,000 or more during fiscal years 2019 and 2020."

It seems that all of them have contributed financially (though of course, it's unclear which ones do so at present).

"annual contributions of $25,000"

I think includes hardware/service contributions (eg the IBM hardware and the bandwidth suppliers). It's a bit unclear though.

Give them the credit after the sponsor actually finalises the updated contract.

I help to run the OpenStreetMap infrastructure, and for many years the OSUOSL has generously hosted some of our servers.

Without OSUOSL OpenStreetMap would be more difficult to host and significantly slower to access from North America.

I hope OSUOSL can get the financial support they rightfully deserve.

I didn't work at OSL directly, but I did work for the sibling organization (then called BSG, now called CAAS) and it's sad to see this. I donated and would encourage other reads to do so too.

OSL and the sibling organizations at OSU are such an incredible program for students, and OSL in particular has had an outsized impact on the OSS ecosystem relative to its size and cost.

Moreover, it's institutions like this which represent some of the best of the software world, and we can't let them fall by the wayside as the capitalistic focus of larger companies just gobbles everything up with little regard for what they destroy.

OSU OSL provides CI machines for some of the more exotic architectures like Linux on Z and POWER to some ASF projects. It would a loss to close it down.

Maybe some unicorn billionaires could spare a few millions? Especially the ones who built their wealth on top of open source libraries or databases.

OpenAI should fund them - they're obviously very heavily behind Open Source, as it's what they were created to do.

https://help.openai.com/en/collections/12496919-open-source

Ah.

I find it odd that they can provide all the infrastructure compute / storage / bandwidth for all of these projects for $35K

Debian and Fedora on their own must be highly demanding??

Form the article: """ Currently provides infrastructure hosting for projects such as Drupal, Gentoo Linux, Debian, Fedora, phpBB, OpenID, Buildroot/Busybox, Inkscape, Cinc and many more! """

Most of Debian's infra is elsewhere, but OSU does host a backup server, logs server, a bugs.d.o frontend, POWER/MIPS/HPPA/RISC-V build/porting servers.

https://db.debian.org/machines.cgi

Similarly, I assume RedHat hosts most of Fedora's infra and OSU does POWER/IBM Z stuff for them.

I would've assumed that Red Hat/Fedora are well-catered for now in terms of POWER and Z, considering who now owns Red Hat.

You'd be surprised, this is one of the ways in which Red Hat stays independent from their mothership. IBM prefers to donate to OSUOSL and Red Hat uses it for Fedora.

Then there are private labs, internal to Red Hats, that are used by developers especially for testing, but they're not stocked particularly better than they were prior to the acquisition.

Bandwidth is donated by TDS Telecom. I suspect that physical hosting is provided by the University and servers seem to be provided by IBM.

They need university budget for items that aren’t donated in-kind rather than as cash by sponsors. Build servers can be donated by granting OSU access and paying the bill. But someone has to manage those donated services, and none of the existing donation agreements apparently include cash on table to pay for that headcount (since they did not until this year).

My understanding is that a lot of ISPs use linux mirrors as an offset for asymmetric bandwidth, which can lead to one-sided network charges. As such, they're happy to donate bandwidth to the degree it benefits their negotiations elsewhere.

Link to donation page: https://osuosl.org/donate/

Am I reading correctly that of the $250K they need, $150K of that goes to a single staff member for 60% of their time? Does that seem...excessive?

No, you are not reading correctly.

The 60% number is the percentage of the budget, not the staff member's allocated time.

However, what do we know about the duties of this staff member? $150k isn't a very high salary for an experienced systems administrator

It's higher than an SDE in Seattle, but less than a senior position at those same companies, for people who want some perspective.

Firmly "Middle ground of the area"

It is unclear from this request, but if this is the cost to the employer it is almost certainly a larger number than the actual pay which goes to the individual.

$150k is not "higher than an SDE in Seattle", unless you meant to say "higher than the average salary for an entry-level junior SDE role in Seattle."

That does seem to pretty clearly be what they meant, given the rest of the sentence.

There are usually three main levels people talk about, entry, mid career, and sr. If you say SDE, I'd assume a mid-career person, which is 100% earning way more then 150k in Seattle. https://www.levels.fyi/t/software-engineer/locations/greater... has more data.

In Big Tech(tm) you’d have at least lead/staff and principal on top of those levels, who make even more.

Cost of living in Corvallis <<< Cost of living in Seattle

https://www.numbeo.com/cost-of-living/compare_cities.jsp?cou...

From a university grants perspective that likely includes benefits.

Grant hiring math is

Salary + benefits = cost

Where benefits = salary *~.4

Does "benefits" also include the tax contributions the company pays? After being 1099 for so long, those definitely sound like a benefit to me!

I honestly don't know but I assume so.

They technically call it 'fringe benefits'. My university has four categories of fringe benefits:

Full

Limited

Partial

Grad Health

The only things it specifies are that partial includes social security and full includes life insurance. But given that whatever I set for a post doc/research scientist/etc. salary is the amount they are paid, I assume that everything else including payroll taxes are encompassed in that 1/3 extra for fringe.

Presumably the $150k also includes all oncosts, so the actual salary is quite a bit lower still? As a side note I don't understand the arguments about salaries for nonprofits. Sure they should not be outrageously higher than the average, but shouldn't we want to get the best people for these jobs (instead of them working on aware?), or is the argument that if you work for a nonprofit you should be doing it out of altruism and be glad you receive a salary at all?

The argument that I assume you are talking about has some nuance around it. It's mostly about politically connected or nepotistic people who are pulling large salaries for essentially little to no work. I'm sure most regular employees at a nonprofit get treated as poorly as those of us at a normal business.

But's usually not the argument being made, the complains (same in this case) are often about the salaries of the people doing the actual work. Sure I understand the complains about multi-million salaries for the CEOs of some non-for-profit (on the other hand I have the same complains about the ridiculous salaries of CEOs of for-profits), but if that's the nuance, it doesn't come out in the complains.

Well without actual examples from any supposed position, this discussion goes nowhere.

What do you mean? We are discussing an example right here???

I've been in the IT field for 20+ years, but I've never made that much in a year. What's stopping someone from saying, "Hire two cheaper guys, so there's redundancy?"

The lack of people who are willing to do the job at comparable capacity for $80k a year.

>I've been in the IT field for 20+ years, but I've never made that much in a year.

What on earth have you spent your time learning? How do you have 20 years of experience but no specialized skills that bring a modest salary?

Do you not have time for personal development while at work?

I'm not trying to be rude, I'm legitimately curious.

The $150K probably needs to cover other costs, like payroll taxes. Perhaps other benefits as well? (Health insurance would be the big one -- in the rough ballpark of $20K).

Assuming that's the "fully loaded" cost (ie including taxes, benefits, etc), seems like it would translate to a take-home salary of $100k or less.

That's 60% of the _budget_ not 60% of their time.

Also: Lance is almost certainly working more than 40 hours a week. Also, he isn't just a systems administrator. He's a mentor, fundraiser, any literally everything else that is needed to keep the lab running. There used to be more staff, but it's hard to retain qualified individuals. He's been there for 17 years, he's not doing it for the money, he does it because the OSL is important!

Oh, and since he's a public employee, you can look up the current salary and history.

https://hr.oregonstate.edu/sites/hr.oregonstate.edu/files/er...

https://www.openthebooks.com/oregon-state-employees/?F_Name_...

I'll summarize it:

$107k in 2017 and $124k in 2023. I don't know about you, but someone with 17 years experience could easily be making 2-5x that depending on the company and role.

And it's $124k on the west coast, specifically western Oregon. Coastal PNW is notoriously expensive to live in - sure, Corvallis isn't Seattle, but it's not cheap, either. Folks like to balk at numbers when it comes to publicly (or FOSS-donation-ly) funded salaries, but also balk at tying context to those numbers. It happens almost every time anyone dares try to pay their bills on open-source work: a flame war over "you don't need that number, you could move to your parent's basement in Arkansas and survive on $20k instead!".

I grew up near/around/in Corvallis. $124k is quite a bit there. Food is cheap, you can find pretty cheap land/realty, etc. Overall it's pretty reasonable.

That said, $124k is not a lot for what Lance does.

Maybe it was at some point in the past? I have friends who currently live there and it does not sound cheap. Again - not Seattle bad, maybe not even Bellingham bad, but still not "124 is quite a bit" levels (from the sounds of it).

Regardless, we can definitely agree: Lance could be making a ton more elsewhere, but is a saint who cares about his work, and I appreciate his dedication!

I have indeed lived life wrong. I work in HPC as a Systems Engineer (right now, in 2025, with graduate degrees in engineering, and 25 years of systems admin / engineering experience) and do not make what this person made in 2017, much less in 2025, OR 2-5x that amount for that matter (total dream salary, geez)... at one time I was the data center manager and teaching CS classes, at the same time, working 80 hours a week.

How the heck do these people secure these high paying jobs? There is some club, and I am not in it. Sorry to rant, but that 1FTE salary is huge.

Take a look at salary reports from places like https://www.levels.fyi/2024

If you think $124k a year is high compensation for someone with 17 years of experience in Portland, your compensation expectations are way off.

Wow, I read your informative link. Where are these jobs? I went through a round of interviews last year for Sr. positions, across a number of locations in the U.S., and quite frankly, the average salary for the positions interviewed for was $80k less than most of those in the list, and $230k less than the SWE manager in the list.

The page lists the locations, and the businesses, where these jobs are placed. Unless you live on the coast (or end up in Denver/Austin), you're going to have a harder time reaching these salary numbers.

> in Portland

Corvallis

Not if we're talking about an experienced engineer that is the only full-time staff for a year. That's the entire budget, so it feels pretty spot on. Maybe I'm out of touch or misunderstanding your point, though.

GP is pointing out that OSU Open Source Lab is only in jeopardy because a majority of the shortfall is in labor costs. 86% to be exact.

Yes, maintaining stuff requires labour, and people who do that labour deserve to be paid for it. What’s so surprising about that?

Given that it’s a surprise to everyone, it is surprising. Since it’s open source people expect it to be done via donation to unpaid volunteers.

Who is it surprising except you?

https://news.ycombinator.com/newsguidelines.html

"In Comments

Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

Comments should get more thoughtful and substantive, not less, as a topic gets more divisive."

It was a genuine question, not a snarky one. Most of the other comments were making the same point that I was, which is that it is reasonable that labour costs that much.

There was no point made other than to argue why I or others are surprised which I provided an explanation. There isn’t much else to go from here other than your tone was snarky.

> Am I reading correctly that of the $250K they need, $150K of that goes to a single staff member for 60% of their time? Does that seem...excessive?

When I was doing AIX and Solaris system administration in Salem Oregon, they paid me $75 an hour.

A lot of people here are comparing Corvallis to Seattle, but they’re hundreds of miles apart.

Salem is the nearest big city.

TBH, making $75 an hour in Salem was like making $150 an hour in Seattle. You can live REALLY WELL on $75 an hour in Corvallis.

Maybe ten years ago, but not since the pandemic.

Seattle is still more expensive, but $150k is just buy-a-house and have-a-kid money in Corvallis anymore.

And that $75 an hour (which annualizes to $156,000) doesn't include employer overhead (taxes, benefits, etc.) which can double their cost.

No? Less qualified software engineers make more with less experience, and while doing orders of magnitude less important work

No, it doesn’t seem excessive at all to me. If the lab delivered the same services for $2.5mil would you still be questioning that $150k salary?

That’s going to be a tough sell to any company. At some point, this is less of a charity and more of a failing company.

I really am constantly surprised at how foolish some posters on this site are, but your comment is pretty exceptional even by recent low standards.

do you actually know what OSUOSL is and what it does? do you care? if not, why are you even commenting on this post?

This is not a company. It is not expected to turn a profit.

Learned about the OSL way back in Brazil via their support to Open Source and some of their alumni who worked in OSS projects too. I hope they figure out a way forward. And I was surprised to see they ask funding for a single position. Always imagined they would be a team of 3-7 engineers.

I am not working in the computer science field any more, but my experience working with computer systems in an environment like this helped me gain a lot of confidence as an individual. This is amazing work and deserves to be funded!

The donation page is not easy to use and one must search the drop down menu for the tag "Open Source Lab Fund". However it is worth the effort. I did my part.

> While the Oregon State College of Engineering (CoE) has generously filled this gap, recent changes in university funding makes our current funding model no longer sustainable.

I suspect this is related to the recent Trump administration actions to withhold funding from colleges and universities. OSU already voted to increase tuition recently amid concerns about future federal funding [0]:

> “New federal priorities and proposed funding cuts, especially for research, may have direct, negative consequences for OSU,” [OSU President Jayathi Murthy] said.

0: https://archive.is/iGX5X

OOT, How'd you distinguish between Oklahoma, Oregon and Ohio State University when you see the word OSU?

OSU.edu is Ohio, all others have their own domains. This is also why Michigan State is the one true MSU, etc. (Being a Michigander, it pains me to give Ohio any credit for anything, but that goes to show how much I think this is a sensible way to distinguish them.)

I want to go to Open Source University!

I hear Microsoft loves open source, so they should be able to step up and cover this, right?

Why stop there? Facebook, Google, OpenAI, Netflix, Amazon…

Oregon State University has a $1.651 billion endowment according to Wikipedia…

https://en.m.wikipedia.org/wiki/List_of_colleges_and_univers...

Would that be an option to save it if corporate sponsorship doesn’t work out?

Most gifts that fund university endowments are earmarked by the donor for specific purposes. And for the money not earmarked, you're competing against all the other priorities, including making up for various unplanned shortfalls of Federal funding.

The Foundation itself has nearly $1b in assets. The problem with these foundations is they're often streched across the entire university. Which means the foundation behind OSL is also the foundation behind intercollegiate athletics and tons of other completely unrelated programs.

https://projects.propublica.org/nonprofits/organizations/936...

I ctrl-f'd Oregon State and didn't find it in the link you provided. I think you found UofO's endowment: University of Oregon - $1.651

Wikipedia states that OSU's endowment is $829.9 million (2023).[0]

[0] https://en.wikipedia.org/wiki/Oregon_State_University

You’re right, I mixed them up! Thanks for correcting me on that.

people have started using the word "endowment" against colleges like it's a term for slush fund.

indeed they have. In reality they're like complicated heterogeneous investment funds with untold numbers of varying restrictions governing them.

They're not just a pile of gold coins in the university presidents basement.

Indeed. Everything is earmarked, occasionally even earmarked in a way that the university cannot use the at all (e.g. something like a donation "to pay for an earnest white Christian man to teach phrenology").

Reply on news.ycombinator.com