asciimike

> A silly mistake can cost you so much money and they tend to word the e-mails strongly.

They tend to word emails strongly to affect action.

As for the cost overruns: I don't know anyone who wrote one of those blog posts (or emailed support) who didn't get their bill refunded.

We can argue about spending caps or hard limits (which people generally dislike for their production apps), but the people side of the business is (or at least was) solid.

Source: worded emails strongly to affect action at Firebase and refunded a number of bills.

> I've talked with multiple ex coworker developers at Google about the 3 to 6 months it takes to actually figure out how to deploy something.

In typical developer environments, you end up with a bell curve of "easy things are relatively easy (host a static site, 30 seconds), but some things end up being totally impossible (build a global CDN with 99.999... availability)."

Developing at Google is not at all like this: building anything at Google is medium-hard. It takes longer to do simple things, but at the same time, basically nothing is technically impossible. In many cases, doing the hard things is much easier than elsewhere because, yeah, sure, you have to set up 20 different config files, but those config files abstract damn near anything a production system needs to operate.

Example: I wanted to add improved image serving functionality (e.g. imgix style URL params), and I was able to integrate the existing infra Photos uses in about 15 lines of code and a few hours. I don't think there are too many places in the world where it's possible to provide that functionality in an afternoon. Why it never shipped is a separate story involving non-technical reasons (cross-PA politics).

> Also the constant overhead of approvals and oddball systems (gerrit!) thereafter.

IIRC only the Android codebase used Gerrit, the rest of Firebase and GCP were on Google3. The approvals and other stuff... true.

> Eventually good DX seems unimportant and even associated with failure.

I think this comes down to a definition of "good DX". I think the HN definition (indeed by default one) is "how quickly can I solve for an issue using the happy path." Indeed, Firebase solves that for a good number of problems (why people use it, why there are people who love it).

At some point though, you can't solve for _all_ use cases along the happy path (see e.g. SAML, ABAC, any $ENTERPRISE_FEATURE). If you want to expand the business, you need to solve for some things that don't have a perfect happy path, and that's where things start getting messy.

IMO, the goal of Firebase's acquisition was to keep Firebase solving the 80-90% of things that had a happy path, and GCP was the remaining 10-20% where folks had to break out of to do arbitrary things. It seems like a lot of folks in other threads here are complaining about that because the abstractions leak out--I think it's a fair criticism, but I also am not entirely sure how to solve for that.

> The first application I wrote for Firebase was back in 2015, before it was acquired. I don't remember the specific issues, but I know I spent about the same amount of time trying to keep it running post-acquisition as I did building the app in the first place.

Nit: Firebase was acquired in Oct 2014. New functionality was launched in early 2016, and the major change that would have affected you at that point was the auth system, but that should have been a one time change with low ongoing maintenance.

> I should point out that averaging a single breaking change every five years means that an application has a 20% chance of being affected each year. I'm not willing to roll the dice, generally speaking.

What technologies are you using on a routine basis that fall into a category you consider acceptable?

The RTDB APIs have remained stable and available since ~2012. Storage has existed in the same way since ~2016. Do you have some specific examples of build APIs changing on you in ways that break your applications?

You can't sell to people buying Macallan 12, you have to sell to people who _don't_ currently buy Macallan 12 because $REASONS (significant price difference, ESG angles like lower carbon footprint, they can't get it in their location, etc.).

Same. I keep a bottle of Blue Label that I refill with Green/Gold/Platinum (depending on how generous I'm feeling) that gets served to guests who visit my home bar and ask for Blue Label because they know nothing about scotch except that it's expensive. I actually think those colors are better than Blue, but that's neither here nor there...

My personal trend over the past few years has been to buy from independent bottlers, of which Costco/TJ usually carry (e.g. Alexander Murray). I've also switched to other beverages: armagnac, mezcal, rum; all of which have lower brand recognition and interest here. Overall it means I can get more variety (and value) at a better price, which is a win.

Short answer is likely "yes" though I think there's a surprising amount of variation even there.

Longer answer is that I'm not sure the direct comparison ever really happens because in a spirits tasting competition where there would be double blind testings, it's never pitted against the whiskies it's being "marketed against" (e.g. a "real" bourbon or scotch). Sip 2021 puts it in "other whiskies" in which it gets the lowest awarded medal: https://sipawards.com/content/results/sipawards2021results.p... (though interestingly in 2022, Spice, their "scotch" blend, won double gold in the same category: https://sipawards.com/content/results/sipawards2022results.p...).

Obviously you could host your own with a few friends and give it a go (e.g. https://www.keacher.com/1334/scotch/)

Interesting article, I've got a few thoughts on this space.

Endless West (https://endlesswest.com/) has a "molecular whisky" (they aren't legally allowed to call it whisky because it wasn't distilled from a certain mash, aged in oak for a minimum number of years, etc. etc.) that's the same idea, minus the machine learning (which probably isn't that helpful, IMO [1]). More on the process in [2].

To my knowledge, it hasn't sold particularly well, and most reviews (granted, likely biased by the origin of the spirit) tend to say "it's fine, but it's not the same as 'real whisky'."

The issue with the "luxury spirits" market is that (to paraphrase the Scotch distillery Bruichladdich), "terrior matters". Like fine art, one's enjoyment of such a beverage comes from both the tangible (taste, smell, bottle presentation, etc.) and the intangible ("having good taste", "buying a bottle of whisky older than you are"; generally signaling value). Like most (all?) luxury goods, the tangible costs account for a small percentage of the overall cost, with the intangibles and associated signaling value.

There's a reason that "The Macallan" which is marketed as such costs significantly more than vs "the macallan" that's been private labeled by Costco or Trader Joe's. Same juice, different intangibles.

All of this is to say that:

- I think the technology is super cool and I want to see it come to fruition

- I don't think "luxury goods" is the right segment to target because nobody buys a birkin bag to carry their laptop to work

   - Create a novelty for mass affluent consumers; not super high margin but make up for it in volume (what Glyph was attempting)

   - Target a niche consumer group who cares about a different signaling metric, e.g. eco conscious consumers who want the same "end resuly" but are unhappy with traditional processes (which is what Endless West seems to be doing with e.g. [3])

[1]: Cognitive Cooking with Chef Watson (https://ice.edu/partner-with-ice/IBM) is a great cookbook because chefs fed IBM Watson a bunch of recipes and then asked it to create new recipes, which it did, with some _very_ wacky results that chefs then tweaked. Definitely possible to create new/unique/interesting things, but I think it's hard to get people to buy into the end result, especially if the majority of folks would initially reject it as disgusting/too weird.

[2]: https://fortune.com/2019/05/25/endless-west-glyph-engineered...

[3]: https://shop.endlesswest.com/kazoku.html

This is the route Intel is going down: https://bitcoinmagazine.com/business/intel-launches-new-bitc...

Time will tell if it works out (different chain, etc., etc.)

There are a few of these type of things, e.g. RNDR token (https://rendertoken.com) and rent a flop (https://rentaflop.com) in rendering, and golem (https://www.golem.network) and sonm (https://sonm.com/) in the "general purpose computing on the blockchain"

I consider a long term PPA a hedge against near term volatility. C.f. https://www.fia.org/marketvoice/articles/derivatives-markets...

My understanding is that for their own data centers (e.g. us-east-1 or us-central1) they are able to negotiate excellent costs of power by soliciting 100MW+ long term contracts, so they don't really need to. Energy is a lower margin business than tech, and they'd rather not get into the messy business of power generation.

For colocation data centers (e.g. Tokyo or Singapore regions), they're more reliant on the data center provider to do that negotiation, and they negotiate for rack space prices, so it's slightly different (thus why prices in regions can vary dramatically, though this is also labor, what the region can bear price-wise, etc.).

I assume they also have teams hedging power prices to keep those costs reasonable, while OVH and other tier 2's likely haven't reached that scale.

I'll briefly shill my product (crusoecloud.com) because we _do_ vertically integrate our power production and data centers by colocating modular data centers at oil wells (utilizing flare gas), wind/solar, etc. as a way of both keeping energy costs low/stable as well as solving related environmental issues (e.g. reducing methane emissions or providing base loads to renewables).

Odd, I just tested on Safari mobile and it went through without issues. Mind sending me an email at mike at crusoecloud dot com and we can get you set up?

See above post, but the TL;DR: is "we capture methane that would otherwise be flared."

Agreed that a lot of carbon offsets look like, "we were going to cut this section of the rainforest down, but if you pay us, we won't do that for X period of time." This is _actually_ reducing existing emissions.

TL;DR: we run data centers on-site at oil wells and take natural gas that would otherwise be flared (it can't be economically transported as natural gas in a pipeline or turned into electricity and transported) and combust it completely. Methane is a significantly more potent greenhouse gas than CO2, so it ends up being a net reduction in emissions vs what's currently happening.

https://www.crusoeenergy.com/digital-flare-mitigation has some more information.

If the goal is to increase the GPU vRAM (which I believe it is, given that's the constraint on image size), the answer is "not really" for consumer cards. You need NVLink bridges for pairs of PCIe cards (which they have, but then you'd only double the vRAM), or NVSwitch on the high end data center servers (DGX/HGX A100).

Morgenrot Cloud (https://morgenrot.cloud/) is the main consumer grade AMD compute provider I know of. Not quite vast.ai in that they're centralized, but they've got the hardware.

Lots of good options for cheap GPU clouds, including Paperspace (mentioned above), Coreweave, and Crusoe Cloud (crusoecloud.com). Crusoe Cloud's angle is that our GPUs are powered off otherwise wasted energy and are carbon-reducing; running one for a year provides an emissions reduction equivalent to taking one car off the road.

Disclosure, I'm head of product @ Crusoe Cloud. Feel free to ping me at mike at crusoecloud dot com if you've got questions or feedback.

This is true in X-Plane as well though, right? I would kinda expect a better default Cessna model, though I admit that most people probably aren't going to either flight sim for lots of time in a Cessna (though I'd actually be really curious to know what % of planes are flown what what amount of time).

Here's a great example of why the flight model is... lacking: https://www.youtube.com/watch?v=6GX9lTvKuqU

It's perfectly fine for most things, but _really_ bad with things that could actually kill you.

Meanwhile, in X-Plane land: https://developer.x-plane.com/2022/02/x-plane-12-flight-mode...

I have an ergodox and a moonlander, and I shared the same frustration on initial use; however, after about a week of pecking around and doing a good number of typing exercises (e.g. https://www.keybr.com/, https://www.speedcoder.net/) using a basically modified mac layout, I was back up to 99% for standard use, and definitely a bit faster for common things (e.g. typing usernames/passwords).

I don't use tons of layouts, I'm mostly there for the ergonomic benefits, which I definitely feel vs a magic keyboard and trackpad.

Japan also has Furusato Nozei where (TL;DR) you can choose which province your taxes go to. In exchange, they send you local goods (crab, beef, etc.). This website shows some of the items available: https://www.furusato-tax.jp/

Crusoe | SWE, SRE, Hardware/Firmware, Networking, Product, Design, Devrel | ONSITE SF Bay Area, Denver/Boulder, Chicago | Full-time https://crusoeenergy.com and https://crusoecloud.com

Crusoe is hiring engineers to build a high performance, carbon negative cloud platform. We're a series C company (just raised ~350MM @ 1.75B) with a growing engineering team (currently ~20) looking for folks to align the future of computing with the future of the climate.

Our stack: GCP, K8s (and related ecosystem), go, Typescript and React frontend.

Job descriptions: https://www.crusoeenergy.com/careers

Feel free to email me at: mike at crusoecloud dot com if you're interested!

(Disclosure, I worked at GitHub at one point)

> Github is a great example in saas area. Utterly trivial for most customers to switch away or self host. They have no moat so they can't really jack prices.

I'm not sure I agree with this. It may be true for single developer code hosting, but GitHub as a business is much more than just that.

GitHub makes money off three things, only one of which is the "core" code hosting business and two of which have solid moats: - GitHub Enterprise {Cloud, Server} - Actions (Azure compute minutes) - Advanced Security (code scanning, secret scanning, etc.)

For customers operating in this environment, it's _extremely difficult_ to move because the latter features are built in to their business process (e.g. a compliance requirement that there are no known vulnerabilities dictates the use of GHAS, and they've already gotten signed off so switching to a different provider requires going through a more stringent audit in the future). GitHub doesn't have to raise prices, they just slowly add new functionality that boosts the usage of Actions, GHAS, etc. which both provide actual benefit to those customer segments and increase revenue (and eventually get new, larger customers on board).

That large scale usage also offsets the cost for their free tiers, which in turn keeps the single developers hosting their code on GitHub. Even though it's fairly easy for them to move their repos elsewhere, the halo of other features plus the community (which is really the asset) keeps developers on the platform.

Adding a few I know of:

https://groq.com/ - From the team that built the original TPU at Google

https://lightmatter.co/ - Light tubes instead of copper

Crusoe | SWE, SRE, Hardware/Firmware, Networking, Product, Design, Devrel | ONSITE SF Bay Area, Denver/Boulder, Chicago | Full-time https://crusoeenergy.com and https://crusoecloud.com

Crusoe is hiring engineers to build a high performance, carbon negative cloud platform. We're a series C company (just raised ~350MM @ 1.75B) with a growing engineering team (currently ~15) looking for folks to align the future of computing with the future of the climate.

Our stack: GCP, K8s (and related ecosystem), go, Typescript and React frontend.

Job descriptions: https://www.crusoeenergy.com/careers

Feel free to email me at: mike at crusoecloud dot com if you're interested!

> As someone only a little older than yourself, I would say this: your job will never love you back, and especially in the software realm, it may not be as fulfilling as something like woodworking or therapy. It does pay the bills, though.

Reminded me of https://github.com/docker/cli/issues/267#issuecomment-695149...

> I no longer build software; I now make furniture out of wood. The hours are long, the pay sucks, and there's always the opportunity to remove my finger with a table saw, but nobody asks me if I can add an RSS feed to a DBMS, so there's that :-)

Relevant discussion of that thread: https://news.ycombinator.com/item?id=24541964

I think you summarized it well :)

Throttle is drive by wire, brakes are hydraulic (as is the clutch), and steering is hydraulic.

That said, I'm not railing against computers in the car, I'm mostly saying I like manually shifting my gears.

I've met a few where it's not explicitly required on download (e.g. GitHub has their SOC2 available for download on the enterprise admin page, but by the time you're using GHEC you've signed a few pieces of paper), but agreed that most companies aren't giving them away for free.

In case you're coming to this thread asking yourself why there are a bunch of comments like this, I really like https://astralcodexten.substack.com/p/book-review-the-new-su... as an overview of "how we got here in the first place."