joatmon-snoo

AFAIK it's been done for a while (ex-Googler but non-YT, left recently).

Vitess was most definitely a special snowflake: most Google production servers don't even have disk anymore[0].

(There are a whole host of other reasons for doing Vitess->Spanner, all of which fall under "special snowflake infrastructure".)

[0] https://static.googleusercontent.com/media/sre.google/en//st... (see ch2, diskless)

ITT: people who have never dealt with things going _wrong_ in a large codebase.

It still is done that way, but even individual LSC (large scale change) CLs touch easily >100 files and rely very strongly on the atomicity guarantee of the monorepo.

Plus, when most people think LSCs, they think about the kind of stuff that the C++ or TS or <insert lang> team do, not someone refactoring some library used by a handful of teams, which themselves usually impact anywhere between ten to a few hundred files.

Re this thread specifically, I suspect medicine is viewed differently in the US than in the UK. There's certainly an impression, imo, of the profession in the US that's derived from the upper percentiles of the earning potential here; I've heard (but am not sure) that the range is less extreme in countries with superior healthcare systems.

Anyone who has ever done analysis of raw data knows that the process consists of iteratively excising bad data until you can actually precisely and accurately explain your analysis results.

Tools like this, where the data consumers (campus admins) have no possible way of understanding the data analysis methodology, let alone the raw data themselves, when the _responsibility_ for the consequences lives with the consumers, have no business operating like they do today.

Rust at Google is in a bit of an awkward state. There's a bunch of teams/projects that have used it, but we haven't yet staffed a team to manage an internal Rust toolchain/ecosystem.

The most recent one was Kotlin, in large part because our Android ecosystem was increasingly having to play catch up, so we'll see if/when a Rust team gets staffed.

One of the more creative tricks I remember was writing a cheatsheet in red and blue text overlaid on each other and using 3D glasses to read it, which allowed you to squeeze more on. Never actually did it, so I don't know if it was useful...

LSP was a pretty inevitable development in software IMO- there are increasingly many codebases where local compilation is a nightmare and you need some kind of remote/distributed build system to manage it for you, and once you have that you need to push xref and indexing out of the IDE as well.

Relevant: https://research.google/pubs/pub46584/

From the abstract:

> We focus on a code-review based approach and consider the effects of surfacing mutation results on developer attention. The described system is used by 6,000 engineers in Google on all code changes they author or review, affecting in total more than 14,000 code authors as part of the mandatory code review process.

There's horizontal scaling, and then there's horizontal scaling.

A lot of software assumes a single server is fine and as a consequence doesn't even bother trying to abstract that away; the singleton nature of the software then percolates virally through it all.

On the flip side, some software actually isolates their scaling limitations and pushes the bottleneck into a single part of the system.

With the former, you're screwed if you hit your vertical scaling limits (to say nothing of what your system availability probably looks like, if scaling is actually a concern). With the latter, you can play games with your scaling, perhaps by shoving a hierarchical or replicated cache in front of your scaling bottleneck, and effectively buy time to build a solution with better scaling. (Or perhaps never bother with that, and just be acutely aware of your system's scaling dimensions, like StackOverflow.)

There's no such thing as a system that truly scales infinitely, just systems whose scale limits exceed our imagination.

(Admittedly, most bugs of the form "INT64_MAX overflow" are generally a leak and not the result of growth :).)

Re-inventing the wheel can be useful for learning, but can have very real costs, often in the form of production outages.

* "You can usually use user metadata field X for billing" - except for those users for whom field X actually maps to something else, for tech debt reasons. (Is it stupid and bad? Yes. Is anyone going to be able to fix it this year? No. Is this going to result in Very Big Customer TM getting mad? You Betcha.)

* "Oh, I'll just roll my own fake of Foo" - congratulations, now anyone looking for a fake needs too decide between yours and the other one. (Yes, this is highly context dependent, but the moment you have multiple fakes in common/util libraries this usually starts being a problem.)

* "I can just use raw DB writes for this, because I don't want to learn how to use this API" - except the abstraction exists because it guarantees you can do safe incremental, gradual pushes and roll back the change, whereas your home-rolled implementation had a small bug and now the oncaller needs to do manual, error-prone surgery on the backup instead of the usual undo button built into the API. (Oh, and legal is going to have a field day because there's no audit record of the raw writes' content.)

Cargo-culting is bad, yes, but reusing existing abstractions is often important because they handle (or force you to handle) the various edge cases that someone learned about the hard way.

And of course, if you find a bug in the existing abstraction, well, congratulations- you just found a repro and root cause for that infamous support case that's been giving everyone data integrity nightmares for months.

Forget about inherent complexity, just having good error messages is hard!

I spent hours this week debugging a test that kept failing in

  EXPECT_EQ(result.status(), SUCCESS)

Since this was a codebase I hadn't ever worked in before, I spent quite a while flailing around trying to figure out where I needed to add logging and eventually realized that `result` had more error details that the assertion was hiding from me.

I have a similar gripe with Python's unittest: because some of the assertions allow `expected` and `actual` to be interchangeable, there's always some extra mental overhead to remembering which is which. For example, compare this:

  >>> unittest.TestCase().assertEqual(1, 2)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python3.9/unittest/case.py", line 831, in assertEqual
      assertion_func(first, second, msg=msg)
    File "/usr/lib/python3.9/unittest/case.py", line 824, in _baseAssertEqual
      raise self.failureException(msg)
  AssertionError: 1 != 2

with Java's Truth (https://truth.dev):

  assertThat(actual).isEqualTo(expected)

which results in errors like this:

  java.lang.AssertionError: expected:<[guava, dagger, truth, auto, caliper]> but was:<[dagger, auto, caliper, guava]>
    at org.junit.Assert.failNotEquals(Assert.java:835) <2 internal calls>
    at com.google.common.truth.example.DemoTest.testBuiltin(DemoTest.java:64) <19 internal calls>

Googler but nowhere near Gmail, so just educated speculation:

* We have a lot of automation/tools to prevent incidents when mitigation is straightforward (e.g. roll back a bad flag, quarantine unusual traffic patterns), which means that when something does go wrong it's often a new failure mode that needs custom, specialized mitigation. (e.g. what if you're in a situation where rolling back could make the problem worse? we might be Google, but we don't have magic wands)

* Debugging new failure modes is a coin flip: maybe your existing tools are sufficient to understand what's happening, but if they're not, getting that visibility can in itself be difficult. And just like everyone else, this can become a trial and error process: we find a plausible root cause, design and execute a mitigation based on that understanding, and then get more information that makes very clear that our hypothesis was incomplete (in the worst case, blatantly wrong).

+1 additionally, if you end up in a scenario where you don't even have enough capacity in a given layer to fail quickly, your only options are either increase capacity or throttle load pre-server (either in the network or clients)

This isn't a deliberate choice though on their part, is it? Isn't it just a consequence of everyone depending on readline and readline using emacs-style key bindings?

It's defense in depth.

Everyone introduces a cache (aka eventual consistency) at some point. Everyone ships a bug here and there. Everyone messes up a rollout eventually.

Just like you should have backups even if you never intend to DROP TABLE *, you should also be confident that the invariants you design your system around are actually enforced.

I tend to reach for a design doc when I have (1) an ambiguous problem, (2) no obvious solution but N possible candidates, and (3) need feedback from multiple parties.

The primary functions (as people have mentioned elsewhere) are to (1) drive consensus and (2) record thought processes at the specific point in time.

Here's what I was able to dig up, hopefully someone will be able to take this and run with it:

* nothing obvious at https://www.nationalgrideso.com/news * minutes from a 2011 meeting suggest that ESO was talking to Beacon Power at the time: https://www.nationalgrideso.com/sites/eso/files/documents/18... * http://credenceturbine.com/business/flywheel-energy-storage-... mentions Beacon Power providing a frequency regulation flywheel in 2014 for a 20 MW plant * "Techno-Economic Assessment of Energy Storage Technologies for Inertia Response and Frequency Support from Wind Farms" 2020 July 02, https://www.mdpi.com/1996-1073/13/13/3421/pdf discusses some details

Mine is v% :)

I don't think you ever stop learning C++... :)

But I trust cppreference more for the details, which are often necessary for C++ because, well, the devil's in the details. (And to be totally honest, the styling is just nicer.)

Compare the docs for std::vector for example, cplusplus[0] and cppreference[1]. The former spends three paragraphs explaining the internals, whereas the latter only spends one. The latter also includes information about what's been introduced on which language version (11, 17, 20) and assorted facts that the former doesn't discuss, e.g. the various container contracts that vector fulfills.

[0] http://www.cplusplus.com/reference/vector/vector/

[1] https://en.cppreference.com/w/cpp/container/vector

It isn't so much the total LOC that I write (and delete) in the process of making a change so much as it is the friction and mental overhead of the change. The more I can focus on making the semantic change that I care about, the better.

Admittedly my vim setup doesn't even have YCM because I don't care enough to do it. But there are other minor things (swapping : and ; in control mode, mapping jk as <ESC>) that I've done that I hate working without now.

meet.google.com is Google Meet, yep.

This made me stop and think because I've never really wondered about what belongs and doesn't belong in Unicode, and then I went looking at what strange corners Unicode has and wow the "Miscellaneous Technical" code block is such a strange thing.

* It has APL symbols (checking the Wiki article on APL syntax and the APL standard, it seems that APL programs can be represented entirely in Unicode - which makes sense, but is still a little surprising).

* There's a benzene code point: ⌬

* "ERASE TO THE LEFT", better known as backspace, is a thing: ⌫

...and a little more bizarro: https://en.wikipedia.org/wiki/Miscellaneous_Technical

Google applies ErrorProne in the default javac toolchain to mitigate this for Java builds: https://errorprone.info/bugpattern/FallThrough

I'm thoroughly entertained by seeing "ALSO IN THIS ISSUE: It doesn’t have to be Turing complete to be useful" right next to "The decisive factor in what makes something a programming language (or not) is known as Turing completeness."

If you crack open developer tools and watch the API requests go by, you'll be able to confirm that it's the same thing :)

Makefiles are worse shell scripts. /religion

I do agree with the spirit of this though. There's a big difference between "click a button in your IDE" to compile and learning about all the code-ish things that go into a real software project:

* choosing a build system (and package system too, I guess?)

* setting up CI

* static analysis

* setting up CD

* config-as-code for managing the state of prod

* staging environments?

* production monitoring

* production alerting

hm. dropped an email in my profile.

Current Googler here; my solution to staving off complacency has been constantly asking myself if there are other problems I can and will work on - and they don't have to be problems on your team. (Although, I will say that at a certain point, if you're growing, I think your responsibilities should be shifting from solving problems to identifying/prioritizing the problems.)

Every team has a looonnngggggg list of things that they want to do but can't because they don't have the people for it (even if they're not keeping track of it). Frankly there's also a non trivial amount of core language work that's done by people who don't actually work on any of those teams. Getting context can be hard sometimes, but I've done it plenty of times.

Am always happy to chat, my username is pretty obvious from my profile.

Worth calling out that Google literally compiled a list of outages caused by `required` fields before killing that field parameter.