Google is 2B lines of code and it's all in one place (2015)
wired.com
wired.com
Try the powerful search, including free text search, operators like f: or symbol: (the full list of operators can be found at https://developers.google.com/code-search/reference)
Then try the references feature by clicking on symbols.
The cross references underneath Google codesearch (both internal and external) are created by, roughly speaking, partially compiling everything and groveling around in the AST.
One of our teammates Luke Zarko has a decent overview of this: https://www.youtube.com/watch?v=VYI3ji8aSM0
> Bizarre, especially when you consider that the whole underlying infrastructure of google codesearch has been open-sourced.
Some of the code is also open sourced at kythe.io, though it lacks a considerable amount of the internal logic necessary to scale it out to all of google3. So it's not exactly plug & play. But, it should be enough for a company the size of Microsoft/Github to get started using it.
Admittedly, we need to do a better job of articulating this difference, but if anyone would like help getting spun up with precise code intelligence on Sourcegraph, please DM me: https://twitter.com/beyang.
cs.chromium et al experiences instantaneous dropoff from "wow this is kind of amazing" the moment you start wading through JavaScript or Mojo glue code. Here's the JS source of the dino game; notice how you can't click anything: https://source.chromium.org/chromium/chromium/src/+/main:com...
A quick Google for "most popular languages on github" just found https://madnight.github.io/githut/, which (if it's correct) reveals that GitHub's most popular language (by commit activity) is Python (17%), very closely followed by JavaScript (14%). Then you have Java (12%), TypeScript (8%), Go (8%), C++/Ruby (both 6%), and PHP (5%). So on GitHub C++ apparently represents 6% of PR activity, while the top two languages (Python/JS) which represent 31% of PR activity are not only interpreted but also dynamically typed.
Which tells a very interesting story about the benefits of being able to generate an AST with type information: you can scale insight that much more broadly and deeply. :(
> why don't you have decent crossrefs
We do have compiler-accurate cross references for many repos. Some examples:
- TypeScript: https://sourcegraph.com/github.com/sindresorhus/got/-/blob/s...
- C: https://sourcegraph.com/github.com/neovim/neovim/-/blob/src/...
- C++: https://sourcegraph.com/github.com/KhronosGroup/Vulkan-Sampl...
Why are not all repos covered?
Because different languages have different build systems, so inferring the right build commands, dependencies etc. is not so straightforward; these are necessary pre-requisites for compiler-accurate cross references. We're working on fixing this with auto-indexing: https://docs.sourcegraph.com/code_intelligence/explanations/...
For C and C++ specifically, auto-indexing is challenging because of the large variety in build systems, informal specification of dependencies (such as in a README instead of a machine-readable format), and platform-specific code.
Outside of auto-indexing, we do have an indexer for C and C++ right now (https://github.com/sourcegraph/lsif-clang) which can be run in CI; that way one can generate an index and upload it to Sourcegraph on a regular basis. It is 'Partially available' (https://docs.sourcegraph.com/code_intelligence/references/in...) right now. We're keenly aware of the interest in C++, and are working our way through different languages based on usage.
You need to be able to build the code to even have a chance at that level of understanding of the source. And really you need to be able to build the code with a specialized toolchain. This is tractable with a huge investment if you get to choose or dictate your tools. If folks are bringing their own tools it is basically impossible.
Pushing for standardization of fact output from toolchains could be a path to getting somewhere when you can't dictate toolchains. Or really just having some kind of fact output from default toolchains would get you a long way.
You'd need to do a lot of educated guesses to properly build the cross-references, and within Google you can just build the code or whatever analyzer you need.
But you're half right here, using blaze makes it way easier.
https://source.chromium.org/search?q=comment:%27%5B%5E%5C%5C...
https://source.chromium.org/search?q=comment:%60%5B%5E%5C%5C...
In this case... about even unfortunately lol.
Disclosure: I work at Google.
What is it with all the Googlers systematically prefacing their comments with "Opinions are my own"? Is that some kind of policy at Google? It's so ubiquitous now that it feels almost like some kind of humble brag.
This is because people attribute the actions of one person to the whole group. An example from the HN page yesterday - Microsoft insults dev, then takes credit for his idea (https://news.ycombinator.com/item?id=31284419). The title was changed later by HN mods to something less incendiary.
But the point stands. 1 person working at Microsoft out of 182k said something sarcastic. That sarcasm was interpreted as an insult. Then the entire organisation was blamed (Microsoft insults dev). This happens because it’s easier for us to think reductively. Nuance is too much effort.
That’s why employees go to great lengths to add these disclaimers every comment.
The point at which I started doing it religiously though, was after I made a comment about one of Red hat's competitors. Somebody wrote another post where they talked about Red hat employee Said certain things, so that's the red hat perspective or that's what red hat thinks or something like that. I want to make absolutely clear that everybody reading anything I write knows that I do not speak officially for Red hat. My opinions are truly my own.
"VMware seems to dominate the bare metal landscape. Even though Red Hat has a competing product, there's still a ton of customers that want to run OpenShift on top of VMware so we end up working with it a fair amount. They made some design decisions that I wouldn't have made, but overall it's solid technology if you don't mind proprietary."
I didn't (and still don't) think that was a negative comment. It felt very factual and fair to me. But "overall it's solid technology" is my opinion, not Red Hat's, and it seems ridiculous to me that people would say it was Red Hat's opinion, but ¯\_ (ツ)_/¯
Previous companies I worked at (like Cisco) explicitly said in my employment contract that I couldn't post things online about the company without prior approval from PR or whoever.
Part of why I do it is also so people know I may be biased as I'm financially motivated to have Google seen in a positive light and be successful.
https://news.ycombinator.com/item?id=31279881
We are specifically told to disclaim that we don’t speak for AWS when we speak on social media about anything related to the company.
Even if not, if I say something positive or get in a discussion about AWS on Hacker News that is opinion based, I disclaim so people can consider my biases (“It is difficult to get a man to understand something when his salary depends on his not understanding it.”)
Also when I post on r/aws, I sometimes disclaim. It depends on the question.
If someone ask “how can I stop API Gateway from timing out after 30 seconds?” - ie something that can be answered factually (you can’t) and post documentation, I don’t bother about saying where I work.
On the other hand, if someone asks, “what are the steps I need to take to improve security on AWS?”. Then my usual reply starts off with, “I am a consultant in Professional Services at AWS, I’m not your consultant. All opinions are my own”. The last thing I need is to be in the middle of a controversy because a company got breached based on taking my very generic advice and saying they were listening to me.
Wouldn’t you think it was kind of dishonest if I said “I think everyone should move to the cloud” (I wouldn’t say that) and not disclose my biases? Most of my arguments are much more subtle. But I would rather err on the side of disclosing than people think I’m astroturfing.
There are lots of people on HN that work at companies that get discussed all of the time. It would be dishonest if we didn’t say where we worked when talking about them.
It’s considered the honest thing to do as you’ve seen many of us do when we talk about the company we work for.
We all have biases. I was biased toward cloud computing when I worked for a 60 person startup. But if AWS had gone belly up then, I would shrugged and start working with Azure. Now I have a financial interest. Of course I’m going to be more biased.
If there's no actual policy for these sorts of things, that flips the coin and makes it very unfair that I don't know where you work, sounds like an interesting place :P
The standard cop out is to say that line even when it’s completely unnecessary. It’s just easier to add it than consider how your words may be missunderstood.
If you wore an I’m with her! tee, you would not claim to have a neutral opinion on politics.
Someone literally explained to you that it's a matter of transparency, and you're jumping to, "well obviously they're just bootlickers".
If you work for Google, comment in public on Google stuff, with an account that says “I work for Google!”, I would say it is not possible for you to remain independent, no matter what kind of spell-of-disclaiming is attached to your post.
When it comes to Google matters, one either comments as a Google employee, or not at all. It comes across as distasteful to try to do wade in while waving off claims of bias with “opinions are my own” with the implication that they are unbiased.
Maybe what these people mean is: I work at Google so this is biased towards them, but they didn’t tell me to write this, and perhaps I should be reading it that way?
Good thing nobody said this, then. They're just making it clear that they're not explicitly relaying company opinion.
> with the implication that they are unbiased.
Nobody's trying to imply this. Obviously if you work at company X, you're gonna have at least some subconscious bias. The point is to make your potential bias transparent, so others can judge for themselves.
> I work at Google so this is biased towards them, but they didn’t tell me to write this
Yes, though of course the amount of bias can vary. I used to work at Amazon, and my opinion of Amazon didn't seem to significantly change when I started working there, or when I left.
(Of course some people develop a more negative opinion of a company after working there, rather than a positive one)
I'd say that a situation where you feel the need to add that sort of disclaimer generally means you should keep your mouth shut, but stopping people from talking about things they know about will never happen.
In other industries, you'll find similar boilerplate disclaimers like "this is not investment advice." Particularly amusing when it is attached to even character-limited tweets.
To borrow your broken analogy and fix it, if someone were to post "Hey, here are all the reasons I'm voting for (candidate) and I think you should too", and didn't disclose that they work for that candidate's campaign, don't you think any credibility they might have had is gone once it's found out?
What I’m getting at is that opining, disclosing, but then claiming the opinions are unbiased just does not compute.
If it's positive, then leaving off the disclaimer is basically astroturfing.
The point of the "I work at X" disclaimer is to explicitly point out that you do have a conflict of interest that the reader needs to be aware of.
Disclaimer: I work at Google and my thoughts are my own. ;)
Disclaimer: All my opinions are Google's ;)
Why do people think it? If they see me say "I work at Google" in the comment, and then say some sort of opinion or recommendation or prediction, they might think that that's an opinion that's widely held or officially promoted within Google, or that the recommendation is something that Google recommends or that the prediction is something Google predicts. The sentence "I work at Google" could be thought of as something I'm invoking to say "I'm an authority believe me", or "I work at Google, so I can provide you info from Google", rather than a disclaimer about a conflict of interest. That's why "I work at Google" is often followed by "but my thoughts are my own".
The phrase "identify as" refers to considering that a nontrivial part of one's identity.
These are separate concepts that happen to use similar wording.
If you're wearing an I'm with her! tee, you also aren't her spokesperson.
It's not neutrality, it's non-representative
I remember a time when I was younger and I too would present much of my persona as being based in the company for which I worked.
As time passes this fades significantly. I feel much more like my own person now, from whom my employer benefits, and not the other way around.
Delusions of grandeur? Moi? Or the true wisdom of age? Why not both!
Especially in my case, I do speak for the company in a limited scope when I am working with one of our customers or when I do approved communications in the form of blog posts, presentations, etc.
I’ve worked for 8 companies over 25+ years. The current company I work at (AWS) is another means for me to exchange labor for money.
I also highly recommend this ACM article talking about our source control from 2016.
https://m-cacm.acm.org/magazines/2016/7/204032-why-google-st...
There's also a YouTube video which is linked from the wired article that is good.
This case, when an employee speaks about their employer in a public forum, is pretty rare, so rare in fact that people are surprised when they see disclosures -like in this very thread.
Google Is 2B Lines of Code, All in One Place - https://news.ycombinator.com/item?id=10227000 - Sept 2015 (325 comments)
This sort of opinion appears rife somehow, people who make it clear their raison d'être is make money above all else are considered incompetent because, whilst making massive amounts of money, their actions are highly deleterious in other areas.
Do you just not believe that those in control only care about the money and couldn't care less about search quality other than to the extent it's necessary to make money?
I definitely believe this, but I also think companies fall victim to the classic quote from Upton Sinclair:
"It is difficult to get a man to understand something when his salary depends upon his not understanding it."
It would not surprise me if Googlers had a rationalization for why these "worse" search results are somehow a better user experience.
Other examples: Apple's tight control over the iPhone software experience is controversial. Facebook employees often say they're benefiting the world, while outsiders think they're harming it.
I don't think these are black-and-white - someone can credibly argue a case for each. But I think the Upton Sinclair quote tends to tip the balance for what insiders believe.
Can someone explain why there have never been any leaks?
What really happened is that China hacked Google, and it pissed Google off when they finally discovered it, many months later. This wasn’t some small intrusion or data breach. It was a systemic, coordinated, widespread, very deep hack, which among other things, gave China all of Google’s source code. It was very similar in scope and ambition to the Solar Winds hack recently out of Russia. It forced Google to completely rethink their security, which at the time was an Igloo model (hard on the outside, soft on the inside, a Gary Larson reference I think), and they had to migrate to where internal access was also limited, which took years.
I’ll share with you, confidentially wink wink, that most companies way overvalue their source code. It’s actually their engineers who are their biggest asset, because the engineers can reproduce the source code if it’s lost (quite quickly at that), whereas the source code can’t do fuck-all on its own. Moreover, most source code bases are so ugly that you couldn’t give them away. But Google had what they now call “HIP” (High-value Intellectual Property) scattered through their source code, which are the tuning parameters and constants for various algorithms and AI models which are the true Secret Sauce to Google’s Search and Ads dominance.
Does anyone really believe this?
At both small companies I worked at, the real value was in the business relationships built up (suppliers etc). The code could be rewritten. It wouldn't be bug-compatible, but all the ideas and design were backed up in engineers' minds.
There are even advantages from a clean slate methodology; if there’s sufficient buy in from management.
Reason being: if you have done things before and bore the weight of architectural misgivings, you can clearly reproduce the software.
The biggest barrier to doing this normally is:
A) time pressure.
B) pressure to use existing log available tech
C) pressure to be cheap (which is manifested most often in A&B but is it’s own thing too).
This causes “large rewrites” to be rushed, subpar and lesser quality as the emphasis often isn’t on getting to feature parity.
Regardless, if google lost their code- entirely, we would probably have search by next week, auth in a fortnight, mail by next month and something approximating google cloud by the end of the year.
Because a lot of the really hard lessons have been learned, a reimplementation is just time.
1. All of Google's Source, All New Engineers
Vs.
2. All of Google's Engineers, No source
I'd probably bet on #2 getting up and running quicker.
Most of snippets of Google's code wouldn't be usable without the entire ecosystem around them. Most things are doable because of libraries, internal infrastructure, etc.
If any state actor who stole that code could deploy such engineering capabilities, they don't really need the source code tbh.
I believe the YouTube video linked from the wired article goes into this, but there's more machine generated code checked in than human generated code at this point. A lot of this is boilerplate or config data that is stored in source control.
I feel like Google definitely believes that config is code, so we have a ton of configuration of our systems checked into Piper.
If you commit all your codegen’d stuff you don’t need to worry about how to run codegen in most places. You can enforce a lot of security properties, you can run large scale analysis on the code in production because it’s all “statically” available, etc.
That said, I think Google’s source control ecosystem is really quite different to what’s publicly available with systems like Git/Hub/Lab etc, even to Phabricator. These approaches work well at Google but don’t necessarily translate to other tools in the same way.
Don’t take these sorts of things as gospel, or even as google advice about good engineering. Understand what they achieve, and then figure out a way to achieve that same result with the tools available to you.
For example, "everything comes from a repo" would make object (.o) caching that much more straightforward.
And you can reason both about the logic behind a service and how it was configured from a single commit hash/CL.
But you know, there are tons of configs, all kinds of.
> Do they do an immense amount of code generation?
Blaze (aka Bazel [0]) has provisions that make it easy to generate code but this happens as a compile step rather than something that is checked into a git repo.
[0] - https://bazel.build/
For L4 maybe there are more days where one can do several hundred LOC in an afternoon than L5, but there are also many days that result in 100 or less over an entire week, because there's testing, debugging, profiling and figuring out what to code.
But, maybe I'd be wrong about Google. Afterall, it might be easier to dump lots of code into the codebase if there are 10 independent chat apps in development at any one time.
So working some week-ends and no vacations at all?
There are about 260 week days in a year. Add bank holidays and vacations and you should ve closer to 230 working days in a year.
A 200 line change typically doesn't grow the code base by 200 lines.
I believe in my 2.5 years at Google, I've actually deleted more lines of code than I've added.