Cloud Source Repositories
cloud.google.com
cloud.google.com
> After profiling non-abusive activity on Google Code, it has become clear to us that the service simply isn’t needed anymore.
And now they've created a new product which does pretty much the same thing... Wonder how long this one will last.
I think the assumption is that most folks will use github / gitlab , etc. as their primary, and mirror to GSR for deployment.
code.google.com hosted public repos, issues, wiki, downloads, etc., and was generally just ripe for abuse. Google created it largely because Sourceforge was the only thing that existed at the time, and it wasn't doing a great job. Today there's GitHub, Bitbucket, Gitlab, and more.
CSR is primarily for hosting private code, shared among teams, and adding integrations with other Cloud services, in the Cloud Console. It also adds code search, which is pretty great. Hosting private repos, it's much less easily abused (it's not impossible). There are plenty of private repo hosts, and this one aims to be the one best integrated with GCP. It has, I think, a much better business reason to continue existing. It might not make money (I legitimately do not know) but it helps drive GCP adoption/usage, unlike code.google.com which didn't.
Disclosure: I worked on code.google.com, and helped turn it down, and helped create CSR from its ashes. I don't work on either anymore.
Understandable considering that AWS has codecommit and MSFT owns github now.
You can use their public search UI or install the extension.
That being said, the tool to migrate to GitHub worked well. It was an easy migration.
Under the umbrella of Google Cloud you've got Google Cloud Console, Firebase, Stackdriver, Source Repositories, Dialog Flow, probably a couple others I'm forgetting. Almost all of these have entries on the Google Cloud Console which open a new tab and link you to a new website.
It makes keeping track of where things actually are extremely difficult. For example: Firebase functions on the Firebase UI are also Google Cloud Console functions. The old Google Cloud Datastore product, which lived on the Console, is now Cloud Firestore on the Firebase console. Stackdriver logging used to be on the Stackdriver UI, now its on the Cloud Console, but there are still references to Stackdriver all over that product, and they continue to invest to Stackdriver (Istio service maps).
AWS does this a little bit, in four or five of their hundreds of products. Lightsail is one example. Azure does this a little bit as well (DevOps is a separate page even though its essentially an Azure product).
GCP is the worst offender. I want to see a better unifying vision for what being a Google Cloud customer actually means, instead of all these confusing and isolated experiences.
The thing you've got to understand is that none of these products are "Google Cloud" products. They're Google services (or, even moreso, services of random third parties which Google has acquired) that you happen to be able to hook into from your GCP project and access through the GCP CLI.
The closest comparison I can think of is Heroku's "Elements", which are third-party offerings (usually of their own services, with their own service dashboards and everything!) that happen to be subscribe-to-able and high-level-configurable through Heroku's dashboard/CLI.
Except, in this case, many of the third party "elements" that are subscribe-to-able and high-level-configurable in GCP, happen to be created by other departments/sub-companies of Google.
Still nothing to do with GCP, though, any more than a random Heroku Element has to do with Heroku.
(I am one such engineer, and have worked on the CLI, cloud build, and container services)
1. that’s not where many GCP-aegis’ed services are right now, and
2. newly-absorbed Google properties will always start out looking like this, so there’ll never be a time when GCP won’t look at least somewhat like this (unless the post-acquisition integration process is changed drastically.)
Firebase and Stackdriver are underneath GCP.
I think everyone here is missing the point. The purpose here seems to be similar to github actions in providing workflows around repository events (though also including some basic ability to view the code as well). Having a clean way to cut a release that updates app servers on app engine and serverless functions in tandem is needed right now.
I don't think this is intended to be used by anyone that doesn't have their platform on GCP.
For highly sensitive codebases within an organization, they may not want to host it on microsoft's servers as well—but this seems secondary to the workflow functionality.
It's possible I'm incorrect though and I do think the messaging here should be improved. Chiefly that it shouldn't lead with "Unlimited private Git repositories for free"
This is the entire field of CI/CD with dozens of providers, including Github and Gitlab too.
[1] Microsoft name shuffling is so much fun...
Internally as Google, you have a substantially power powerful version of Codesearch, which is semantic, i.e. it relies on actual binary artifacts produced by builds to index code. All of the code base uses Blaze ( open sourced as Bazel) to express the rules.
The version of code search here is cute, but it's still not a full blown structured semantic code search, which I really miss. Searching for and browsing code on Github feels underwhelming in comparison, because you wan't really find or explore code the way you think of it.
You can see a demo of semantic Codesearch tech externalized here at:
https://blog.bazel.build/2017/12/14/introducing-bazel-code-s... http://cs.chromium.org
For example if you click on a header or a symbol, you can get proper cross references to a symbol.
https://cs.chromium.org/chromium/src/pdf/document_loader_imp...
I really wish they add structured code search to Cloud Repos, even if it comes with some heavy restrictions such you must use Bazel as a build system.
It would be a killer feature, and to my knowledge there is no equivalent to such a product. IDEs offer some similar overlapping functionality, though other code search engines still operate in terms of strings rather than actually real binary symbols ( I think Sourcegraph is trying here but that still does not seem to use some kind of accurate and structured information like you have in the code, it still seems not use build rules).
How do you deal with compiler provided internals, where someone might be using clang which has behavior X, but gcc / icc / msvc doesn't?
These problems are more prominent in C/C++, but are everywhere... Not easy...
Google can probably check in everything for themselves... Probably not so easy for other companies? Maybe? Can Google solve this without sinking too much resources into it?
This means you can do hermetic builds, and thousands of other engineers can reuse the build artifacts you produce.
And in the particular context of tools such as code searching, your build artifacts update the code searching index pretty quickly post committing a change.
Sourcegraph does use language servers (same as your editor would use) to get precise definitions and references.
For example ("find references" on some open-source code):
Go: https://sourcegraph.com/github.com/theupdateframework/notary...
TypeScript: https://sourcegraph.com/github.com/ReactiveX/rxjs/-/blob/src...
It understands many build tools, but not all. Go, JavaScript, TypeScript, Java, Python, and PHP support is all pretty solid. If you use the most common build tools for your language, it usually works well. It degrades un-gracefully if you use custom build tools or scripts, unfortunately, but we're working on making it support more build tools and customization. Sourcegraph development is open source; if you notice any problems, let us know in the issue tracker at https://github.com/sourcegraph/sourcegraph.
My top piece of feedback is considering some form of a strict mode, where your vision is not best effort support support for some legacy tool for a language, but fabulously amazing knock your socks off good support for users who take the time to invest in making their build friendly with your search index.
It might seem tempting to go after the longest common denominator audience here, but personally I think it's far more valuable to show users how code search can be amazing if they do their part to invest in it, vs. code search being "kind of sort of useful" if you have a fallback.
And maybe my Xoogler experiences are showing here, though Bazel / Kythe are in my opinion the perfect vehicles to realize that vision.
We used Google's source repository functionality for a while because there is no real choice if you want to use Google Cloud Builder, which we were testing out for cloud-based Docker builds. GCB was pretty disappointing, and we moved off it.
We also found the source repository service annoying. GCB can't pull code from Github, so you first have to set up a Google source repo and turn on Github sync; the source repo is, in other words, only there to be a mirror. Unfortunately, this introduces problems. You get latency -- your Google repo is always behind. Worse, we frequently experienced a situation where the sync was broken, either due to Github being flaky, or the syncing itself being flaky. In such cases, it was impossible to tell what the status of the sync was; all you could do was go into the UI and see that it was not up to date; no way to force a sync. So this setup became a liability. At the time (not sure if this is still the case), there was also no API to set up new mirrors, it was all manual. The whole UI around GCB and source repos seemed to have been created in a rush, very bare bones. A lot of Google serviced start out like this.
I suppose this is a strategic play. For developers, you'd get a better UX if Google simply had tighter integration with Github and Gitlab, but Google probably doesn't want that. However, the low quality of their offerings so far (especially, cough, StackDriver) mean that this stuff will have an uphill battle in proving itself to developers, most of whom already have an allegiance to better tools like Github. If you're already on Github, why would you use this stuff?
Having worked for some of the big 5 American tech conglomerates, I would say this is the exact reason why they did not support it initially nor fully. The bigger the company, the more risk their is running/depending on someone else's software SLA/scaling/dependencies/security/privacy/laws... You just had a bad experience with Google cloud but you don't really know if it was google's fault or Githubs.
Plus, they have so many software developers, they can build their own version that conforms to their requirements themselves for practically the same costs.
>it was impossible to tell what the status of the sync was; all you could do was go into the UI and see that it was not up to date; no way to force a sync.
We've heard this was a problem for users so we're adding the ability to force a sync and to view the current sync status. These features will be live in the next few weeks.
First, have people register their origin repos. In fact, don't let people register individual repos; just their accounts or organizations. So instead of "Source Repositories", it's just a list of repos that GCP has access to.
Then, behind the scenes, mirror everything you have access to. All projects that you can read. Keep it internally. If you offer a UI to browse code and so on, pretend the code is remote. All references need to show https://github.com/someorg/myrepo.git as the remote, for example. Pretend your mirror doesn't exist.
Only add Github hooks for projects where real-time syncing is needed, i.e. build triggers.
Then, use Github hooks like you usually do, but have it fall back to pull-based polling when hooks aren't working (something I bet you guys can detect simply by noticing that hook activity for all projects is below average!). Anything that needs a build trigger could also have a manual button that checks the upstream.
Consider the developer's perspective. I already have my code on GH. I don't want to also store it in GCP. Why should I? It's right there on GH for any API consumer to read. Keeping it on GCP is just more work.
Being able to build with GCB without involving the indirection of a mirror would be even better.
What if you don't? The fact that Google needs to acknowledge that CSR isn't where existing projects are primarily hosted (and thus needs to accommodate projects primarily hosted elsewhere) doesn't mean they shouldn't try to be a primary host.
Obviously you have to point GCB at something. Today, you can only point GCB at SR/CSR. If you go to GCB and try to add a trigger, and select Github, it will go and set up a mirror on SR/CSR for you.
Github/Gitlab are still far more useful as they are more than just repos. The issues, PRs, wikis, and general management features are what most projects need. Azure DevOps is probably the best competitor from the clouds.
I am curious how they relate to say Github or other services.
Also as a sidenote, the Quickstart link for Debug in production and Detailed audit logs is the same. Is it so for a purpose?
>Also as a sidenote, the Quickstart link for Debug in production and Detailed audit logs is the same. Is it so for a purpose?
Thanks for pointing this out. It's not intentional. I'll make sure it gets fixed.
Github, Gitlab, Azure DevOps are also either free or very cheap and have 100x more features and functionality.
Previous HN discussion: https://news.ycombinator.com/item?id=14413035
However, Google has recently launched a raft of new CSR features, including semantic code search: https://cloud.google.com/blog/products/application-developme...
You get unlimited private repositories, 5 project users, 50 GB of storage, and 50 GB egress per monthly for free.
Monthly overage charges beyond the free quota are $1 per project-user, $0.10 per GB of storage, and $0.10 per GB of egress.
So for 10 project-users, 100 GB of storage, and 200 GB of egress in a month, you pay $5 for users, $5 for storage, and $10 for egress, for a total of $20.
A project as described here refers to a "Cloud Project" which is the basis for creating, enabling, and using all GCP services including managing APIs, enabling billing, grouping resources, etc. Within a project, one could have tens or hundreds of repositories. With 6 users on a single project, you would pay $1/month total (because the first 5 users are free) and all those users would have access to every repository in that project.
I appreciate your feedback that you find the pricing model and project concept difficult to understand. I'll bring this back to the team and we'll look at different ways to improve this.
https://cloud.google.com/resource-manager/docs/creating-mana...
"Project-user" here means that access is counted per project. So if you (x@) use GCR for projects A and B, meaning you check in/out code as x@A and x@B, that's two "project-users".
Disclaimer: I work for GCP.
https://cloud.google.com/images/products/source-repositories...
I used to work at google many eons ago and I spend 90% of my day wading through billions LOCs with their internal code search tool.
I believe it's the same one used by chromium: https://cs.chromium.org/
It is indeed using the same code search infrastructure as the Chromium code search site as well as the Google internal code search site.
You can read more about the Cloud Source Repositories code search feature in a blog post here: https://cloud.google.com/blog/products/application-developme...
Can someone explain (not sure, if I am missing anything) why should I move to CSR? If not, I will better stay with gitlab. TIA.
I've always viewed source control as a productivity tool more than infrastructure. With that mindset, it will take a lot more to convince people to do more than mirror their code here. Maybe that's enough, but mirroring has its headaches...