Sourcegraph Master Plan
about.sourcegraph.com
about.sourcegraph.com
Programming is hard because it's hard to define the problems that need to be solved, hard to design good solutions and often hard to implement them efficiently. A smart IDE is a useful tool but it does little to nothing to address the fundamental difficulty of programming. The real challenge to making programming more accessible is how to teach/train people to be better thinkers and problem solvers (and I include myself in needing that).
The blue box half way down is really bad on mobile. Too much margin.
This is because the best people to build language servers are the language maintainers/experts themselves. A great language server should be a core part of every programming language, just as most languages have packages for parsing their own AST, typechecking, etc. (E.g., the javac API, go/types and related packages, and the TypeScript API.) This is incentive-aligned because great tooling boosts language adoption.
Rust has done a great job here, with an official language server (https://github.com/rust-lang-nursery/rls). The Go team is investigating building an official language server, which we would love (we built an unofficial one and are happy to deprecate it in favor of an official one, or help make it official--whatever is easiest for the Go team/community).
For languages that don't yet have a good language server, we would dearly love to sponsor the creation of one, but it's quite time consuming to do so. It takes a while to find the right person to build it for each language (esp. for languages we aren't familiar with), and the majority of these projects have not yielded a working language server, so that hasn't been as easy as we had hoped. This is the main bottleneck.
And in all cases, we want to make it so there's a huge reward to building a great language server. When you build a language server, you can use Sourcegraph to run it for all code you view on GitHub, Bitbucket Server, and (soon) other code hosts. That means you get hovers, go-to-definition, find-references, etc., in code files and PRs in your existing tools. (See https://chrome.google.com/webstore/detail/sourcegraph-for-gi... or the Firefox extension to start using that.) The process for taking a language server and deploying it to Sourcegraph currently requires some intervention from us, but we are changing that with https://github.com/sourcegraph/sourcegraph-extension-api. This will also make it much easier to take a language server and make it work in all editors, without writing per-editor shim extensions.
We have, however, built and/or sponsored many language servers. (Those, in turn, build on compiler APIs that other people tirelessly created, to which we also owe thanks.)
This is where the deep experience of Microsoft in building developer tools together with the desire to be cross platform are shining.
As a user of Emacs under Linux for the past 20 years to code in Fortran, they succeed in me using Visual Studio Code[0] and hack into a Fortran Language Server!
[0]: https://www.ceondo.com/ecte/2018/09/vscode-visual-studio-cod...
BTW, very cool that you built the Fortran language server!
I am not the author of the Fortran Language Server, just fixing bugs and trying to improve it. But as the protocol is simple, this makes the work to improve it also way simpler. This is the part where I am now sure that the LSP will take over the world for the benefits of all the code editors, closed or open source, cross platform or not. And this is good for us as programmers :)
I started it as a VSCode extension (via LSP), and am halfway to getting a Sublime Text extension to work, and gave up on intellij. Emacs/Vim shouldn't be hard, but I have no idea how I would release updates. While I agree LSP is technically making it an "n-plus-m" problem, implementing the plugins that wire things up to the IDE is hard.
My main pain-points have been:
- Initial integration: Each IDE has its own extension framework, and it takes a lot of work to get my LSP server to work at all
- Deploying updates: Once I have it working, I then need to spend days automating updates for each IDE.
- Distributing LSP updates: For VSCode, I bundle the executable, but that means extension updates for every LSP update. I could have system level installation, but I don't want to ask users to take that risk (nor manage installers myself)
- Syntax highlighting: There is no good standard for semantic highlighting (technically textmate, in practice not). LSP doesn't include highlighting.
- Settings management: Each IDE has a different way of managing settings, and gluing that to the LSP isn't easy.
Basically, I have to spend a week or two of my free time to support an extra IDE at all (not well), which is frustrating. Probably a month or two for full support, and that would be for maybe a dozen users.
These are lofty goals, and sorely needed.
We've come a long way but there is a lot of "suckage" both in developing software and in using it still.
I wish Sourcegraph all the best :)
Take Spotify for example, a fantastic product by many measures, but which even so does suck in some ways. Being able to create playlists and to enqueue songs is a start, but think about a list as a special case, limited kind of graph, and think about what kinds of operations we are missing out on by restricting ourselves to just lists.
And that's not all. Spotify is also annoying in terms of looking at the currently playing list of songs. If you go someplace else and look at another list and then go back to the current playlist you'll usually be brought back to the top and if the list is long then you have to scroll a lot in order to get back to where you were.
And I would like to make arbitrary queries like "show me all playlists in which song x occurs".
And I would like to have access to the full history of songs I've listened to, including information about leaving the song before it was done, and skipping forward and back. And I would like to have the point in time in a song where I starred it or added it to a playlist.
Then I could query for example "show me all songs that I've listened to in the past six months, that I listened to at least 80% of the full length of, but which I have not listened to within the last two weeks" and play the resulting list.
Of course, the playlist being a simple concept makes it easy for users to understand. I guess the amount of people that want what I want is probably not many...
And there's another issue on top of it, the company 'crown jewels' problem. If Spotify makes it too easy to query Spotify, then it becomes ridiculously easy to compete with Spotify. Which isn't a bad thing necessarily, but not something we should expect Spotify to expend millions of dollars to enable.
Yeah that's true. But mainly the data I want to be able to query is the data that is directly connected to my own activity. If Spotify made it possible to get access to metadata about my activity and the songs that were involved in my activity then it's useful for users but still limits somewhat what a would-be competitor could do. Combine that with a ToS that forbids redistribution of the data and I think Spotify could provide this without helping would-be competitors too much.
Even if a would-be competitor had access to various data from Spotify they would still need to both negotiate agreements about streaming rights with the IPR owners of the music itself, and they would need a lot of marketing in order to get users. In other words they would need a lot of capital. Where would a would-be competitor even get that kind of money? I don't see why any VC or startup accelerator would be willing to fund someone to build this given how big Spotify is and how limited the group of people probably is that would decide to leave Spotify for this feature.
Whereas on the other hand, this kind of access to data could be a premium feature, and then it could be another reason for users that cared about these things to pay for Spotify Premium. Open source third-party clients could implement query engines, building on exactly what you mentioned in the first paragraph;
> Arbitrary data querying are literally the problem databases exist to solve.
Open source clients exists today, and using them with Spotify requires premium. If Spotify provided the data, someone would follow suit in making use of the data in open source clients. Which could further boost sales of premium membership, while at the same time like I said would still not make it too easy for would-be competitors.
Actually, the client wouldn't even need to be open source, it could be a commercial product, and by integrating with Spotify Premium they would still not be a competitor, and they would not need to go through licensing, and marketing might be easier as well. "Enjoy advanced interaction with your Spotify Premium listening experince by using our client."
Though, I do still see the point of your example.
I think making development as simple and accessible as possible is essential for our software-driven economy to scale efficiently.
Of course, there are no integration with current dev tools, but the core is there to make it work, maybe?
[0] - https://kythe.io
From our POV, the more people trying to accomplish these things, the better, because:
1. As developers and humans, we want these things to exist!
2. Our business and product is a code search and browsing tool, which benefits from better code intelligence/analysis. We don't sell language servers directly; we sell a tool that makes use of them.
(For interactive tools (editors, code search, and code browsers), which is what we're primarily targeting, LSP (https://microsoft.github.io/language-server-protocol/) is a better fit than Kythe.)
Sure, im totally ok with this vision, given this is how people create new things, even if it looks 'solved' in a given context.
> As developers and humans, we want these things to exist!
The more iteration, the better.. we definitely need more polished tools in this space.
> LSP (https://microsoft.github.io/language-server-protocol/) is a better fit
Oh thanks, thats kind of the context i was looking for.. in what problem domain you guys were pointing to.
It makes perfect sense giving the current goals of Sourcegraph, to create a more complete solution to the dev community it targets.
(I didnt wanted to sound the "Dont reinvent the wheel" kind of guy here, just curious about what you were trying to acomplish, and used Kythe as a reference to the problem domain)
Have you tried VS Code? I would highly recommend you try it if you haven't yet. I think you'll see what you are missing.
(Of course, Emacs has many awesome features/modes that other editors lack, too. The same principle applies there.)
For example, the Go language server (if enabled) in vscode-go is only responsible for hovers, definitions, references, and a few other things. Auto-imports, code lens for tests, helpful commands, coverage, etc., are all implemented in vscode-go's TypeScript extension code.
I recommend just trying VS Code for the languages you use in Emacs to see what you're missing. I bet a blog post about your experience would get to the front page of HN. :)
(One solution, of course, is to make it possible to write all of those things in a way that can be consumed by all editors. Check out https://github.com/sourcegraph/sourcegraph-extension-api to see our work to let you write any kind of editor extension in an editor-independent way (not just the language/code intelligence features). When this gets into beta, we'll feel comfortable updating this master plan to denote the progress toward the goals.)