Apple's investment into Clangd and refactoring tooling | Hacker News Reader

Apple's investment into Clangd and refactoring tooling

lists.llvm.org

289 pointsartisdom79 comments

So, just because there seems to be confusion:

Clangd already exists. It already supports LSP. It is already part of the llvm project. It was written by !Apple, and used by a wide number of people.

(I want to make sure these people still get credit, i already see people in this thread thanking apple for supporting LSP, which is not actually what happened).

What apple has proposed contributing is: Making clangd work with xcode, and extensions to LSP.

This is great (as long as the second ends up standardized over time).

But the benefits people listed here so far already exist in clangd. The main benefit to work is "xcode support" and "better refactoring" which is awesome :)

The proposal is to split the LSP implementation into a transport layer and a logical layer. The current LSP implementation is directly tied to JSON-RPC.

Apple's end goal is, as you say, to get clangd to work with Xcode. However, having a replaceable transport layer for LSP could benefit lots of people other than Apple.

Sure, it's nice, but again, being pragmatic, nobody else has desired such a thing that much.

Ie so far the "lots of people" have not materialized, even outside clangd.

I can certainly think of use cases for such a thing, but most people seem happy enough with json-rpc at the moment.

Part of that is likely that trying to speak LSP over a non local connection may be a fools errand anyway. It's a chatty protocol no matter what the transport.

Just because Apple is the first doesn't mean there won't be others. Maybe others have not materialized because the upfront cost of updating clangd was judged too high.

JSON-RPC has several alternatives that very large software companies have vested interests in: gRPC and XPC.

You seem more knowledgeable of these technologies than I am, though. It just seems to me a good thing that 1) Apple, and consequently, all iOS and macOS developers will be using clangd, and 2) Apple is putting significant effort into improving clangd. I'm not sure why any of that news or optimism needs to be quenched.

My point was precisely that plenty of languages have LSP implementations at this point, an f even in those, nobody has cared much to try to find an alternative to json rpc.

As for 2, all I'll say is based on history in llvm projects, I'm not anywhere near as optimistic as you. The history here is simply not one of all great things. The PR is, but the reality is much more mixed (some really great things, some really not so great). So while I certainly am hopeful, it's definitely not the unbridled optimism everyone displays here.

You don't think apple's support of llvm and clang has been important to the project? That's REALLY pushing it.

That's not what i said at all. I said some things have been really great, some really not so great. This isn't even a controversial view among long-time llvmers.

Heck, last time i asked him (a few months ago), even Chris Lattner agreed with my view, and he used to run his stuff at Apple, so i don't think it's unreasonable.

I think it's perfectly okay to be cautiously optimistic instead of fawningly happy.

Do you actually have data or a real argument you'd like to use to suggest otherwise (IE that it's been all overwhelmingly positive?)

While I agree that not everything is always great, I'd say the same for our own contributions ;) I've definitely let patches unreviewed for way too long due to internal priorities, and have gotten my share of shit at conferences for that...

In this case, the collaboration with Apple and Ericsson on the underlying architecture has been going on for a while, and while not everybody always agrees on priorities and design details, I'm thrilled to see Apple double down on their involvement here!

The [follow up](http://lists.llvm.org/pipermail/cfe-dev/2018-April/057672.ht...) message suggests that Google already internally using a non-JSPN-RPC transport layer.

Yeah, but that is likely to actually go away, not happen more. It's not a desired thing in that case, it was just easiest at the time for various reasons. We didn't want the public project to have to support our special snowflakism, hence it was done a different way.

clangd already has a layered architecture, where you can either use the JSON-RPC mechanism or more directly control clangd and implement your own transport. There are no plans for this to go away, as generally large companies often have environments in which they want better control of features than a standard protocol like LSP can give you, and we don't discourage that.

We also just talked with Apple folks at Euro LLVM about this, and while the plan for the near future is for them to have a transport conversion to JSON-RPC, I expect them to want the more fine grained control in the long term.

Really? I desire a protocol which is not coupled with JSON-RPC.

I'm glad to see that Apple is putting work into making their tooling better and easier to work with, but I'm still not clear on the benefits of this change. Would someone care to enlighten me? Currently, the benefits of libclang that I see are that I can easily drop it into a project without requiring a lot of dependencies, and I don't need to figure out how to get permission to run different processes and manage interprocess communication between them–if anyone doubts that this will be an issue, on iOS neither of these is really supported at all. In addition, I don't require a daemon to constantly be running to analyze my code. (Why is this bad? Take a look at Swift's sourcekitd.)

> if anyone doubts that this will be an issue, on iOS neither of these is really supported at all

What for you might need it on iOS? Server is meant to run on a development machine.

> What for you might need it on iOS? Server is meant to run on a development machine.

While I totally agree, I can see the author's point. Think for example about Swift Playgrounds.

> Server is meant to run on a development machine

There's no reason to discount iOS as a development machine–it's perfectly fine for coding, as Swift Playgrounds shows. (Actually, there are some issues with executing code in third party apps, but I'm working on a way to bypass the restrictions around this. Check back again soon!)

To be able use it from any editor that is able to understand LSP, instead of having everyone writing their own bindings to libclang, specially if the editors are written in type safe languages.

Yes, exactly this. If you have 3 different editors and 3 different languages, you potentially end up with 9 different implementation. LSP means that the same can be achieved by one implementation per language and one plugin per editor.

With LSP, you end up with one subpar implementation, and you need to duplicate all refactoring functionality for each language.

While with common IDE plugins, you have one implementation of the refactoring functionality, and each language plugin just exposes an AST.

So LSP actually leads to more duplicated code and worse editors.

VS Code will never be able to match the functionality of IDEA unless each language server reimplements all the functionality slightly differently. What a mess.

And no other editor can implement IDEA’s feature set except from scratch. What a mess.

Or they can start off with reusing IDEA’s stuff, and building an LSP 2.0 based on it.

LSP is already on version 3.7, and publishing an update every few months: https://microsoft.github.io/language-server-protocol/specifi...

Is your complaint that they're not iterating fast enough? Because this is something I'd prefer they take their time and get it right, especially once we get past the low-hanging fruit and into complex things like debugging.

My complaint is that the entire concept on which LSPs are based today — each LSP reimplements all functionality itself, the editor is only a dumb textview — is wrong.

Language servers should expose an AST, not some actions. Language Servers shouldn't even know where the cursor is — or how many there are.

With a proper LSP, you can also write a linter just by using the protocol, you can call the same action on a thousand places at the same time, and the refactoring functionality is implemented only once, and reused everywhere.

The entire design of LSP is flawed.

LSP supports refactorings via Code Action Request messages.

Oh, you're right. We should just dump all other editors and use IDEA. /s

Or maybe we should build an LSP that exposes an AST in a way that is more useful to IDEs, like IDEAs plugins do today.

In fact, the code in IDEA that does this is open source, so you can start with it, and build a better LSP based on it.

This would be an interesting idea, provided that editors go all-in on supporting such refactoring logic on the client side. The IDEA code defines "extension points" so language specific support for common actions can be implemented in a modular fashion. When the user decides to execute a refactoring, it is just the matter of iterating through the loaded extension points and choosing the one for the language being edited. Having worked with the IDEA code myself to build an LSP server, there end up being language-specific parts such that the support between e.g. Java and Kotlin have to be handled separately. However that was probably due to the way I wrote the server; these could be generalized to a single "refactor" entrypoint with some work. It's just that using the current LSP model with the server being the part with AST awareness, there isn't much gain in doing so.

Can you go into more detail please on why it had to be subpar?

Technically solvable if somebody writes a LSP proxy server that merges together backends.

I'm implementing an LSP server using IDEA as a backend, so hopefully VS Code will be able to match the functionality sometime.

https://github.com/Ruin0x11/intellij-lsp-server

That being said, as an implementor I'm finding cases where LSP's feature set is underdeveloped. For example, there is nothing in the spec to account for renaming files, so I can't put in my rename action without the behavior being incorrect (because if you rename a class in Java you have to rename the corresponding file, which IDEA handles). Also some things require custom LSP requests, like getting the list of build configurations to run. I think this is a product of the Java support in IDEA being so much more comprehensive than anything in LSP so far. But hopefully as the spec is improved these issues will be worked out

You mean, with common IDE plugins, you end up with one implementation of refactoring functionality per IDE. That's what's being avoided here (because, for one thing, there are far more IDEs than you think, and they don't deserve to be second-class citizens.)

No, all IDEs could use the same AST backend.

Just like LSP. Just a more intelligent protocol.

You have one implementation of refactoring, instead of, as today, one per language.

I'm no expert, but one major distinction I can see is that libclang would be used in (editor) process, and a clangd server would operate outside the editor process and communicate over a protocol with the editor.

// Why is this bad? Take a look at Swift's sourcekitd.

Actually, if this were just a lib in Xcode, then instead of sourcekitd crashing, Xcode would crash all the time. Would you really prefer that?

You're always free to spawn your own process, which links with libclang, if you really want to and communicate with it over XPC. But then again if this crashed Xcode all the time maybe Apple would have more incentive to fix it ;)

LSP converts a M x N problem (M editors, N languages) into a M + N problem.

Just to clear what seems to be some confusion in what you wrote:

Clangd already exists (it was contributed and mainly maintained by Google, IIRC), and works great. It supports LSP, and works fine with vscode.

What apple is investing in is making it their default tooling.

So the benefits of this change are exclusive to apple xcode (ATM) :)

The benefits to using clangd in general are (among poher things) editor agnosticism and scaling.

If everyone uses libclang, libclang must scale to every codebase (or a least, the interface, and then you'd have to replace/relink it).

If everyone uses LSP, you just need something that speaks LSP, and scales to your codebase.

At the scale of Google's codebases, for example, the first is untenable, the second is doable.

Well, I mean, it's a little more than that unless clangd is straight up finished software. Apple will now be contributing to clangd as well, which benefits all clients of clangd.

That is why I said "at the moment". It also depends heavily on what extensions are contributed and how and whether they are standardized, etc.

Certainly missing LSP functionality helps everyone, but the rest depends quite a lot on approach and details.

The main problem with libclang is actually that it puts another abstraction on top of the AST that nobody really had time to maintain. That's not made better by the amount of work changing libclang requires just due to its backwards compatibility promises.

There is an article from 2010 titled "Emacs is dead" that is being reposted on HN from time to time. The author args that the greatness of Emacs is to rely on external, editor agnostic tools where text act as an universal interface/medium but that the practice is fading out, and thus the dead of Emacs. Now, ten years later, Apple announces this. Emacs, the undead?

https://tkf.github.io/2013/06/04/Emacs-is-dead.html

I think you might be right, but the greater context is highly ironic.

The company resurrecting the Emacs concepts is Microsoft. They're the ones that created OmniSharp, bringing the idea of language servers to the programming mainstream. They then created the Language Server Protocol for what now is Emacs's biggest long term rival in the flexible editor arena: Visual Studio Code.

Visual Studio Code is for the moment on a somewhat bloated and shaky foundation because of Electron. But otherwise its design is quite solid. And even that foundation will probably become a lot stronger in the next few years as WebAssembly gains wide adoption. Visual Studio Code itself is written in Typescript and it's not hard to imagine Microsoft adding a WASM backend to it.

Here is how Lucid implemented their C++ knowledge repository for Energize C++.

"Foundation for a C++ Programming Environment", chapter 7, Protocols

https://pdfs.semanticscholar.org/84a4/8824fc7dd872414efa0ad6...

And how it looked like with their Emacs customization in 1993 on UNIX.

https://www.youtube.com/watch?v=pQQTScuApWk

Lucid went bankrupt as far as I remember :)

Just financially, but never intellectually at least. ;)

And that proves what?

Sadly the best ideas aren't always the ones that get widespread adoption.

Here we are now, about 30 years later still trying to get major C++ environments to offer some of those ideas.

It proves nothing, it's just about my original comment about Microsoft:

> bringing the idea of language servers to the programming mainstream

Lucid failed to do this, Microsoft succeeded.

The simplicity of you statement ignores what it meant to get a computer running Energize C++ properly in 1993 and an Electron (VSCode) app in 2017.

Specially the money part.

The Microsoft Tablet PCs and the Apple Newton failed because the hardware wasn't there yet. C'est la vie...

I think Emacs's greatest strength comes from the ability to use a single scripting language throughout the whole editor. I can quickly write up some ELISP statement to do anything, if I like it I can put it in a file and load it. If I want to share it, I put it in an extension.

I've just looked up the official tutorial on making an extension in VS Code. It seemed cumbersome. You need a code generator to lay the foundations, you need to consult the API docs etc.

With Emacs you can learn as you go and the editor itself guides you if you ever get stuck. You can read the source of any part of Emacs you like, change it on the fly, evaluate and try it (except for some core modules written in C). And I think that's the beauty of Emacs. Emacs is easy to tinker with.

Yeah, but I have a million monkeys^Wdevelopers at my back for Javascript/Typescript. How many divisions does Lisp have?

Out of curiosity I looked it up. It's a factor of 200: 3788 packages on MELPA, vs 650000 packages on NPM.

Seeing that this has nothing to do with tools where "text acts as an universal interface/medium", no.

It still concerns "external, editor agnostic tools" -- but that wasn't dying out in 2010, and is not coming from the dead today.

JSON-RPC is text based

https://en.wikipedia.org/wiki/JSON-RPC

It's a parsable format. Not some arbitrary stream of text, like what UNIX streams deal with...

What do you mean by "arbitrary stream of text"? I fail to understand, to be honest. A text always need to be parsed, even a simple number, but the point is to deal with characters instead of raw binary data.

>What do you mean by "arbitrary stream of text"? I fail to understand, to be honest

That it's structured and specific.

UNIX pipes, for example, can deal with any kind of text -- it's up to the user to ensure that the commands (e.g. "cut -d," etc) work ok with them, whereas the JSON RPC communications of the language server are not only in a specific syntax (JSON), but they also have predetermined content schema.

"arbitrary streams of text" are also parsed. Line breaks matter, whitespace matters as well, and in some cases tabs and spacea matter.

There's a difference if I tell you: I will send you any kind of text at any time, it's your responsibility to deal with it and parse it accordingly to whatever it might be, than if I tell you, I'll send you JSON with this schema.

Looking forward to it.

It is really nice to finally start having C++ environments close to the promise of Energize C++ and Visual Age C++ 4.

https://www.reddit.com/r/programming/comments/25r6pw/a_demo_...

They mentioned incremental compilation in the first few seconds of the video. Didn’t watch the rest of the video yet.

I’ve always wished for incremental compilation. Imagine incremental compilation that was so fine grained it would recompile only the functions you changed. Of course this makes full-program optimization impossible but I think most of us don’t compile at the highest optimization levels during development anyway because doing so slows down builds without any benefit for development builds. For production builds we use the highest optimization levels of course.

Additionally I would like to have hot patching. I know some people have this, personally I’ve never had that.

If anyone knows about any systems for hot patching Rust code please let me know :)

While not perfect, Visual C++ makes it a nice experience.

https://blogs.msdn.microsoft.com/vcblog/2016/10/05/faster-c-...

https://msdn.microsoft.com/en-us/library/4khtbfyf.aspx

The future is a lot easier to build when you're vertically integrated. The risk with vertical integration is that keeping all parts of the integrated stack at the cutting edge is too costly and you start to fall behind.

A marketplace approach with broad consensus around interfaces is ideal, but it's hard to bootstrap. LSP via VSCode seems to have done the job.

I'm assuming this is the "LSP" discussed? https://microsoft.github.io/language-server-protocol/

Mapping json-rpc onto xpc... Sure why not

He said that's an intermediate step so that they can get to testing faster.

Woah I had never seen this before. That's really cool. So in theory you can make your own editor and provide a ton of IDE-like features by communicating with this server?

Are there any other implementations of LSP by other orgs?

https://langserver.org/

I have mixed feelings about this.

On one hand it's really nice to see Apple jump on the LSP train and use standard tooling for their IDE to interact with language tools.

On the other hand, instead of having a consistent strategy of using proper LSP (which by specification is JSONRPC), they are shoehorning LSP on top of their in-house XPC transport. To do that they're planning to introduce additional complexity to clangd (a transport abstraction), while actually defeating one of the main purposes of LSP, which is reducing the m-times-n complexity problem of matching compilers with tools to an m-plus-n complexity problem. No other tools (unless they support LSP-over-XPC as well) will be able to talk to XCode. XCode won't be able to talk to other tools. I hope they rethink that decision, keep clangd simple and instead adopt proper (JSONRPC) LSP in all of their other tools for Swift, etc instead. That way they'd not only open up XCode to clangd, but also all other editors to their refactoring tools for Swift, etc.

Could someone explain what's the difference between both approaches and what are the advantages of Clangd?

In process vs out of process; and language-specific binding vs JSON-RPC.

Moving stuff out of process means, increased reliability and potentially ability to share data across multiple clients.

Moving everything to LSP with two supported transports (JSON-RPC and something more native) means that editors like Emacs, vim, VSCode can benefit from tooling without needing to link against a library and work with C bindings.

Moving language parsing/refactoring/etc into a separate daemon is a step towards unifying tooling around specific protocols for interacting with compilers. Microsoft's effort towards a single language server protocol is worth mentioning: https://github.com/Microsoft/language-server-protocol

I'd love to see a future when new languages/compilers can implement this protocol for their external compiling daemons and just get support in all the various text editors/IDEs automatically.

Clangd implements the "Language Server Protocol" (LSP). More info about LSP here: https://microsoft.github.io/language-server-protocol/ https://langserver.org/

A quick cut-and-paste from the first URL explains why LSP was created: "Adding features like auto complete, go to definition, or documentation on hover for a programming language takes significant effort. Traditionally this work had to be repeated for each development tool, as each tool provides different APIs for implementing the same feature. A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication. The idea behind the Language Server Protocol (LSP) is to standardize the protocol for how such servers and development tools communicate. This way, a single Language Server can be re-used in multiple development tools, which in turn can support multiple languages with minimal effort."

I hope that it also means there will be Apple-provided LSP server for Swift soon.

There is one already: SourceKit http://www.jpsim.com/uncovering-sourcekit/

This is great news! More parts of xcode are being exposed as open source and clangd is getting more engineering resources.

Does anyone have any details about Apples cross language indexer they mention? A good indexer is the key to and efficient language server and I think clangd’s is a little weak right now.

I tried building Clangd in trunk but failed to build.

We build and use it every day, so ... report it on the list?

Reply on news.ycombinator.com