Srclib: A polyglot code analysis library | Hacker News Reader

Srclib: A polyglot code analysis library

srclib.org

76 pointsanant32 comments

Thanks for posting this! I'm one of the original creators. It's still alpha, so there will be bugs. But it can do some pretty cool and useful stuff already in your editor (and it powers https://sourcegraph.com/).

Anyone have feedback about how we're explaining the problem and solution, and the first-run experience of using the editor plugins and the "src" tool? We would LOVE to hear it so we can fix/improve it.

Hey! Quick question.

Are looking for the source code to see if you're working on a PHP implementation. Hard to find any links though. Then I started the think, maybe it's not open source at all?

Didn't find any information so, is it open source? In that case, where?

There's a Github link on the website right below the email signup

https://github.com/sourcegraph/srclib

Did you guys consider hooking this up with llvm since it provides call graph generation for the supported languages? If so what was the conclusion? If not I might look into it, no promises though.

We haven't looked at llvm yet. None of the tools for the existing languages we've targeted so far (Python, JavaScript, Go, and Ruby) had llvm-based tools that we chose to use. It would be AWESOME to hook it up to llvm. We'll help however we can, if you are interested. I'm sqs@sourcegraph.com.

For a tool which is all about being language agnostic, I'm curious why you went with Go struct definitions to define the formats instead of something like JSON Schema[0].

[0] http://json-schema.org/

Simplicity and lack of time. :) We'd love to move to something more language-agnostic for the schema definitions. If you're interested, we'd definitely accept a patch.

Fair reasons :) I'm tied up for the next couple weeks, but I'll keep that in mind for September!

Looks really awesome! Love the Github and Sublime plugins.

Unfortunately I can't try it out as there doesn't seem to be a Windows version. Any plans for one?

We would love to have Windows support in srclib, but we honestly have no idea what (or how much effort) it would take. It might work in Cygwin as-is. If you or anybody else wants to try it, or at least identify issues, we would help.

One related thing is a lot of folks want to add C# support. I assume that running Roslyn requires Windows, which'd require Windows support for srclib. https://github.com/sourcegraph/sourcegraph.com/issues/157

Actually, Roslyn targets Mono as a first-class citizen, so it'll run on any Mono-supported platform :)

Not sure how much progress they've made on that yet though as it's not release ready.

Oh, that is fantastic!

From a quick try, getting it to build from source on Windows requires a minor modifications to task2/logger.go to remove the dependency on syslog and that's about it. But then src.exe toolchain -install-std fails on symlink creation via ln -s C\sourcegraph.com\sourcegraph\srclib-go C\sourcegraph.com\sourcegraph\srclib-go (looks odd btw). No stack trace to chase down to replace that with mklink, but it shouldn't be to hard to trace down.

Ah, thanks for trying this. It's probably making some Unix assumptions in the pathname creation.

Awesome project! Any timeline on vim support?

None of the original contributors use vim, so we wouldn't make a very good plugin for it. We'll help anyone who wants to make one, feature them on srclib.org (obviously), etc. Do you know anybody who'd be interested in making a vim plugin? A good place to start looking to see what's involved is in the code for emacs-sourcegraph-mode, sourcegraph-sublime, and sourcegraph-atom.

Sign me up. azmenthe, you want to help?

Someone else was interested, too, over at https://github.com/sourcegraph/sourcegraph.com/issues/107 (@lazywei).

Based on reading some of the docs (can't check out the source on this phone), looks like this has great potential; for a long time I've wanted something better than cscope, and not limited to a single language, to use with vim.

My main concern is that any system based on scanning entire source files offline is unlikely to have the same interactivity as IDEs. That is, if I type a line of C++, I don't want clang to have to churn through the whole file and any header files before the navigation updates...

Yeah, truly interactive editing and incremental updates are something that srclib doesn't handle right now. We have some plans/prototypes around using presentation compilers, lenient parsers, and a persistent "src" process to provide the same kind of interactive editing support that IDEs have, but there's nothing in srclib or any of the language analysis toolchains yet. We are /pretty/ sure it'll be possible to get the kind of responsiveness you'd want.

If you have time to take a look at how we've implemented the existing toolchains and architecture, it'd be awesome if you wrote down any suggestions you have for how we could implement interactive code assist.

Just serialize the continuation to the database and everyone can fork() the state language analyzer with their own queries.

Wow, this looks like an awesome project. As a pretty fanatic Emacs user who doesn't much like IDEs, I've always wanted a nice editor-agnostic IDE ecosystem that I could just plug into Emacs. I've seen language-specific things, which are great but naturally linked to their language, and Emacs-specific monstrosities like CEDET, but never anything trying to be truly universal.

Very promising.

For some reason though, I can't help but mentally parse "Srclib" as "Scribd" :/.

Thanks for the kind words! Please send feedback/complaints.

Re: the name "srclib", I was on the phone with our domain registrar when we got srclib.org/com, and the rep referred to it as "Sr. Clib" (as in "Señor Clib"). So that's stuck in our heads now. :)

IIRC IntelliJ does some things like this:

1. It's a polyglot IDE and open-source.

2. Specialized tools for exporting to multiple languages/DSLs.

I wasn't able to find the source code for some of IntelliJ's dynamic language support, like Python and Ruby. If it is indeed open source, I would love to check it out.

IntelliJ is awesome. We're just trying to make something that isn't coupled to a specific editor and is super non-monolithic. Our goal is for you to be able to run a single command ("src") in a directory, or have your editor plugin (or other kind of tool) invoke something like "src api describe --file FILE --start-byte N", to tap into a fully analyzed database of the source code in a package.

That's a potentially very useful project, thanks so much for working on this!

A couple of points on the website, https://srclib.org/

1. Can't find an actual Download link, even in the section called "Try it now". Oh - turns out the download links are under "Docs", which is kinda strange.

2. Downloaded for Linux. Unzipped, ran, got

    $ ./src --help
    bash: ./src: cannot execute binary file

I wonder if the links are correct, as all three download links on the Docs page (https://srclib.org/gettingstarted/) include the phrase "...?os=darwin&..."

Thanks for posting this.

Just fixed the download links.

What do you think of the new installation section on the homepage, at https://srclib.org/#install?

related work: https://github.com/facebook/pfff which focused on languages srclib didn't. Also a polyglot source analysis library.

Yeah, pfff seems very cool. It seems that pfff has written parsers and type inferencers for a variety of languages in OCaml. That is an impressive achievement.

For srclib, we are way lazier. We try to use existing libraries (written in the languages they target) for parsing, type checking/inference, dependency resolution, etc. We hope that makes it easier for JavaScript hackers to hack on srclib's JavaScript support, Python hackers to work on srclib's Python support, etc.

Yes, lately I've also tried to reuse existing "parsers" ... for instance I now use existing libraries for the java bytecode, ocaml typed AST (.cmt) and clang ASTs.

Looks awesome!

Hope to see Vim plugin soon.

Reply on news.ycombinator.com