Hacker News Reader

Comment by wladimir | Hacker News Reader

It's 2012 and we're still navigating source code, which is effectively a nested graph structure, using grep and simple keyword queries.

No (reverse) call trees, no high-level views, no help on how data propagates through the program. We could do so much better, especially in static languages. Sometimes it seems like reverse engineering tools that work without source code (such as IDA) do a better job of cross-referencing than tools that have the actual source code.

Is there any good/usable point and click code browser / code comprehension tool these days? Or is the state of the art of 1998 still that of today?

View on HN

We're still doing this for two reasons: it's easy and it works surprisingly well. More complicated solutions would be difficult to implement and would probably be less flexible. It's just difficult to be significantly more efficient than somebody really good with Emacs using mostly code agnostic tools, except in real edge-cases (finding the right variable named x our of hundreds of xs is going to be difficult, but if your code looks like that you have bigger problems).

Also, I really wish the myth that "point and click" is inevitably superior would go away. It may be easier to learn, but my experience is that with some practice keyboard-based programming tools are almost invariably faster.

I wasn't trying to claim that "point and click" is superior. It was just my question whether such tools are available. I'm also more of a command line monkey, but it gets old at times to do the same thing I was doing 10 years ago over and over.

My point is that much more intelligent, useful and efficient queries could be possible by making use of the fact that code is a data structure, and not just text. Various things could be inferred automatically, which would be especially handy in legacy C source code where it's really hard to make head or tail of (I see a lot of that in my job).

It should be able to answer questions like "where does this value come from", "how is it computed" and show a tree/chain of statements. Or "where is this structure accessed" ... and so on.

As clang makes these things easier, I hope there will be a new resurgence in intelligent code comprehension tools. Those don't necessarily have to be point and click, but could have different UIs...

Google are planning on open sourcing a DSL for searching c++, apparently. http://www.youtube.com/watch?v=mVbDzTM21BQ

Modern IDEs already support the static version of what you want. You can ask Eclipse for all the call sites of a method, for example. This will not perform a text based grep, but a semantic code search. A dynamic version of this that works on a concrete execution trace instead of the static code base also exists. You choose a point in time on a timeline and ask "who called the currently active method" and it will give you the exact call site that called the method at that point instead of all possible call sites. For example researchers at CMU have created the Whyline. You can watch a video of it here: http://www.cs.cmu.edu/~NatProg/movies/whyline-java-demo-web.... Also related is a technique called slicing.

The Roslyn CTP for Visual Studio is an API for the C#/VB compilers. Visual Studio itself uses this for e.g. refactorings. You can write VS plugins that examine the abstract syntax trees of the files in the project, instead of working with the flat text.

It is very hard to do this for C/C++, because C/C++ code is customized at compile time by the build environment and usually can't be parsed and analyzed without that customization.

The tools therefore need a fully working build environment, and the ability to temporarily override aspects of that build environment.

People at Mozilla have created a lot of neat tools for working with large C/C++ code bases.

Relavent Links: https://developer.mozilla.org/en/Dehydra https://developer.mozilla.org/en/Treehydra http://dxr.lanedo.com/ http://corp.galois.com/blog/2010/6/4/tech-talk-large-scale-s...

Dehydra and Treehydra are GCC extensions; they have to be integrated (painfully) into your build environment.

DXR is a source code search engine. It appears to be cscope with a database. That's not nothing, but it's not exactly taking us out of the opaque blobs of text and into semantics.

I agree that it is pain to integrate. Once integrated, however, these tools offer a start on the road towards semantics. Here is an overview of some of the techniques used in Treehydra: https://wiki.mozilla.org/Abstract_Interpretation

DXR is currently a pain to get up and running. There's been talk on the mailing list on streamlining the install process but it would still require a lot intervention depending the compilation process. https://groups.google.com/forum/#!topic/mozilla.dev.static-a...

Does DXR do anything particularly interesting, as far as source code search engines go?

Interesting! I've seen DXR before but it used to be pretty limited, no advanced queries, and the project seemed to be dead. Nice to see there is at least work in that direction. Large projects such as Mozilla could especially benefit from it.

It's a hard problem indeed. Hopefully that will motivate some people :)

Is there any good/usable point and click code browser / code comprehension tool these days? Or is the state of the art of 1998 still that of today?

That's a lot to ask for code that the editor may not know the compile and link options for, which may not even compile using the right options, or which may only be understood by a compiler that isn't installed on the system where you're browsing the code.

Basically what you want is an IDE with a project set up telling it exactly how to compile the code or what the runtime setup will be for a dynamic language. Then you'll have decent code navigation, if the IDE is integrated with a compiler that can handle the code, and if there aren't any bugs! I can tell you that for Java the problem is thoroughly solved. The language is simple, and the IDEs are mature. For C++ you're SOL. Scala is getting there with Eclipse and IntelliJ. I don't know about other languages.

I for one enjoy knowing how to handle unstructured text, because you can never count on having tools that will understand everything you come across. XML configuration files, Java projects that have Ruby build scripts for unknown reasons, log files, there's an unlimited number of text formats that some tool somewhere knows how to parse, but it isn't worth your while to spend half an hour setting up an obscure dev environment for a one-off job that you can do in five minutes in emacs.

Eclipse gives you a wealth of tools to do that for Java. It's exceedingly rare for me to plain search (as in Ctrl-F) for anything in source code, I'm sure I go days without it[1].

It goes way beyond "find references" and "go to definition", too. For instance, marking a variable indicates all the usage points near the scroll bar, and ctrl-,/. moves between them. Ctrl-T opens an index of all classes. Ctrl-O has an index of the current file. Shift-Alt-Up/Down selects increasingly large expressions (across lines, as well).

I guess it all sounds trivial, but taken together I navigate within a Java project in an entirely different way than I navigate source code for other languages or unstructured files.

[1] Grep and its ilk are obviously still common tools for things like log files.

The best tool I have ever used for that is CodeSurfer: http://www.grammatech.com/products/codesurfer/overview.html.

It supports slicing operations: Which downstream changes happen if I change this variable? What upstream control structures influence the value of this variable?

Great stuff.

That's a great list of features, and exactly the kind of intelligent tool that I'm looking for. Too bad it's horribly expensive :/

Visual Studio has some fairly decent tools ("Find all references", "Go to definition/declaration"), although they tend to fall over in macro-heavy C++.

A lot of my coworkers swear by Understand, but I haven't tried it myself.

If only these tools would wirk reliably. I have seen too many cases when find all references did not find the reference I was looking for (even though it was in no way special or obfuscated). Goto definition works pretty well though.

This would be much easier if compilers/interpreters would expose their parse trees in a friendly way. They have all this knowledge about the code and won't give us.

I suggest taking a look clang/llvm. They have a friendly API for access to parse trees and intermediate metadata exactly for purposes like this. And it's open source too!

All the good (heavyweight) Java IDEs have had excellent cross-referencing tools for well over a decade at least.

Reply on news.ycombinator.com