Facebook Open Sources flint, a C++ linter written in D

Using third party lexer generators means your project is now dependent on that tool. This means:

1. the tool must exist on every platform your project will ever be ported to

2. the same version of that tool must exist on every such platform

3. if the hosted compiler changes, the tool must be updated in sync with that compiler - out of sync means your project won't build

4. if there are blocker bugs in the tool, you're not in a good position to fix it. You're stuck with developing workarounds.

5. Everyone building your project has to get the tool successfully installed.

The current build of D's runtime library has a dependency on libcurl. Supposedly, since libcurl is ubiquitous, that shouldn't be any problem. But it has been an ongoing sore relentlessly sucking up multiple peoples' time, of the issues I outlined above.

There's a fine line to walk between "Not Invented Here" and suffering through an inadequate tool. We at Facebook have a long history of trying stock tools and see them failing at our scale.

Walter Bright puts out a good argument against lexer generators here: http://www.drdobbs.com/architecture-and-design/so-you-want-t.... Besides, tokenization is hardly the interesting/bulky part of the task. The verifications are.

> Everyone uses regexes; using a lexer generator should (in my opinion) be as easy as using a one-off regex. I think the key is to make the lexer generator an embeddable library like regex libraries are.

There actually is a really powerful parser generator for D by Philippe Sigaud: https://github.com/PhilippeSigaud/Pegged It's much more pleasant to use than something like Boost Spirit.

Somewhat relatedly, D's standard library has a compile time regex engine that compiles regular expressions down at compile-time resulting in some of the fastest regular expressions in the world.

Writing parsers is fun, and having done it several times, I know roughly how long it will take. Using a highly specialized tool has risks of its own: that I'll sink a lot of time into learning it, only to discover it isn't a good fit for what I want to do, and that I'll have to go through the whole learning curve again a few years from now because I don't write parsers very often and will have forgotten how the tool worked by then. Then there's the fun of debugging Other People's Code.

Not to mention the way that open-source tools tend to drag in a godawful mess of dependencies, most of which they don't really even need. In my experience, I can write a basic recursive-descent parser from scratch in C++ before I can get a warning-free compile out of many open-source packages.

So yes, there are reasons to roll your own parser. They may not be good reasons in your view, but don't shoot the messenger.

(All of this being said, there's no way I'd try to bake a fully-compliant C++ language parser from scratch.)

Well one problem could be that C++ does not have a context free grammar, and in fact its grammar is notoriously difficult to parse, so perhaps it's just as easy to write it by hand as it is to bend an existing parser generator to fit C++'s grammar?

In fact, I am not aware of any modern C++ static analyzer or compiler that uses an automatic parser generator.

By the way: I'm glad you mentioned Ragel. It's a fantastic dependency-free state machine generator that everyone should check out. http://www.complang.org/ragel/

See my comment here https://news.ycombinator.com/item?id=7293796, to make anyone able to add a company-specific check quickly is a valuable asset (to ensure decisions like "We don't want to use idiom X anymore").

Moreover, fake parsers like cppcheck will lint your code even if the code does not even build, so they are really easy to setup.

Are there any studies available comparing development effort for the two approaches, using an existing tool vs. writing one's own? It is not obvious to me in how many percent cases the former is better.

I have many a times used an existing tool and after getting the thing to work, have come out with a feeling that it may have been easier to do it myself. I used Flex in a compilers course on Coursera, and getting the lexer to match the specifications took about full two days, which sounded too much for the specifications at hand in the assignment. I have written a CFG parser from scratch in less time (not to say that these two tasks are comparable from development effort point of view).

> Everyone uses regexes; using a lexer generator should (in my opinion) be as easy as using a one-off regex. I think the key is to make the lexer generator an embeddable library like regex libraries are.

Interesting read:

http://commandcenter.blogspot.ie/2011/08/regular-expressions...

Ouch! This seems to apply to me. I've written lexers manually in the past but never bothered to learn a lexer generator properly. I guess I should. Would Flex, Ragel or something else be the best starting point? I have unpleasant memories of really cryptic error messages from the compiler when trying to get someone's toy project in Flex to work.

Clang-format aims to provide this for C++.

http://clang.llvm.org/docs/ClangFormat.html http://www.irill.org/videos/euro-llvm-2013/jasper-hires.webm

http://astyle.sourceforge.net/ linters are more important that pretty formatting to my mind. c++ has had both linters and reformatters for a long time.

I suppose the distinction is that they are not built in.

Huh? You know that a linter is not a formatter, right?

well there is clang-format and clang-tidy and they are really easy to set up in CMake (I use "make format").

If you enjoyed that you should pick up The D Programming Language by Andrei. It's a good read. Metaprogramming with D is a dream compared to C++ so some of the techniques in Modern C++ Design aren't even necessary.

http://www.amazon.com/D-Programming-Language-Andrei-Alexandr...

> I wonder if anybody's tried it on a big C++ code base outside of Facebook?

I used a similar tool (cppcheck => http://cppcheck.sourceforge.net/) which also does not try to build a correct AST and does no proper semantics. The results are surprisingly useful, and most importantly it's very easy to add user-defined checks (edit: of course you also get some related false positives but it's very flexible).

It doesn't seem that flint parses at all – it only tokenizes, and then you write "rules" on "streams of tokens." It kinda sounds like a set of fancy regexes, but on tokens instead of characters.

D is actually a pretty nice language. (The caveats are around the libraries, especially the standard libraries to choose from.) I can see D being a better C++, in that it better solves that problems that C++ is supposed to be good for.

This is likely what they are looking at. Start with a program everyone uses, but isn't critical production system, evaluate the language and get people familiar with the language. Follow it up with some more rewrites and eventually give up on the old C++.

Not facebook examples, but things I've seen are as follows. All these were fixable to be correct for both GCC and Clang (it's worth it for the clang static analysis).

GCC can call the base destructor explicitly in destructors (an error in clang). i.e. this->~BaseClass(); This is normally a coding error anyway, since the call is added by the compiler.

You sometimes have to add this-> in templates to disambiguate which function is called.

GCC has, or had, a better (different?) two-phase look up for templates, so you can #include a file after its template is used in another header. Clang is stricter and requires you #include "A.h" and #include "foo.h" before using A<foo>, even if the overall compilation unit does eventually declare the A template and the foo class.

GCC supports variable length arrays, so you can do `MyClass x[someValue + 1];`, but with Clang you need to use a vector of MyClass, like vector<MyClass> x(someValue + 1).

This sort of thing compiles in GCC, but not clang:

    template <typename T>
    class List : public list<T> {};
    const List<string> tmpList;

It is missing a default constructor, GCC finds one from somewhere :)

GCC allows friends of derived classes to call static base class protected functions. i.e. Base::protectedFunc() can be called if the current class is a friend only of BaseSubclass. In clang you need to call BaseSubclass::protectedFunc().

Clang requires more explicit template instantiation to avoid link errors. Not sure exactly when, just "sometimes" and you'll know it when you see it :-)

Then as mentioned elsewhere clang shouts a lot about mismatched forward declaration (struct vs class).

Here are some examples that we've run into in the past:

Some of our code used GCC nested functions, which Clang did not support.

We were using __sync_val_compare_and_swap with __int128 values, which wasn't supported by the Clang version we were using. I believe the current version of Clang supports this.

Clang is stricter about some things than GCC, e.g. Clang doesn't like it when you declare a class as struct in one place and in a class in another place.

I think there is also a difference where Clang deduces that your enum is unsigned if none of its declared values are negative, and warns you if you compare it < 0. Since we compile with -Werror, this will cause compilation to fail.

None of these are unsurmountable, but they do mean we can't just drop in Clang and expect everything to work.

Here's a gem that clang produces when compiling folly:

"clang-3.4: error: unable to execute command: Segmentation fault"

I believe clang also doesn't currently support the "ifunc" attribute, which we also use in a few places.

D is crazy good at boilerplate generation.

On the contrary, a small standalone tool like a linter is a perfect place to start experimenting with a new language before introducing it to your core projects. I'd expect there'll be packages available for it soon if it's really too much effort to build.

Why? I'd rather put some crazy requirements in my build tools, than in the delivered product.

Eg hypothetical: who cares if it only builds on, say Linux? We have VMs for that. As long as the output program runs on the target platform, say Windows, we are golden.

Clearly that's something to worry about. What won our team over was the (sometimes spectacular) speed gains compared to the C++ linter.

Next sentences:

    Labeling the data as such potentially generates one instance of the
    static data inside each compilation unit, including that header. Fixing
    these issues has led to measurably smaller and faster executables.

When you include that header in two different .cpp files, which compile to two different .o files, you have that data twice, local to the .o file (compilation unit).

Now link them together and you have that data twice in the executable. You probably did not want that. (But -- as with some more of their examples -- maybe you did indeed want that?)