Hacker News Reader

I don't disagree, but these points needs highlighting:

> It's easy to complain about complexity, harder to offer a simpler but equally capable alternative.

Want real stuff? Check out VPRI's work: http://vpri.org/html/work/ifnct.htm Right now, they're working on a 20KLOC OS (including desktop publishing, messaging, and the whole compilation chain). That's about 4 orders of magnitude smaller than current systems. Here is their last progress report on the STEPS project: http://www.vpri.org/pdf/tr2011004_steps11.pdf

> The most important part by far is to leave out anything you possibly can.

This is also the most overlooked part by far. I deal with something similar at my day job: I noticed that I am relatively poor at dealing with masses of complexity. My co-workers fare better. On the other hand, they don't mind small unjustified complexities. The latest bit I saw was this (in C++):

  class Foo {
  public:
    out_t1 bar(in_t1 in);  
    out_t2 baz(in_t2 in);  

  private:
    static mem data;
  };

  // example
  Foo foo;
  out = foo.bar(in);

As it turned out, this "class" didn't have any state whatsoever. There was `data` of course, but it was never modified after its first initialisation at program launch. I devised a simpler interface (which happens to be backward compatible):

  class Foo {
  public:
    static out_t1 bar(in_t1 in);  
    static out_t2 baz(in_t2 in);  
  };

  // example:
  out = Foo::bar(in);

It's not much, so they say it's no big deal, and act as if it does not count. But it adds up quickly, often to the point where I can reduce the line count of a procedure by half, without even understanding the code! (I know how to do correctness preserving modifications.)

I've seen these VPRI links before. I've read through that PDF and various web pages, but I still have a poor understanding of what exactly they are doing.

Is it possible to summarize in a paragraph how they are able to achieve this LOC reduction? Is it simply that systems like Linux et. al. have been cobbled together by many hands over many years while VPRI has a single vision? Are there coding techniques I can use today in my own work? The only example I've found was a mention of some networking stack that was able to make use of a parsing engine, rather than implementing a custom networking-specific engine. This sounds like nothing more (not to trivialize this task) than choosing the correct parts to turn into reusable libraries and then reusing them.

> they don't mind small unjustified complexities

This is something I've noticed as well. Complex code of my own creation grates on my nerves until I am able to erode it and smooth it out. I get the impression that not everyone feels this way.

> Is it possible to summarize in a paragraph how they are able to achieve this LOC reduction?

I think it hangs on two principles:

1) more expressive tools at the price of performance.

2) use existing data as much as possible, but only complying with standards where really necessary.

1) is using e.e. Ometa & friends, which make it possible to (e.g.) implement a reasonable JS engine on top of any reasonable dynamic runtime with 100-200 lines or so.

2) is using e.g. the RFCs the define IP, TCP and UDP as input to a processor -- thereby, having a guaranteed-to-match-specification implementation, without having to repeat any of the struct/constants/details. The RFCs do not count among their 20K lines -- they are probably 3K lines themselves. The parser that parses RFC into data structures and some executable code IS counted in those 20K - but, last I looked, was less than 100 lines.

1) But as it turned out, the cost wasn't so high after all. (They didn't expect their graphic stack to run fast enough in a laptop, yet it does.)

2) What, they do not count the RFC they parse? I'd count that as cheating: if the RFC is part of the meaning of the system, it should be taken into account.

> What, they do not count the RFC they parse?

No, they don't.

> I'd count that as cheating: if the RFC is part of the meaning of the system, it should be taken into account.

I don't think it is cheating. The RFC text is essentially arbitrary, and "god-given" as far they are concerned -- and must only be adhered to for compliance with other systems. (If they designed their own network protocols, I would have agreed with you). By parsing it and generating code, they guarantee that the code conforms with the spec (which is more than you can say for any other network stack implementation!).

And it's all down to what they are trying to achieve: They are NOT trying to minimize LOC -- if they wanted to do that, they'd build everything in obfuscated APL or J.

They want to get to a state in which a person can understand / audit the entire software stack of a system they are running. 20k was selected as what a person can completely follow & understand in a reasonable amount of time (weeks, not years or centuries)

You cannot (average human "you") understand/audit a TCP/IP stack without reading the RFCs; therefore, they don't think they need to double-count it, and neither do I. Furthermore, if tomorrow the world switches to curveCP, they wouldn't include the related RFCs either -- only the code they had to write to make it work.

I count 3 "miracles". In ascending order of awesomeness, these would be:

(1) No feature creep. They provide essential functionality, little more. I say that's a good thing, because if you miss functionality, the system lets you build it relatively easily. My guess is, it explains about 1 order of magnitude (it divides code volume by 10).

(2) Factor everything. Again not very impressive, but it goes a long way. Just good engineering principles applied systematically across the system. For instance, they have one graphic stack, which draws everything in every program, including the window manager. They also have one document type. A side effect of this approach is a greatly increased integration. My guess is, it explains about 1 order of magnitude.

(Warning: the divide between (1) and (2) is somewhat arbitrary. In some ways, the STEPS system provide more functionality than current systems. I think the trick is to stop thinking in terms of features, but in terms of capabilities. For instance, UNIX don't have a feature to sort Apache log files by IP address. But it has `sort` and `awk`, and therefore it has the capability.)

(3) Build the right tools. The tools being mostly domain specific programming languages. And that is the impressive part. Their language are so expressive and simple that they can reduce code volume by two orders of magnitude without breaking a sweat. I don't know exactly how they came up with such languages, yet they did. The tree core languages that may interest you here are Ometa http://www.tinlizzie.org/ometa/, Maru http://piumarta.com/software/maru/, and Nile. Ometa is like Lex+Yacc, only much simpler and more general (it lets you transform trees, and flatten them, which makes it suitable for all compilation stages). Maru is a not-so-inefficient self-implementing language, based on 2 cores (lambdas and objects) which can implement each other.

So, 1+1+2=4. Or 1010100=10,000. I'm oversimplifying of course, but I think this gives an idea.

The other comments here already explain how they do it on a high level, here I'll try to explain it on a slightly more technical level.

They use DSLs very heavily. The heart of their system is OMeta, a high level pattern matching and transformation language. It works both on flat data (in which case it acts as a parser: transform flat text into structured data) and on structured data, transforming it to different structured data. The compilers for all their DSLs, including the OMeta compiler itself, are written in OMeta.

All these compilers are very short (very roughly 200 lines each?). Because they can define new languages with so little code, they pretty much have a different language for each problem they want to solve. For example for the graphics rendering they define a concise data parallel language, they have a low level intermediate language, and a couple of others (e.g. Maru (Lisp like) and Ants (for implementing WYSIWYG editing) and amusingly the TCP RFC).

Of course DSLs are not enough. If you wanted to make an exact copy of Ubuntu or Windows then no matter how good your DSLs, you're not going to make it in 20k lines. So they simplify the personal computing stack a lot. For example they don't have separate programs for document editing (MS Word), presentations (PowerPoint), internet browsing (NOT the web -- their own web format instead of HTML -- you're never going to be able implement the traditional web stack in 20k lines), email, spreadsheets, etc. Instead they unify all of this in their universal documents. So they don't try to copy, but they try to get something that's functionally equivalent. The result is pretty powerful and potentially more useful: you can put a spreadsheet table in a document, in an email, on an internet page, etc. When they extend the universal document with a new feature (for example graphs, or math typesetting) then all these individual uses benefit, instead of doing each feature N times as we do now.

Links to source code repos can be found at http://www.vpri.org/vp_wiki/index.php/Main_Page

Aside from the things described by the other commenters, a big thing they're doing is accepting major performance hits that are no longer relevant. xterm on this netbook renders text onto the screen by copying prerendered glyph rectangles loaded by the X server from PCF files into the framebuffer, because that's how you had to do it in 1984, because you were on a barely-1-MIPS workstation, and if you tried to do something fancier at the time (including masked copies, so you don't need to have a character in your font for á, but can overstrike a ́ and an a) xterm would have displayed text noticeably slowly. But this level of optimization is kind of a waste on this 1000-MIPS netbook, let alone a 30,000-MIPS desktop machine. And it costs a lot of code complexity.

By contrast, STEPS treats each letter as a polygon or group of polygons, and rasterizes your polygons when it's time to draw them. So, among other things, you can rotate, scale, and color your text just like any other object, and the amount of code devoted to text rendering is tiny. The performance cost may be one to three orders of magnitude in that case, but that's acceptable for an interactive windowing system.

> Want real stuff? Check out VPRI's work

I didn't mean to make it sound like simplicity is not possible. Quite the contrary, I believe it is possible and extremely worthwhile, and it's something I have dedicated an enormous amount of time to in my own work.

All I'm saying is that the essay was not worthwhile to me, because it offered no insight that helps us get there.

> It's not much, so they say it's no big deal, and act as if it does not count.

I agree that your version is far superior, and that it is a big deal. If I saw code that constructed a class but the class didn't have any state, it would interrupt my flow while I stopped to figure out wtf is going on.

I didn't mean to imply that you meant what you didn't mean… That was poor phrasing on my part. Actually I agree with you. Saying that something sucks is worthwhile only when few know it. When everyone does, we ought to take the next step, and propose solutions (or at least analyse the problem more deeply than it has been). I was just saying "there! There! A solution!!".

The failure isn't in that we're adding too much complexity (we have to be able to create complex structures) or that we're not layering it enough (leaky abstractions build up). It's a general, and much more far-reaching, problem: our abstraction mechanisms are failing us. We need to stop complaining and put more effort into research. There are four or five projects working on this and, literally, hundreds of photo-sharing startups.

Any links to such projects?

VPRI's STEPS project is one such project: http://www.vpri.org/pdf/tr2011004_steps11.pdf

"our abstraction mechanisms are failing us... There are four or five projects working on this and, literally, hundreds of photo-sharing startups."

But it's the same thing! It's the same problems

And IMHO in the technical places it's getting worse every day.

The reason some photo sharing apps are bought by a billion dollar is the same as why lots os people use PHP even though "it sucks". User experience and ease of use.

You don't need to limit yourself to a high level abstraction (like gnome) or provide the whole "under the hood" controls like you're flying an aircraft.

Complexity has to be managed and split into stages. C'mon, even VIM has a smooth learning curve (and not a learning abyss, or being shut out)

And stop the very prevalent mentality in technical circles of "if you're technical you have to read the manual". User experience has a place even when editing .conf files or using a command line tool, but some people make it out of the way harder just to make it accessible only "to the true believers".

Yes, everything is mixture of complexity. When an artist paints, she has to decide how simple or complex to make the work. But what we're talking about here is that once you pick up a PHP brush, you're screwed. That's why it sucks. You can't change brushes; you have a big fat crayon. And while that works for filling in the sky, you can't get any detail on the faces, for example. (Choose a fine brush and you have a different problem of spending all your time on the sky.)

What I'm talking about when I say "our abstraction mechanisms are failing us" is the need to use any size brush at any time.

And, as an aside, photo sharing apps are bought for billions for the same reason that tulips used to be worth a fortune. Uploading pictures of your cat is about as valuable as a 'Carnaval de Nice'.

Btw Instagram is an example of opinionated simple interface. You might be underestimating the complexity of creating such interface.

To add to your point: physics shows that it is possible to describe complex phenomena using simple models. But discovering such models is hard.

I entirely agree with you - but I think the hardest thing is SPLITTING THINGS UP into coherent units that compose the lower level API you write about. The problem is that we can divide things in many possible ways - what makes one division better then the other? This is hard, layering is the easier part (but still far from obvious: http://perlalchemy.blogspot.com/2012/04/breaking-problems-do...).

Dude, goddamnit. This is a startup forum. People complain about shit being hard, and it's our job to make it simpler. Even if it's the temple itself.

I believe it's possible, and worth doing. Moreover, I have spent several years of my life trying to do just that in the space of network parsers and serializers!

All I was saying is that it's not easy or straightforward to achieve, and that the author's essay doesn't offer anything that helps us get there.

Yeah sure you can keep layering till you get 47 different layers (some in parallel).... aka

"X Windows"

Layering does come at a cost. Speed and various hacks that have to reach below the layer. Partly because of X Windows Linux still has one of the slower and fragmented graphics shell.

In my reading I certainly saw insight, I think you're not pulling your 'world view' back enough when reading it and looking too closely at the examples.

I think his point is not that "ls" needs less flags, it's that it needs no flags.

Or that libpng is low-level versus high-level, it's more that there shouldn't even be a concept of low-level.

And the other thing he seems to be saying is that backward compatibility should be abandoned to allow libraries to actually mature instead of just bloat.

It was certainly food for thought to me. And you even seem to agree in the paragraph of Let me tell you what it's like to design small and simple software...

"I think his point is not that "ls" needs less flags, it's that it needs no flags."

Then don't use the flags. The default bare 'ls <wildcard>' is sane 99% of the time for me, and I consider myself a junior power user.

same here. in fact i'd call it a power law--90% `ls`, 9% `ls -l`, .9% `ls -ltr`, .09% `ls -lhSr`, .009% `ls -d` .0009% `ls -i`, etc.

which suggests the default behavior, and arrangement of the flags, is very well designed.

(if i could change one thing, it'd be to make all the sorts ascending by default....)

>The author makes it sound like simple is easy. As if it's just a matter of saying no to complexity, like saying no to memcpy() whenever we have a memmove() that's good enough. This is not the case. Simple is not easy. On the contrary, simple is hard.

Simple by design might be hard, but "simplify in retrospect" is easy. It just needs you to be willing to sacrifice meaningless backwards compatibility and inflict some temporary pain (rewriting stuff dependent on it) for the benefit of tomorrow.

>So you think "ls" has too many command-line flags; how are you going to cut the number in half? Which ones do we keep and which ones get axed? How are you going to provide for all the same use cases that those 35 flags were added for?

You don't. Fuck ALL those use cases. Keep only those that are used a large percentage of the time by the large majority. For those that need the extra juice, let them use the old, convoluted version. You don't have to have "ls" be a be-all-end-all. Have both ls AND a power-version if anyone cares to maintain the second.

And sure as hell, don't keep competing or duplicated flags just for backwards compatibility.

> You don't. Fuck ALL those use cases. Keep only those that are used a large percentage of the time by the large majority.

Disagree completely. Because if you do that, the moment someone comes along and needs something that you're unwilling to provide and can't be layered on top of your work, they're going to make their own "ls" that overlaps a lot but satisfies their need. If we have 15 different "ls" implementations, we're back to more overall complexity.

Basically if you want to write code that lives at the bottom of the stack, you have to think of the most crazy hard-core shit that anyone might want to do with your code and how you can accommodate that.

Two fantastic examples of software that achieve this in spades: Lua and zlib. Both let you plug in your own memory allocator instead of malloc() if you really need to. Both let you perform your own I/O if you really need to.

Heartily agree on Lua. Embedding lua was one of the best library experiences I've ever had. The implementation and language is small and simple, and I think many people undervalue that. Compiling it was as simple as dragging the source files into Xcode (not an ideal long term solution, but quick and easy to compile to any target, and at 10k lines of code, reasonable compile times). As a comparison, I spent 3 days trying to compile Spidermonkey for ARM (with eventual success attributed mostly to finding a useful patch on github).

"crazy hard-core shit"

This is the problem. When basic understanding of a system (e.g. I/O) is viewed as "crazy hard-core shit" we are in trouble.

You do not even need ls in today's UNIX.

Globbing, printf, echo and stat can do it all.

And if necessary it's easy for the user to write her own custom utilities for displaying file or directory information from the stat struct as long as she is provided with good documentation for the standard C functions.

If you don't know how to do these things then you need to learn. And I say that as a dumb end user who can do things in UNIX that apparently most programmers of today cannot or will not because it's "too hard".

To me, learning all these high level scripting languages is crazy hard core shit. It would take me forever. They are large and complex.

Meanwhile I can do things the short, simple way. And learn about "crazy hard-core shit" like basic I/O at the same time.

People might use only 10% of features that your program provides but the problem is that different people use different 10% (it or similar was said about MS Word bloat).

Yes, by Joel Spolsky. Only it's not 100% accurate.

Some people DO use a different 10%, but the grand majority uses the same 10% (and occasionally has a need for something outside of that).

Users need to be able to get the functions they need. But the unix-way provides a mechanism for getting ls plus the functionality you want: have a shell macro run the minimalist ls and pipe it through a formatter.

Unfortunately, although unix scripting was good by the standard of its era, it's still not particularly good. Unix as we know it hints at great ideals that are achievable but lives up to them poorly. As a result, early packagers shoveled hacks into existing layers and we live with the result.

Evidence: arguments in commands like cat and ls; the crudeness of ps to represent the process table; inability for scripts to access unix systems calls; kitchen sink shell interpreters.

Why do we look to shell interpreters for command-line history? That should be done in a separate layer closer to the terminal. Had it been, we'd have two small unbloaty layers. Instead we have tcsh, bash and zsh.

You could have your ls formatting macros live in your personal terminal emulator and you'd never need to copy .bashrc hacks to each system. The formatter code might not even live on the remote system, it might be part of the terminal.

I suspect you could lay [racket scheme] + [a transform that converts python style whitespace code into parens code] over a unix that has /proc, and use that to create the layers you needed. With that in place, you could then start taking noise out of the unix tools and work towards a smaller distribution.

Plan9 is talked of as "unix done right" but I'm not convinced. Some things it does well, but it has lots of baggage. For example, you have to buy into their desktop to access the OS. How is forcing me to use a mouse and a particular tiling window manager a necessary part of a better unix experience?

I'd like to play with a plan9 fork where the only user interface assumption was a terminal, and where you could get a secure connection into that terminal. (Too inept to do it myself so far, but keep playing with idea of running it on linux, ssh'ing to linux as the jump box, and then telnetting into the plan9 instance. Still - what a hassle)