Comment by kstenerud | Hacker News Reader

Comment by kstenerud | Hacker News Reader

"These languages are very powerful, but they don’t scale. So even when you get ten times the productivity, the mostly tribal complexity will not scale beyond your “pizza-sized team” and therefore you now have a hard upper limit on your total software development capacity."

Ultimately, the problem boils down to the languages giving too much freedom. More conventional languages like Java can scale to a massive degree because they standardize so many things, which means that you can now share components which amplify your work. Lisp et al are cool toys and small team tools, but you're not going to see any large scale db libraries and parsers and concurrency libraries and web servers and cachers and HA systems that can interoperate with each other without a ton of shoehorning.

When every foreign chunk of code requires a ton of ramp up time just to figure out what it's doing and how you could even integrate it, or requires changing your IDE and workflow in ways incompatible with what you have already, you lose more than you gain.

Worse is better. Worse gives you interoperable, reusable, amplifiable components, in a common framework everyone understands.

> the languages giving too much freedom

This is why the D programming language doesn't have macros, neither text nor AST. It's been a controversial decision, and I've had trouble convincing people of the reasons why. Others explain it as "Walter doesn't like macros."

The reason is that macros enable one to invent one's own language withing the language, and that language is incomprehensible to others. It happens with every language that supports macros and macro-like constructs. (It happens with C++ "expression templates" and operator overloading.)

People swear up and down that they'd never use macros that way, they'd use them responsibly, and the incomprehensible mess they create is indeed responsible.

For a hilarious example, an engineer I know recounted a story at Big Corp where there was an aged piece of software written in assembler. The programmer for it had long since moved on. The code had some problems and needed updating. Unfortunately, the programmer had invented his own language using the macro system, and nobody wanted to touch it.

My engineer friend said he could do it, and had the code fixed in a couple hours. Amazed, the manager asked how he'd figured out those macros. He replied "I didn't. I assembled the code, ran it through a disassembler, converted that back to source code, then fixed it."

Metaprogramming is amazing though and in languages that allow it like Ruby make amazing DSLs.

The beauty of Ruby is that you only need one metaprogramming call to make a DSL in Ruby. Define all your custom methods in a DSL class, then use instance_exec to run the code using the DSL in the context of an instance of that class. The instance can use instance variables to keep track of any state that the DSL requires. Super-elegant.

For C gcc -E helps a lot for macros. Does C++ have a tool that expands all templates?

gcc has options for intercepting various intermediate representations of a translation unit; look into that.

I just found this in the man page (for a fairly dated version of gcc):

       -fdump-translation-unit (C++ only)
       -fdump-translation-unit-options (C++ only)
           Dump a representation of the tree structure for the entire
           translation unit to a file.  The file name is made by appending .tu
           to the source file name, and the file is created in the same
           directory as the output file.  If the -options form is used,
           options controls the details of the dump as described for the
           -fdump-tree options.

If this were Lisp, we wouldn't be scared of what "representation of the tree structure" will look like.

It doesn't say anything about whether templates are expanded/instantiated.

D does have macros, it just has bad macros, which prevents people from going overboard with them.

CTFE returning strings to be mixed in is 90% of the way to a proper macro system. (It just needs quasiquoting, really.) The only thing keeping it from taking off is how ugly it is to use, keeping it relegated to hidden tooling code. And of course there's UDAs, mixin templates having full access to the lexical space they're mixed into, `q{}`... really, D has all the pieces of a fine macro system, you're just refusing to take the final step and make them consistent and easy to use.

It kind of reminds me of Path of Exile and their refusal to implement proper player shops. It doesn't mean there's no item trading, it just means there's only bad and annoying item trading. I guess I don't believe in gimping tools, especially when you're already willing to take them most of the way.

None of these enable ordinary D code to not be ordinary D code, which is the crucial difference.

Similarly, operator overloading is deliberately constructed in such a way as to allow new arithmetic types to be smoothly designed, but to make it clumsy to do things like C++ IOstreams.

They do enable expanding the definition of what is ordinary D code. With UDAs, you can write declarative data structures that are more UDA semantics than D semantics. Macros do the same thing, if with more range.

No experience with D, but using Go a lot, and it is the same thing there with lack of metaprogramming.

Turns out everyone uses reflection instead to get anything done.

Not sure how much of a success that is.

D fully embraces metaprogramming which is a very powerful part of D. But it stops short of being able to redefined the language, which is what macros do.

Do you consider e.g. BER MetaOCaml approach as "being able to redefine the language"?

I don't know enough about OCaml to have a non-gibberish reply.

The lack of generics kills any joy I feel for go.

The problem with macros and dsls is that there are many good programmers in the world but very few good language designers. Plenty of people can design a language for themselves, which probably explains the productivity gains of lisp for a one person project. But there are very few who can design a language for others.

And the problem of programmers not being good language designers is that the design of APIs consisting of nothing but functions and data types is also language design. Oops!

Unfortunately, not having powerful macros that can redefine the language hasn't made D popular, so it doesn't really factor into the line of argumentation here.

In any case, this whole debate, starting with the submitted blog and on down from there, is a lot of nonsense that completely ignores the fact that humanity has produced thousands and thousands of programming languages, only a handful of which were ever popular at any given moment. The numbers are stacked against any language, regardless of its peculiarities and merits.

What is popular in this area is a combination of luck, promotion by some big company, or bundling with an operating system or application platform.

Well, due to these decisions D seems to have become a (quite unpopular) "middle ground" / "centrist" language:

- people who want something more dead-simple / no-magic go for Go

- people who want the full combo of "max performance" + "performance-wise-free abstraction" + "extra safety" go for Rust

And conservatives stick to C++. Hence D didn't get very popular. I think it's a programmer psychology issue: people just don't like "middle ground languages" they want to clearly be in one extreme and accept its tradeoffs.

(There's Swift as a counter-example currently growing outside of iOS-dev towards ML, but I guess it's more like it being "like Go but with operator overloading and some macro-like infra features that ML people want & need"...)

I think the main driver for Swift is that it's a modern replacement for Objective-C, which feels very dated at times.

Well, ignoring Apple s ecosystem uniqueness, being an Obj C successor is kinda the same as a C++ successor

> More conventional languages like Java can scale to a massive degree because they standardize so many things,

But didn't Java go through the whole Factory/Design Pattern/Deep Inheritance nesting that created a large amount of complexity on big projects? Was that really better than creating a DSL that a newcomer to the team had to learn?

Maybe the reason is that Java was just more popular and more familiar to a majority of programmers. Worse is better often because worse is more popular for historical reasons.

There's nothing stopping you from using lisp as a transpiler to generate the java, or different interop glue for the lowest common factor code everyone will surely understand.

The magic was always that it was always an AST all the way down, and you can create and experiment with your own personal semantics and vocabulary, so long as the language is expressive enough to define them in the domain specific way you are doing development. For example a language that is aware of what libraries and databases you're tying together and abstracts the way they interconnect, can compose multiple libraries together without the kludgy plumbing, or autogenerates boilerplate along the way that's required by your build and tests. As long as you generate readable clean Java or whatever, your coworkers won't care how you got there

Where would you rank a language like Kotlin in the JVM community or F# in the .Net?.

Both allow the massive freedom of something like a Lisp but you get to leverage the massive existing codebases in both environments.

Reason I ask is because I'm just starting to learn Kotlin (or will when the book turns up from Amazon) and I'm curious.

Are you comfortable in Java? If so you won't need any book just the documentation and stack overflow will give you all you need.

Reasonably comfortable in Java though not used it much for the last few years, much more comfortable in C#.

I decided to go the book route because I've found that when learning a new language the book route works better for me since it gives a structured walkthrough of the language rather than a piecemeal one and it's more time efficient.

YMMV but for me you still can't beat a good programming language book.

Gotcha. Books never worked for me for some reason, only by diving right in to building something can I learn.

Best of luck to you! I've never been more productive than I am with Kotlin.

>Lisp et al are cool toys and small team tools, but you're not going to see any large scale db libraries and parsers and concurrency libraries and web servers

We already have many of those for Common Lisp.

Reply on news.ycombinator.com