Programming Languages Are Simply Not Powerful Enough
ivanjovanovic.com
ivanjovanovic.com
I hear this a lot to explain the idea of abstraction to people, but the trouble with applying it to software is that picking really good abstractions is very, very very hard. Abstractions evolve over a long time under pressure of peer review and evolution of software systems. It took us ~2000 years to get a solid foundation for mathematical concepts and notation (we understood sum way before that, but you get the idea). We don't have 2000 years to pick really good abstractions for software systems.
I'm convinced that a programming language must be really good at allowing you to change and evolve your programs as quickly as possible as your understanding of abstractions improves. This is far more important than more powerful ways to describe abstractions IMO. Functions are fine. Just let me change things drastically without breaking too much or having to rewrite half the darned thing, and I'll be happy.
Personally I think that the ability to define your own language constructs is ultimately a lot less useful than things like rich libraries and robust tooling and I'm still waiting to see real-world proof of the advantages of programmable programming languages.
Most people are not willing to learn something fundamentally new. They might be content going from something they know to something very similar, say Java to Python, but they will resist anything truly novel and different. And the others? They're mostly the ones already advocating Haskell or Lisp or Forth or what have you!
Also, people have some really odd reservations when switching to a new technology. They are often not willing to take any visible steps back: the new might be better in a whole bunch of non-trivial ways, but if it's obviously worse on some easily identifiable property, people will avoid it. A new language might have a long list of advantages, but if it has an inefficient default string type, or slightly wonky syntax, or a sub-optimal packaging tool or any other superficial but obvious shortcoming, people aren't willing to make the switch.
Another problem is that there are different kinds of productivity. There are "dense" productivity changes: if you have to write your own JSON library or deal with a broken build system, you'll be spending a contiguous amount of time on it. You'll have to devote maybe a whole day or even a week to getting around this problem. There is no way to miss this. On the other hand, if a language improves productivity in a "sparse" sort of way--say you spend 20% less time writing code and 30% less on debugging and maintenance--you won't notice quite as easily. And yet, over any reasonable time using the language, you'll come out far ahead even if you have to sink in days working around problems and libraries.
A particular--and particularly important--example of this is in learning. As I said above, one of the main reasons people resist new technology is that they don't want to learn. They're too busy to spend a whole week or even month picking up something new. And yet, learning time is essentially a constant expense. Productivity gains, on the other hand, are at least linear in how much you program. If a language makes your code simpler, the gains can even be super-linear (that is, you get more benefits as you write bigger programs). These will dominate any learning time as soon as you actually start using the new technology widely. And yet, since the amount of time spent learning is obvious and the ambient productivity gains aren't, people put a larger than warranted cost on the former.
Coincidentally, this does not only apply to programming languages. I've seen exactly the same sort of behavior in adopting any kind of new, non-trivial technology: Emacs, Vim, Git, Linux...etc.
In short, don't trust popularity. This probably makes me sound elitist (and, to be fair, I probably am), but it's just like music or literature: the popular stuff is usually not particularly good and the good stuff is usually not particularly popular.
How does that work? Unless you are creating and supporting Git, git isn't going to work if you are the only person using it. Even then, there will be no github. And I don't know about you, but I won't use Vim if I lose my plugins(directly correlated to how many people are using vim)
Lisp is one of the oldest programming languages in existence, it's still taught in Universities and has been for over 40 years. Age and exposure and still not popular.
I'm not sure why you keep mentioning new technologies when the article is about old technologies and your examples (Emacs, Vim, Linux) are old technologies! The only example of a new technology, git, has seen an absolute rapid rise and completely changed the version control landscape in just a few years.
A new language is not going to have a long list of advantages -- it's going to a have different list of trade-offs. Because any language feature that is objectively good with no downside or trade-off has already been implemented in some popular language somewhere.
And if you can prove that you're doing something really different and really better but there's a strong learning curve people will learn it. Git is the perfect example.
Because any language feature that is objectively good with no downside or trade-off has already been implemented in some popular language somewhere.
But C# and now C++ and Java have all very recently and quite suddenly included lambdas and closures?
It took javascript, a hybrid language, to show language designers just how powerful those features can be. It took a practical application of the concepts in an almost OOP setting to allow people to understand just why they're so useful.
I think the entire time I've been on HN people have been saying everyone's just about to switch to functional programming. We've had a lot of advocacy for Haskell, Scala and F#, but no big switch. It's fairly obviously never going to happen at this point, but they still say it.
In the same time frame MVC has transformed web programming, with Django, Rails, Symfony & ASP.Net MVC all becoming a norm in web programming.
If functional programming really were that compelling, we'd have seen a similar switch by now. Instead what's happened is that all the main languages have adopted the best bits of functional programming and left the bits which make it hard to write large programs.
I'm not defending Lisp specifically (in fact I don't like it very much.) It's that the notion that a language can be "better" or "flawed" in an unqualified or objective way really irks me. The viewpoint that the only reason anyone would ever program anything is to create production software is, in my frank opinion, an intellectually stunted one.
This almost implies that there is no opportunity for anything new in programming language research, which I find laughably depressing, especially if you take "some popular language somewhere" to mean the popular industrial languages. It would be rather sad to think that Java (or C++, or C#, or Ruby, or Python, ...) represents the pinacle of programming language design, and we're stuck with it and its ilk from now on, forever, since there are no good language features left that are worth implementing.
Progress is ongongoing but it's evolutionary not revolutionary - and that is a good thing.
To quote John Carmack:
To the eternal chagrin of language designers, there are plenty of externalities that can overwhelm the benefits of a language, and game development has more than most fields.
How else do you explain the popularity of languages like Javascript and Objective-C? Javascript is an objectively terrible language, hacked together by Brendan Eich in an afternoon. Yet, it's a tremendously popular language, and tremendous amounts of money and time have gone into building good tooling for it. Same thing with Objective-C. It's a shitty, ad-hoc, cobbled-together patchwork of C and Smalltalk, yet it's tremendously popular now because that's what the iPad is programmed in.
I have much a bigger beef with the type system (or lack thereof), the array-and-hashes-as-one-datatype stupidity and the lack of a module system.
http://www.lysator.liu.se/c/bwk-on-pascal.html
Most awfully, "length of the array is part of the type of the array". This made some programming rather painful (strings and numerical programming, for example). Maybe other dialects of that time didn't have this problem, I don't know. I do know I found C to be a great relief.
Javascript is a good example of historical accident, on the other hand. It's got just enough extensibility that its flaws can be worked around. But its popularity is not due to its careful design.
Going from Pascal to C was like a breath of fresh air. Pascal is like a combination of everything I hate in all programming languages that I know and I'm glad that the world moved on. C is small, elegant and portable. It's a systems programming language that does what it's supposed to do.
> Javascript is an objectively terrible language
Our tastes are different and all proof to the contrary, as Javascript proved that it's an objectively good programming language, that has been burdened with legacy, incompatibilities due to the browser wars and more recently slow evolution due to its huge popularity and the stagnation of IExplorer. It has quirks and it isn't what it should have been, but considering the context it's the best outcome we could have had.
Name one other programming language that's (1) a true standard governed by a standards body with multiple platform-independent implementations and (2) has less problems than Javascript.
With Javascript your argument falls flat on its head too, because all of the competing technologies failed because of really good technical and political reasons ... lets not forget VBScript, Java Applets, Flash and Silverlight. Yes, technologically speaking Javascript was and is better than all of them.
I hate C with its lack of proper arrays handling and brain dead string manipulations that open the door to so many security exploits. Additionally the lack of modules/namespaces is a joke.
Mac/Turbo Pascal, Ada, Modula-2, Modula-3, Oberon(-2) are all examples of system programming languages done right.
That said, if you're the kind of person who doesn't understand pointers, there are numerous libraries out there that'll provide a variety of functions for working with arrays and strings more easily. C itself doesn't prevent you from using these alternate approach, but rather it enables them.
Combined with the fact that there's no warning when you've stepped outside the bounds of an array, I think makes it an extremely dangerous language to use for large scale programming.
Years back, a friend was maintaining a system collecting and processing scientific data, in C. Several papers have been published on this data over the course of many years. I remember the sinking feeling we both got - mixed with real fear - when he told me he had discovered an error where an array's bounds had been miscalculated. I don't even want to think about the possible ramifications of that...
For some application domains low level control is necessary of course, or for others the performance gain may make the tradeoff worth the cost.
In fact no application should be written on top of a language that doesn't do garbage-collection by default, unless you've got the resources of big companies. Any language that doesn't do garbage-collection by default is dangerous for large scale programming.
On array bounds checking, you can't do such bounds checking at compile time, so you have to rely on runtime checks and metadata. As my highschool professor kept telling us, C is a medium-level programming language. It's optimized for moving bits around in memory, while the code remains reasonably platform-independent. Adding array-bounds checking by default doesn't make sense for a language like C, because it stays in the way of moving bits in memory.
Any proper language for systems programming allows to turn off bounds checking if required-
This is the way it should be done, disabling the bounds checks on the few spots that require them being disabled, after profiling if they being on really affects the performance of the application.
I'm talking about things like its extremely broken equality operators, its extremely broken array handling (especially the behavior of the length property), semicolon insertion, its broken scoping, its lack of support for proper class-based OO, its lack of support for proper namespacing and modularity, its lack of proper typing, and so forth.
These are far worse, and in many cases much stupider, than any problems we see with C, C++, Java, C#, Python, Ruby, Erlang, Perl, Haskell, Scheme and whatever other mainstream or semi-mainstream language you want to choose.
PHP is perhaps the only widely-used language that exhibits the same kind of idiocy when it comes to many core features. But at least it has been moving in the right direction, with its much more sensible support for OO and namespaces, for instance. We keep hearing about how ECMAScript 6 will finally bring at least some sensibility, but this is still very much up in the air.
It's incorrect to say that JavaScript is an "objectively good programming language". It's objectively terrible, and all of the evidence proves this very conclusively.
> I'm talking about things like its extremely broken equality operators,
Don't use language features that are broken. Use === and !== instead. JSLint complains about this.
> its extremely broken array handling (especially the behavior of the length property),
You're going to have to elaborate here, I have no idea what you're talking about.
> semicolon insertion,
Put semicolons at the end of each line manually. Or don't use crazy multiline expressions. JSLint will flag this down for you. Other languages like C force you to do this, why the JavaScript hate?
> its broken scoping,
Never put things in the global scope. Always use var. Global-by-default is annoying, but very easy for static tools like JSLint to flag and nag about.
> its lack of support for proper class-based OO,
Javascript's objects are very minimal; they can easily emulate semantics of other OO languages. In my code I usually emulate the Python copy-superclass-on-class-creation semantics of subclassing.
> its lack of support for proper namespacing and modularity,
RequireJS has effectively solved this problem for me. As per above, you should never put things in global scope anyway.
> its lack of proper typing,
If you're looking for built-in static typing, you came to the wrong place. I think it's important to remember that JavaScript the language was designed to be very minimal (IIRC the entire initial language was created in a week); in that regard, it at the very least is not a good fit for static typing. I personally wish that it had stronger types, and can agree with you here that ideally this would be different.
> and so forth.
I guess I don't see your concerns here. I prefer writing JavaScript over C and Java; it allows me far more leeway for creating abstractions through closures and anonymous functions. Think of JavaScript's problem statement: create a simple minimal universal browser-based language. It's hard to imagine a better language in JavaScript's place (Scheme is the only realistic alternative I can think of). Here's a nightmare scenario for you: we could have ended up with Visual Basic instead.
JavaScript isn't perfect, but what language is?
While I agree that these issues are surmountable, I don't think you've rebutted the point that they are issues with the language itself.
That being said, no language is perfect and for being whipped up in a day, Javascript's pretty nice.
C and Pascal were developed almost in parallel.
Being the UNIX's systems language, it meant if you wanted to develop on UNIX C was the default option for most developers.
> How else do you explain the popularity of languages like Javascript and Objective-C?
There is no other option to target the browser, as JavaScript is the only language natively supported in browsers.
Objective-C is the only proper way to develop iOS applications, thus you need to use it if you really want to fully explore the platform.
All the languages you mentioned only became famous because developers were forced to use them commercially.
Back in the MS-DOS days I had zero need for C, Turbo Pascal was enough for everything (with a bit of Assembly).
But the commercial sucess of UNIX meant eventually I was forced into the realms of C. Luckily C++ was around the corner and provided an escape path for a saner language.
I still miss Pascal and the derivative language (Ada, Delphi, Modula-2, Modula-3, Oberon(-2), Active Oberon).
However, I also think that we are making progress to go in the abstract direction, by dint of increasingly having polyglot systems and DSLs that are part of the standard libraries(e.g. regex and SQL access are expected everywhere now). If this trend accelerates we could have a hockey stick effect and before we know it, find ourselves programming almost entirely with tiny languages.
...sometimes handicapping the smartest person, more or less on purpose, makes him better at working in a team (even a "virtual" team like the one encompassing library creators and library users). It's ugly but it works! Of course, the ideal would be to have a team of very smart people - but when you can't have that , the next best alternative is to use a language that artificially "dumbs down" the smartest guys to a level similar enough to that of the others to allow collaboration... sad, I know :|
I'm fairly certain there is vastly more code written in either FORTRAN or COBOL than Java and C++ combined. So I guess that makes FORTRAN and COBOL better than Java and C++ put together . . . ?
[1]: http://www.yosefk.com/blog/my-history-with-forth-stack-machi...
Edit: I agree a lot with the content though - For many complex problems, especially the kind where you can't just throw some standard software solution at it, it would be helpful to first develop a suitable DSL. From personal experience I'd say spend one third of your time budget into developing your domain specific toolset - even if your managers get nervous. If it works you will finish sooner than you expected and you have developed reusable tools and skills, if it doesn't you can still throw in a crunch and get it done. For me it has always worked though.
[1] It's a Japanese office. These people really like their cold bright neon. I think it has to with dark east asian eyes.
The colors aren't that beautiful but it's much less stressful for my eyes, and the highlight colors were spot on as well.
Thanks to emidln for informing that it's Solarized Dark, I've used the default Solarized but I've skipped the Dark one because of somewhat ugly colors.
This can cause problems in some cases; when the language is stepping that far back to allow you the freedom then it is up to your to enforce your own constraints to avoid your abstractions becoming...too abstract (maybe).
Also in my opinion there is not one abstraction form that will appeal to every type of programmer. For me I like the freedom to manipulate and define symbolic forms with Common Lisp for others it might be the expressive type system of the Haskell. I don't think you can truly say one way is better than the other.
I think those languages and a few others are giving the tools of abstraction that will not get in the way of you as the programmer.
Too abstract isn't necessarily the problem. When I've stepped too far away from standard OO structure, the problem typically becomes organization. In OO you know where just about everything SHOULD be, at least. If you're writing your own rules, it's harder to be certain where new variables get set, or what members you should find in what structures.
It takes a certain level of discipline to not shoot yourself in the foot. Or at least to clean things up before they get too messy.
I can only really speak from a Common Lisp point of view and the problem there that I was alluding to is the fact that you are so free to produce your own forms of abstractions whether it be mini DSL macros, dynamic dispatch method combinations, data driven programming using first class functions etc. that if you are not careful you can easily obfuscate the real intention rather than making it clearer.
Lua doesn't have macros (well, you can use a library to get an extremely powerful macro language if you want [1]), but the problem with going too far down the macro/DSL route (IMHO) is that the code becomes harder for people who didn't write it to understand. Or even if the code itself is easy to understand (by virtue of being a good DSL), it becomes harder to extend or modify, especially if the modification involves something not explicitly designed for by the original developer.
If you stick with the syntax of a known and well documented language, then the learning curve for new developers is easier. Though even then, the other issues you point out can obscure the code flow, so probably the only thing you can say is that complex code is complex. ;)
What you call "many (or even most) projects" is alien to a lot of programmers.
I beg your pardon, but I think you're forgetting that there are plenty of business apps that have very complex requirements due to legislation, a myriad of cross-cutting concerns, and audit requirements. Domain complexity is a very real thing, and it often necessitates architecture beyond cramming everything into this week's CRUD framework.
CRUD's apps have the stigma of be "patetic simple projects". Are not! Building erp-like software is dam damm hard. Making a search engine? Where clever math solve it? Where well understood ideas of scalability are clear? NOTHING against the complexity of fly at the speed of a business, deadlines measured not in months, but HOURS. Underfunded, not enough staff, working on legacy code, and that is only the surface.
Building a language to truly be a MASTER of CRUDs apps is something I truly wish for (the most close thing was Foxpro, IMHO).
Despite my love for python, and what I know for my use of .net, ruby, obj-c, I don't think is something well solve yet
I'm not sure that you can come up with a language that is targeted towards CRUD apps, because the entire concept of programming is CRD. The problem is building domain specific abstractions on top of a system that just flips bits and tells you what they say; and by "the problem" I mean "our job."
The weak link was the fragile database engine, but more important, the kill of the tool by MS.
The mix of a language, db engine, gui toolkit, and other things (similar, but far better than Acces) make it a tool truly made for this area of CRUD.
Plus, you could work in their REPL, and do commands as BROWSE and see the full data in a editable grid to use as you wish. I can't describe in words how functional the whole thing is. Is similar in the power of smalltalk, but instead of have a full reloadable code you have the full data at your hands.
Check http://books.google.com.co/books?id=k8fN2KMF1j4C&pg=PA54... for a idea of the tool.
A modern version of foxpro is something I wish for, but without the coupled GUI toolkit and a better syntax
to create a model of an entire personal computer system
that is extremely compact (under 20,000 lines of code) [3]
If you want to know one practical language today that embodies this philosophy and is yet main stream today, its Perl.
Perl 6 takes it further. Infact Perl 6 by design is supposed to be such. Here is comment Larry Wall himself(http://www.perlmonks.org/?node_id=836573). Thought the whole thread was a troll thread. The first time I encountered this was like 6 months back while searching something related to Perl 6 on the internet. And it stuck me that how amazing this concept was.
I then went back and watched some of his keynotes. I will recommend everybody to have a look at it.
Perl 6 is definitely going to be a big game changer in this place. And unlike Lisp its a very practical language for your everyday programmer.
See http://scheme-reports.org/2009/working-group-1-charter.html
I'm unabashedly in favor of the latter.
Ultimately, it's easier to reason about some domain in that domain. I would love my code for any particular project to look very similar to what I would write on a whiteboard when solving the problem. If I'm solving something inherently mathematical, I want the code to resemble math notation. If I'm writing some parsing code, I want it to resemble a grammar. If I'm writing an interpreter, I want it to resemble the operational semantics of the language. This fits into more "practical" domains as well: if I'm writing a web app, I want the routing to look like, well, routing and the database code to look as close to the database structure as possible.
Any details in the implementation rather than the domain should be separated from the domain logic. If you're trying to make the domain logic mirror the domain itself, this is essentially inevitable. This makes your code more general and easier to debug. You can redirect your domain logic to a different backend, and it's much easier to figure out whether you have a problem in the implementation or the logic.
More generally, such a strict separation of concerns is extremely important. The difficulty of a problem increases exponentially with its complexity: all the parts affect each other. If you manage to neatly split a problem like this in two, each sub-problem will be less than half as difficult as the main problem. So splitting away the high-level logic from the IO and the memory management and the other implementation details is inherently valuable. In fact, this is valuable even if the sum of both sub-problems is a bit greater than the original problem.
The malleable language approach gives you a very natural way to split your problems up: one part is implementing the DSL, modifying the host language or writing a library--creating abstractions. The other part is actually using these abstractions. I find this sort of split to not only be very useful but also very natural; it mirrors how I usually think about things.
One very interesting recent development taking this idea to the extreme is OMeta. There your program is essentially a tower of DSLs, with the base language just being a simple way to specify the DSLs. This idea really is extreme: for example, the code for implementing TCP is actually the ASCII art diagrams describing it! [1]
Of course, all the usual suspects like Lisp and Haskell are also worth looking at. Embedded DSLs in these langauges can go quite far, and they have some of their own advantages to consider.
[1]: http://www.moserware.com/2008/04/towards-moores-law-software...
I'd say it is the same with all good computer languages. You can express whatever you want in python and lisp, but following different grammar.
If Functional Programming was mainstream, everyone here would be raving about OOP and its benefits.
Ultimately what you want is to borrow the good ideas of different paradigms to create elegant and practical code.
To put it another way, OOP is a pretty nice hammer, but so many people have been using it to bash screws in (including ridiculous cases like concrete!) that others have reacted by using screwdrivers for everything (including nails). Pick the right tool for the job.
Ah yes, the ol "it's all relative" argument.
Not NEARLY as cool of an implementation as implementing TCP using ASCII art, but there's a command-line option parser that parses the help text to determine the command line options. [1] For example:
-- scale.lua
require 'lapp'
local args = lapp [[
Does some calculations
-o,--offset (default 0.0) Offset to add to scaled number
-s,--scale (number) Scaling factor
<number> (number ) Number to be scaled
]]
print(args.offset + args.scale * args.number)
lapp has been around for four years; docopt is (looking at git commits) less than a year old. It strikes me as a natural way to do things in Lua, which is why (I suspect) it was done in Lua earlier.
[1] http://www.inf.puc-rio.br/~roberto/lpeg/ [3]
[2] http://kernel.org/pub/software/scm/git/docs/git-fast-import....
[3] I actually use the re module [4] as I find the results more readable, and only use the LPeg module when I need something that the re module can't provide.
One thing that positively impressed me were coroutines though. You can do some very interesting DSL things with it.
Using a programming language is like speaking in common tongue to every other programmer who knows that language. And most languages are so similar that most programmers can communicate even between languages.
Having code that looks the same, regardless of what it's actually doing, allows everybody to be able to read it.
As much as I love DSLs, I Couldn't agree more.
>The problem with writing a DSL for every problem (and every programmer writing different DSLs) is that you end up with all these different languages
When writing in a DSL of my design, it is amazing. It feels like the program models exactly how I am thinking about it.
When modifying/maintaining a DSL made by someone other than me, often this can be horrifying, it feels as though this models my thoughts so poorly (and it does, it was designed to model their thoughts).
We are doing web-testing in clojure, and in past year our little framework evolved into a DSL of a sorts.
On one hand it is great, we a have consturct like (with-client [name connection & body]), that behind the scenes sets up a virtual machine, registers it as a client of our web-service and destroys it after commands in body are done. I can't imagine we would be doing setup and teardown by hand.
On the other hand it took me a month to get into it (two weeks learning the language, two weeks messing around with the framework) ... and there are still lots of things only our lead-tester knows how to fix, after our devs decide to change something we were relying on for testing.
I am curious, did you already know the JVM? When I learned Clojure, I was also learning the JVM, and for me the JVM was much harder than Clojure. So I would say it took me 6 months to learn Clojure, but much of that time was the time it took to learn about the JVM and all the related Java weirdness.
Either way, 2 weeks is very impressive.
What really helped me that, we had "programing paradigm" course in college, where I learned haskell for 3 weeks, what was enough that anonymous functions, map, reduce and using lists for everything didn't feel new.
"and here is the 1M LOC codebase, which was carefully crafted to avoid using any of the complicated frameworks; all you gotta know is the C standard library".
Plus one outdated sequence diagram.
//! \brief Transport diagnostic wrapper
class TransportDiagnosticWrapper {
How many people have learned what MVC means from Rails? Or the term "routing" in the context of a webapp?
One caveat I'll give: the DSL must be well-conceived. Writing a DSL that is dissimilar to both the underlying language and the domain vernacular is obviously horrible.
To add to what you wrote, you have to also consider that all developers in one team have to agree on a common DSL beforehand, which is an investment of time in itself, and the fact that if there is an unusually big change in requirements mid-development (which, let's be honest, is not a rare event), you might have to go back and change your "pyramid of DSLs" to accommodate it, maintain compatibility and update all your colleagues on the new languages.
With a DSL, all that goes out the window. The syntax of the language is potentially unknown to you. The semantics are an even deeper mystery. You're in terra incognito, trying to understand the mental process of the developer of the DSL before you can even understand what the code written in the DSL intends to do.
The primary goal of programming languages is not to describe how to perform a complex task to a computer. It is to describe how to perform a complex task to other humans in a form that computer can also execute.
When I talk to someone about what happens in a system it seems like a DSL flows from that conversation.
"It is to describe how to perform a complex task to other humans in a form that computer can also execute."
I think a DSL does that much better than a set of API calls, particularly when you involve non-developers. That is the point of a DSL.
And statements like (paraphrasing the quote in the article)
> You shouldn't build an app in Forth. You should build a language to model your problem domain in forth and use it to build the app.
They just doesn't make sense to me. Isn't that the very definition of how to program?
In a Lisp, you can use macros to accomplish things that go way beyond defining a domain-specific library / API. A typical library is restricted to the capabilities of the language, but a DSL in Lisp can extend the syntax itself.
Want to alter the reader so you have a way to express normal code as remote computations? Go for it. Feel like adding new math operators to use matrices instead of scalars? Knock yourself out.
Common Lisp itself did exactly this when object-oriented programming came along; using nothing but macros and closures, the Common Lisp Object System (CLOS) was built.
But designing a language is hard, including a DSL. Even for experts. e.g. Codd had two attempts at a language for relational algebra, but "mere mortals" couldn't use them. Mathematicians create new notations at the drop of a hat, giving their own definitions to common terms willy-nilly, but this can be an impediment to collaboration.
OTOH, there's an advantage of having many different attempts at a language, inspiring and cross-fertilising each other, til eventually settling on a near-optimal design (if everyone designs their own DSL, this doesn't happen). In practice, you get one group of people experimenting like crazy - academics, visionaries, hackers - and a huge majority that waits to see if something emerges that they can standardise on. Meanwhile, experiments continue. (This is basically the "technology adoption lifecycle".)
The ruby community develops new DSLs very quickly, in the form of frameworks. I get the impression the rapidity of change is challenging, but the improvements are worth it.
* Heterogeneity of object abstractions: in order of complexity, you use namedtuples, classes, and metaclasses. In Lisp you'd use the same kind of abstraction across complexity levels.
* Similarly, there are lambdas, plain functions, and bound methods, all for pretty much the same thing.
* Which do you use when? How hard is it to change? http://lukeplant.me.uk/blog/posts/djangos-cbvs-were-a-mistak...
* The way so many configuration variables in Django are strings, that just "coincidentally" point to a class, module, or function.
I've been using Python for 10 years, and Django since pre-1.0. My current startup is built on it. The parts start to show.
Take SQL, a nice DSL for working with relational databases. Yet people ended up writing ORMs (quite ugly creatures imho - something between an elevator connecting two layers, and as in real life elevators are a mess when too many people want to use them at the same time, and an intermediate pseudo-level, but they are useful) because the pain of climbing up and down the ladder connecting the "relational level" with the "OO layer". "Language layers" or DSLs are a powerful but dangerous abstraction because when they start using them people can't stop, they just make more and more layers like crazy! (The only other abstraction so powerful and so dangerous I can think of are objects - it's amazing how many got them completely wrong so many times in languages like C++ or Java... besides Ruby and Smalltalk I can't think of anything that got them "mostly right"... and Ruby seems to be the most DSL-able language in use too, though this is why I'd take Python over it any day - smart programmers tend to make everything like a freakin DSL in Ruby and and your mind ends up running up and down the ladders until you go insane when reading code).
I've recently been digging hard-core into C++ after making such disparaging comments for years and have been rather surprised to see that it's not as terrible as I once thought -- It's actually quite good.
I agree however with your point about dangerous abstractions. DSLs when implemented with careful, thoughtful restraint are rather powerful. SQL being a really good example. There's nothing stopping anyone from writing their database queries by directly sifting through and filtering datums from the domain using the primitive functions and operations of their programming language of choice... but it's clear that SQL is much more powerful and lets us express such problems in a succinct way.
But just like programming languages, once introduced, a DSL is likely to be misunderstood and abused. The problem I believe lies not with the technique but with the users. A double-edged sword is a fine tool but will likely maim an inexperienced user. Which will probably lead them to writing DoubleEdgeSwordWrapperMachineGun classes and then someone else to write a DSL on top of that...
A sane relational database API would let you compose and extend arbitrary queries, which has profound implications for the way you can write your program and abstract away repeated patterns.
[Arel](https://github.com/nkallen/arel) has the right idea in terms of API, but a poor implementation (lots of things are not truly composable where they should be). [ScalaQuery](http://www.scalaquery.org/) purports to have the right idea, but I haven't tried it, and it's very Scala-centric.
You can bend your statement to your language, or your language to your statement. All essays look the same, regardless of the statement being made. English, Chinese, Persian.
Object oriented programming was a big step because it's so heavily modelled on hierarchies - since found to be one of the most important principles in human reasoning.
I'm no expert in the area, but I'd love to hear ideas on other human reasoning principles or models that a language could be influenced by (particularly if you know about neuroscience or AI).
Yes, abstraction is a wonderful, useful thing that lets us think and do all sorts of wonderful, useful things - our high level programming languages and networking protocols are abstractions over lower level programming languages and networking protocols (repeat a few times) which are abstractions over the physical hardware they run on, and that's great. But abstractions are never perfect, and what you don't know about the lower levels of the stack will bite you, which is a good argument for using as few and as thin abstractions as you can to get your thing done (reasonably ... I'm not about to write this comment in binary ASCII, say).
edit: word use
Most languages I know allows me to form some type of class/data structures for my need. What kind of problems are OP referring to? some kind of extensive math/scientific problem?
'It is known since the early ages of programming, most elegant programs are built as layers of "languages" or "dictionaries". These are represented by abstractions in a program in order to facilitate expressing the ideas in a way that is more related to the domain.'
Overall, I like to draw a line between canonical/ontological representations of models and representations of models designed for calculation. The ontological representation often times exists in a relational database or in an explicit OWL/RDF ontology, but may also exist only ideally, where the storage and calculation representations of the model are each customized for their particular use, especially at large scale.
the following concept, I almost agree with...
'it is our job to express the abstractions of the domain for the best economy of the expression'
As programmers (vs ontologists) I would say that our job is first to express the abstractions such that they enable efficient, maintainable computation that hopefully is expressing those abstractions economically. I feel like programming languages don't lack power, they just make it quite complicated to express complex abstractions.
If you can encapsulate an undefined idea into one stream of commands, can't you just create a function to do it? What does that say about programming languages? Maybe I missed the point.
For example for a web-dev in Java you'd use a MVC framework.
In Clojure you get a query DSL to do your model, templating DSL to create your view and routing DSL to specify your controler.