We need less powerful languages
lukeplant.me.uk
lukeplant.me.uk
Sometimes we also need languages for exploratory work...
But anyway:
> In my years of software development, I've found that clients and users often ask for “free text” fields. A free text field is maximally powerfully as far as the end user is concerned — they can put whatever they like in. In this sense, this is the “most useful” field — you can use it for anything.
> But precisely because of this, it is also the least useful, because it is the least structured. (...) The longer I do software development involving databases, the more I want to tightly constrain everything as much as possible.
And that's precisely why they want that free-form text field. That's why people still use Excel instead of whatever database solution their IT department bought/prepared - because it doesn't limit them to particular poorly understood interpretation of their workflow that was outdated last week. Real world changes, requirements of the job change, and there's nothing more annoying than be stopped dead in tracks because some smartass from IT thought that this particular field should always be a number...
Use text fields unless you're absolutely, positively sure the constraints you want to impose are valid and will never change.
EmbeddedNetworkApplianceInc needs a database to manage their mission-critical data path? That IPv4 mask length field is going to be INTEGER NOT NULL CHECK (masklen BETWEEN 0 AND 32) and they can ALTER TABLE when they need to support IPv6 and the rest of their software is ready to support a masklen of 128 without crashing. There's no reason to let extraneous half-baked crap get into that field.
In the past there was essentially no ordered data-keeping. For several years certain bits of info had been entered into a spreadsheet, like name, title of art work, date created, medium, email addresses and so on.
Data in that spreadsheet was a total mess. Most fields were "free-form" text, with no consistency at all in format of dates, email, URLs, crucial fields left empty, data put in wrong fields, etc. It was a lot of work to clean it up and quite clearly showed what motivates programmers to be "controlling".
To be sure, excessive rigidity can be problematic, but it's hard to make clear to users the importance of maintaining data integrity. Frequently enough, users complain about constraints despite effort to demonstrate their necessity and benefits.
Of course constraints evolve, which is often the challenging component of creating and growing a system. If there's good reason to modify things, because what we conceived of as a number turns out otherwise, then we must indeed find a way to accommodate that reality. OTOH if it really is a number and the user wants it different, we leave it as is and once again patiently try to show the user our reasons.
Finding the right balance between user "freedom" and necessary limitations on data input is hard to achieve. I suppose the right approach has something to do with avoiding the illusion that data system development is actually ever done.
So it's understandable that users often do an end-run around what they choose.
So yes, the end-runs could be understandable. If the developer understands the user issues, and takes the trouble to mitigate the situation, it's a lot less likely end-runs would be attempted.
Any case, the smart developer knows there's a message intrinsic to end-runs, something isn't working and the developer's part of teamwork to see what can be done about it.
I wish. Point me to such a company. Quite often it's the management that's picking, which ensures that neither users nor developers are happy. A developer would at least pick something that works and makes some sense.
I think the reason why it often isn't done like that is twofold. First, there are many incompetent, sloppy developers who just don't care. Second, users are unwilling to pay for the time it takes a competent, unsloppy developer to do the work properly.
For instance, it is not impossible to design an address entry form that accepts unusual but correct addresses and at the same time captures as much structure as possible. Not impossible, but surprisingly difficult if you think about it.
You would think that it's economic rationality that leads to sloppy design and hence to the need for free form text fields. Maybe it just doesn't make sense to design everything with great care? But I don't think that is true. 90% of the time I needed to phone in to a support line was about issues that a slightly better website could have provided easily. Bad software costs them hugely in terms of support and user satisfaction.
It's plain to see that the good companies do get their data entry fields right and bad ones don't. I bet you could even predict share prices of companies based on the quality of their most trivial data entry forms.
I'm an American, but I live in the UK: My home address is in the UK, but I live in a flat and my neighbours are mail stealing cunts so I use a scan+email service for my mailing address which is based in the US.
Similarly, my land line is a UK number, but my mobile is a US number, and when you refuse to let me type in a `+` sign I spend a lot of energy guessing whether to give you a number in NANP or in international formats.
The thing is, capturing information is just that: capture. By doing the traditional programmer thing of validating the field on input, you're pissing me off as a potential consumer. I guarantee that I know more about where I live than you do, so I'm more likely to abort my transaction if you tell me that my address is invalid. That means your non-profit arts association simply doesn't get my donation.
Thing is, I actually accept that my situation is exceptional and not the rule, so I think this is really about programmers being unable to deal with exceptions; treating them as nothing more than dynamic escapes or nonlocal goto, like there's only a choice of more complexity, or more rigidity.
This is nonsense.
Simply capture whatever the user types. That means all input fields are plain text or blobs or whatever. You can try to validate it into your business model when you have some business need: like mailings, or shipping, and then attempt to extract and validate the specific fields when applying the mapping. If you have an array of failures, you can allow the user to review at that point.
I do this with two tables: An input table, and a data table. The input table has forward pointers to the data that is extracted, and the data table has backwards pointers to the inputs. The data tables might be used for business logic like mailings, order processing, login management, or shipping things.
Shipping is a particularly good example: I want to maintain four shipping providers since they offer different rates. This allows me to offer "free shipping" by simply selecting the cheapest provider and pushing that cost into the product. To do this I need to know their shipping zip code for the US, or the shipping country for international that's it. I don't need anything else, and three patterns (/(\d{5})(?:-\d{4})/, /\b([A-Z]{2}$)/m, and maybe a list of common countries) should be enough to extract from most orders. Anything else I can punt to my fulfilment center who can either call the potential customer, delete as spam, or manually extract.
Or maybe I just ship everything UPS: I send it off to label making, and the 0.003% that fail I have to hand-check anyway (after all, are we verifying the city names as well?)
What am I doing with these URLs? Am I visiting them? Am I verifying someone has placed some widget on there? Or am I putting a link next to their name on a bulletin board? Verification means different things depending on the use case.
Missing that crucial email address field? Or maybe there's an extra space on it? What exactly am I doing emailing them? What if the email bounces? What if it gets marked as spam? Verification of an email address has less to do with the characters in it than it has to do with the use-case: If this is for an account recovery, I want to know you can email me and will work with your system to do that.
This approach also means I don't need to "edit" things either, because edits are simply new inputs. Logging is free. Users are happy.
The point is it's not a balance; avoiding the illusion is easier than you think and the hardest parts of the problem of data validity are problems you have to solve anyway.
My home address is in the UK, but I live in a flat and my neighbours are mail stealing...
In my city, police will not respond even if there is theft in progress and they have idle units at Starbucks next door, claiming that there is no state law against mail theft -- it's up to the USPS to deal with it.
I have a PO Box for everything but junk mail. The much-maligned USPS has a really nice feature nowadays: you can sign a (free) agreement allowing them to accept packages on your behalf from other carriers... so I have FedEx, UPS, etc. all going to my PO Box, using the street-address format for the Post Office proper. It's much less expensive than private services like the UPS Store and such.
Your neighbors are probably more focused on stealing your newspaper. Or spouse.
I'm not sure I understood your problem, but the '+' sign can be replaced by '00' (two zeroes) when you cant to make international calls.
If there's an entry mistake and database fields contain the wrong info, it's not a tragedy, we'll find out sooner or later and it can be corrected.
Of course, date entries by necessity can't be as "free-form". For one thing, browsers can be picky about their preferred format for type="date" inputs, accepting "/" separator, but not "-". Requiring a year to be 4 digits between 1980 and current year doesn't seem too onerous.
These might seem obvious, but complaints about such constraints still come up. It just needs to be clear to users that certain data needs be entered in specific format for good reasons. I agree with your comments about street addresses, phone numbers, or personal names which need to be unstructured input (but handled securely). .
I thought by now this stuff would be appreciated as pretty basic to the developer's craft, but apparently it still isn't widely enough taught or known.
But if you start loose and need to tighten it you may be in for real pain, because you may be collecting crap data for a long time. I've worked in places where we axed features because the data cleanup to make it feasible would just take too long.
Ideally, languages would be scalable so that you can easily flip the "more power" switch without rewriting everything. One way of achieving a scalable language is to consciously design it as a subset or a superset of an existing, widely used language, as with a safe regex parser vs a full regex parser. Another way of achieving scalability is a language that supports the more powerful features, but with a clear barrier that you have to consciously cross, as with reflection in Java (as opposed to making everything dynamically typed).
Absent a scalable language, erring on the side of being more restrictive runs the risk of a lot of rework later on.
I'd like to dispute that claim. Specialised apps have eaten a large market share of what were once was Excel sheets. Same thing with Craigslist: about all of its individual categories have been "disrupted" by startups. WordPress replaced a large chunk of what once were hand crafted HTML websites or homemade CMS. Heroku ate a large chunk of the dedicated/VPS/shared hosting market.
Of course people we will always need generic apps and languages as long as they haven't found the specific solution to their problem.
Consider the discussion of terseness. What is the best way to explain something in Englush? Long winded or terse? Wrong axis. Explain things clearly. Expressing a concept perfectly with a small number of words the listener doesn't understand is useless. Beating a dead horse is useless.
I have a huge problem with C because the same characters and keywords are reused in multiple different similar but different ways and it adds cognitive load to things that should be simple (eg declarations) -- leaving me less able to devote my limited mental faculties to the problem I am actually trying to solve. (Not singling out C, just picking a well known example.)
Languages that give you lots of redundant ways of doing things -- especially if some ways are better than others but not obviously so -- aren't more powerful, they simply harder to understand. (This is not the same as arguing for newspeak — words that are similar but not the same are useful.)
What we want is clarity. Clarity isn't simple or easy but it is exactly what we want in programming languages.
Until some one invents the sufficiently oracular compiler power often prevents various effort saving/performance boosting tricks, which is what 2/3 of the article is giving evidence of. Also: formally verifiable code and static analysis.
There were also hardware kits like DDD that did correct by construction hardware from LISP specs. So, power != hard to verify. One should subset, though, for best results.
... So giving up power produces better results?
Sometimes yeah, it's easier to prove things about simpler systems.
Complexity, ease of modelling, and state explosion are the real issues in verification.
You can permit the dynamic stuff as an alternative (as Java more or less does.) But if people actually use it, it screws up the tools' ability to introspect into your code.
As a result, less-dynamic formats like HTML and CSS get better tooling, as a rule, because the tools have an easier job to do. And highly-dynamic languages like JavaScript and Ruby have terrible tools support, because they make the tools' lives quite difficult because of their language and library design.
https://news.ycombinator.com/item?id=10572044
(see second half)
The dialect of lisp that ACL2 verifies is not very powerful at all.
Part of my recurring them that you can have both worlds.
The idea that ACL2 verifies a powerful language is outright false. ACL2's verified language is stripped down (no first-class functions, nontermination, exceptions, etc.) because added power makes it too hard to verify.
That's not always the case. Take GUIs for example. In pre-C++/Java times the user interface declaration was often data. Now it is code. Whatever data is still necessary to function, such as button texts, are now object attributes. That code can then grow functionality and that's when you get non responsive UIs and hard to debug callback spaghetti. So things are done differently in less expressive languages, but it's hard to argue it's less clear in the general sense.
Another example is "application servers" and the rise of enterprise programming where object factories instantiate other objects in order to keep dependencies unidirectional. In a more simple language none of that would exist, and that particular problem would not be there to solve. But would it mean more complex, and perhaps unique, solutions elsewhere in the software? I'm not sure. Git won out over the more engineered and more complex alternatives for example.
His conclusion is, bluntly, Lisp, since you can reprogram that language arbitrarily and it won't get in your way.
It is also excellent for implementing DSLs, which can be as powerful as you want. For example, Viaweb uses (still!) a Lisp-based DSL to provide the non-Turing-complete customization language for their customers, and I suspect that whole thing was a weekend hack.
(Edit: it probably is Turing complete though.)
The ease of creating such restricted DSLs comes from Lisp's power. At some point you'll need a general purpose language, if only to implement your other specific languages. If your team can be competitively productive with a remarkably powerful language, Graham says you should use it.
But, it worked, didn't it? Grow, scale, cash out, dump code on non-founders, become startup messiah.
What is his track record with startup(s)? Do we just count N=1? Or do we include what YCombinator has accomplished?
Now let’s look at what Yahoo has accomplished as a BigCo.
Under the circumstances, would you bet that Paul was right about Lisp, Yahoo about C++, neither, or both?
I’d go with Paul being right about Lisp, and Yahoo, well... I’d call that N=1 at best. Clearly they started from a “We already have hordes of C++ programmers” point of view, and a “We apply hordes or programmers to every problem” point of view.
It might be the only point of view for them. It could very well be that there were hundreds of Lisp programmers available, but none might have wanted to work for Yahoo under any circumstances. Your mileage may well differ, based on the overlap in the venn diagram of language, culture, and problem set.
Perhaps great programmers are great in any language, and a company employing great programmer will have great results in any language, it’s the programmers that are the competitive advantage for a startup.
But although they’d be great in any language, they have taste, and prefer to work in some languages regardless of whether that is a competitive advantage or not. And therefore, they prefer jobs and startups with languages that match their taste.
And taking that a step forward, these mythical great programmers might prefer working with other people they perceive have taste, so they tend to clump together, and they see the language choice as a kind of signal of what kind of colleagues they will have working for a particular company.
If all this handwaving has merit, it could be that choosing to do your startup in Lisp when Paul started ViaWeb was’t 100% about having a competitive advantage from Lisp itself, it was having a competitive advantage from having Paul and Robert, and anybody else they hired.
---
With larger companies, they will never say it, but sometimes they don’t want these “great programmers,” with their taste and their salary demands, and their code that causes an army of middle-of-the-road workmanlike programmers to stare thoughtfully for long periods of time working out how it does what it does.
Even if they end up being more productive, perhaps they end up being less predictable, because a smaller team of better programmers is more vulnerable to poaching as the company goes from developing the next big thing to an endless march of adding small enterprisey features. Even if you write in Scala, Haskell, Lisp, or Clojure, when the work becomes maintaining Yahoo... Maybe you don’t want people who relish a challenge, because they’ll quit.
So you want to anti-signal, by rewriting it all in a workmanlike language, so you get workmanlike people. And if it takes touch longer or costs a bit more overall, it’s predictable because the people are more fungible. They’re easier to replace.
So it could very well be that when you’re launching a startup, you want to go out on a limb and choose the language based on the kind of people you want to hire for a startup.
And when you mature, you might rewrite it in another language based on the kind of people you want to hire for a mature company.
And both choices might be right for their times. And neither might actually have anything to do with the “power” or “expressiveness” or “abstraction” of the language itself.
But, great writing/analysis like this about two levels higher than the HN hivemind can comprehend. X = Y (lisp = productive), sure. But X = Y until Z then maybe Q = F is too nuanced for mass understanding around here. If it can't fit into a pmarca simultaneously smug and condescending tweet, nobody will believe you these days.
That is, some languages are more expressive and/or powerful than others. Your great programmers will tend to gravitate towards these languages because they're able to spot and leverage their inherent advantages.
Conversely, other programmers will appreciate other qualities in languages, which may or may not overlap or interact with expressiveness etc. - e.g. ease (read: familiarity), rich ecosystem (read: lots of libraries and stackoverflow posts).
They will say it. I worked for a startup acquired by Yahoo. I was hired by the founders, but then they all left and I was kind of stranded. Anyway I tried to make a go of it, and proposed how we were going to hire some new great people, and my manager expressed confusion about why I wanted great people.
In his view, if you get great people to do not-so-great jobs, you'll end up overpaying and then they will just leave. For him it was obvious that not all jobs are going to be great. Most of them will kind of suck.
It was 180° away from how I thought, which is the standard "only hire A players" script. But then I realized it did make a kind of sense. Startups and small companies that win have geniuses doing everything, including very unglamorous work. Often they have great solutions which work okay for them, but are incomprehensible to others. (Example: Paul Graham's Yahoo Store builder in Lisp).
Such people are compensated with maybe stock options, but more importantly, with freedom. That wasn't really on offer at a big company. So we had to look for average people to do average things at an average rate, and usually it would be multiple people to do the job that one brilliant person would have taken on.
"Still"? I think that thing has been scrapped for 15 years or so, as Yahoo rewrote the store engine. Or you mean just the user visible language that resulted?
I've never used the thing, so I'm just going off what I've found.
According to Graham, in 2003:
> (a) The reason they rewrote it was entirely that the current engineers didn't understand Lisp and were too afraid to learn it.
> (b) The resulting program is a new world's record case of Greenspun's Tenth Rule. The Yahoo Store Editor called compile at runtime on s-expressions made on the fly. To translate this into C++ they literally had to write a Lisp interpreter.
http://people.csail.mit.edu/gregs/ll1-discuss-archive-html/m...
So granted, "Lisp-based" would mean "based on a custom ad hoc partial reimplementation of Lisp."
>In 1998, Yahoo! bought Viaweb for $49.6 million and renamed the service Yahoo! Store.
Note how PG seems bullish on VC growth and large exits, whereas his company didn't take much VC money and had an insignificant exit compared to today's standards (even after adjusting for inflation).
Not to mention that even before sold, ViaWeb had an actual business model and paying customers, as opposed to the model followed by lots of YC companies today.
Practically it's like PG created and sold something like Basecamp/37 Signals, rather than the kind of startups he advocates for...
Besides, there are more reasons for selling your company than simply maximising the financial return. Maybe his partners wanted to sell, maybe he was exhausted, maybe he lost his passion for the project, maybe he valued never having to work again by selling for a certain payoff now.
Well, Mark Cuban et all sold Broadcast.com to Yahoo in '99 for almost $6 billion, and in the same year, Yahoo also bought Geocities for $5.6 billion.
But t's not just "today" -- he's been advocating that for over a decade -- so the change was much less than 17 years ago.
>Besides, there are more reasons for selling your company than simply maximising the financial return. Maybe his partners wanted to sell, maybe he was exhausted, maybe he lost his passion for the project, maybe he valued never having to work again by selling for a certain payoff now.
Sure. So why do PG (and others alike) piss on that model that worked for him?
The YC website says:
> We try to interfere as little as possible in the startups we fund. We don’t take board seats or many of the other powers investors sometimes require. We offer lots of advice, but we can’t force anyone to take it. We realize that independence is one of the reasons people want to start startups in the first place. And frankly, it’s also one of the reasons startups succeed. Investors who try to control the companies they fund often end up destroying them.
> One concrete consequence is that Y Combinator funding lets you sell early, if you want to. It can sometimes make sense to sell yourself when you’re small for a few million, rather than take more funding and roll the dice again.
I think I've heard that argument before, and it doesn't get more convincing over time...
http://www.paulgraham.com/progbot.html
That's a similar idea, but with more of a focus on a single organically home-grown Lisp environment to make your application easier to write and change.
I love this, by the way:
> Language and program evolve together. Like the border between two warring states, the boundary between language and program is drawn and redrawn, until eventually it comes to rest along the mountains and rivers, the natural frontiers of your problem. In the end your program will look as if the language had been designed for it. And when language and program fit one another well, you end up with code which is clear, small, and efficient.
However, I don't completely agree with the conclusion. I think that what we need is to correctly identify the situations in which giving up power in a specific way will give more power elsewhere.
You can't always know this up front. And you can't just get all this benefit for free by religiously following the principle "always go for the least power possible".
Instead, we need to look at the status quo -- even parts of it we haven't questioned for a long time -- and ask, "Where could I give up a little power in exchange for much greater benefit elsewhere?"
Hence, why we want powerful and versatile languages with optional reductions via DSL's.
Changing requirements and the browsers wars made the web a mess, not this design decision. If the web is a set of linked documents available for public consumption, do we really need to encrypt it? I would say yes (because information about STD treatment may be public, but your interest isn't), but it would be understandable to leave it in clear-text. But what about a worldwide commerce platform? Oops, we'd better layer on TLS.
That's a tangent topic I'm not really focusing on. I'm talking about the presentation, efficient transport/updating, storage, and so on of content & data accessed via the web. Aka web sites and web applications. I'm saying Turing complete HTML doesn't cut it by far which is supported by the fact that hardly anyone uses it: server-side includes and Javascript at a minimum going back to 90's. Leading to...
"Writing and using a DSL in LISP seems to be the definition of reducing your power in a case you find it useful. I'm not sure what nuance you're drawing."
It reduces power for that DSL specifically. This gives us the advantages of reduced power. However, every other requirement can then leverage either DSL's or the powerful language to solve them. Or, like Opa, one can just use a powerful language with good attention to safety. In any case, you get a language that solves your current problem, solves your [likely] next problem, solves them well, is efficient, is compatible with HTML or whatever, and is consistent across the stack.
Sounds better than the hodge-podge of crap that mainstream, web apps are made off. The mess certainly wasn't created by HTML's design but HTML's & its partners aren't suited to dealing with the mess most effectively. Right tool for the job, ya know.
Turing-completeness is overrated and is more often that not a cop-out excusing poor language design. Sub-Turing-complete domain-specific-languages simplify program design, reduce bug surface area, and aid analyzability and understanding.
How many “sub-Turing-complete” languages can you list?
Note “sub-Turing-complete” may not be the right term to use for all languages that do not admit general recursion. There is a particularly interesting subset of total languages for which the preferred term is “languages with control over non-termination”.
http://www.cse.chalmers.se/~nad/publications/danielsson-sema...
https://personal.cis.strath.ac.uk/conor.mcbride/TotallyFree....
There's been some promising movement recently in the direction of DSLs. Reactive programming seems to finally be gaining popularity, which allows us to replace gobs of glue code with simple declarations. Erlang's OTP has been gaining popularity too, allowing declarative design in the large.
> Very few new mainstream languages have anything new to say
> these same languages try to position themselves as complete
> ecosystems, rather than simple tools
SQL is a perfect example of this. The fact that it is (barring recursive CTEs) not Turing-complete is what permits very intelligent automatic query optimization. I can write simple, clear, concise, performant SQL code that blows the pants off anything I can do without extraordinary effort in a general-purpose language. It is this that has made SQL so indispensable and enduring a tool.
Also, I sincerely hope you're not trying to synthesize ASICs out of code from a language with constructs like call/cc.
Many EEs prefer to use functional languages. If you're wondering why, spend some quality time with Verilog and VHDL.
Edit: Though after looking up call/cc, my guess is that hardware compilers don't support it.
You can't synthesize "functional programs"; there's no analogue of recursion in hardware. You can only synthesize things that happen to look like functional programs because they're written using sexprs and macros and expand out to a static circuit graph which is exactly what Verilog is.
https://www.cl.cam.ac.uk/research/security/ctsrd/beri/
Also you might find Synflow's HDL interesting. It's on my backlog as I'd rather get diverse and experienced HDL users to put it through the paces. They have several, non-trivial IP for sale that they coded in it and for cheap, too.
http://scheme2006.cs.uchicago.edu/05-saint-mleux.pdf
http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=E53...
Note: DDD scheme also supported call/cc. :P It wasn't in the final circuit, though, except as an execution feature.
"You can only synthesize things that happen to look like functional programs because they're written using sexprs and macros and expand out to a static circuit graph which is exactly what Verilog is."
Oh, come on, that's kind of cheating as you could say the same about almost any level of abstraction. You might as well say you can't synthesize the Verilog constructs: you just synthesize something that looks like it into a connection of analog transistors that it actually is. Yet, if you can use that model and it works, then it's a meaningful thing to say you can synthesize/use such a model to get X done with Y benefits instead of hand-stitching cell libraries and gates. Likewise, the more functional (or other) alternatives to Verilog with their advantages so long as they deliver on them in the field.
For my examples, SHard is a research prototype that works on some, limited examples. DDD was used on numerous real-world projects while the spin-off was still in business. Gone now. Bluespec is still kicking ass in industry. So, Bluespec at the least proves out the concept and its advantages over Verilog.
The thing with DSLs is that almost by definition, unless you're building something almost cookie-cutter, you can't build a whole system with a single one. So Scala and Clojure are supposed to be the common ground in which you use and connect the programs made in these DSLs. Of course, since these are internal DSLs, it's often hard or impossible to isolate the code written in the DSL from the rest of the program, so it's hard to apply optimizations and such, but you still gain the code reduction that you talk about.
Exactly, but that is also why you lose basically all advantages of DSLs. I don't think internal DSLs deserve to be called DSLs at all. They are simply libraries that tend to use a lot of operator overloading.
The whole point of DSLs is to formally restrict what a given expression could possibly mean. That's what allows inference, optimization and sensible error messages that use the terminology of the DSL and not that of some underlying much more general language.
Look forward to see what you have as this is a topic I could certainly learn more about. Especially with the "LANGSEC" push throwing labor at it.
Everywhere I've worked, I have seen the same things reimplemented poorly over and over again in general-purpose languages. Every system I've worked on is roughly a mash-up of database, dataflow graph, state machine, work scheduler, IPC, process coordination, programming in the large, and very low-level algorithmic stuff. Each of these could be factored out into a common simple language, but almost invariably they're just patterns imposed on C/C++/Java/what-have-you code. This greatly hampers understanding and analyzability.
There are rare occasions I see DSLs used to fulfill these roles, and it's to great benefit. Working with a well-designed PostgreSQL database rather than some half-baked home-grown object database is a dream. You get a concurrency model that works, and you can extract graphs of your data model for free. Same with using a state machine description language like SMC rather than hand-coding stuff as C++ boilerplate. Same with using a process coordinator like Erlang's OTP (which isn't a DSL but is close enough).
I'm not writing these posts to try to get developers to use DSLs more often. I'm writing them to try to get language designers to design DSLs more often.
On Coq:
Coq is not at all a practical programming language, since it has no concept of communication with external processes (i.e. I/O). This is exactly as it should be, since Coq is focused precisely on the domain of functional algorithms (and proofs). And, I have found that, within that domain, the restriction that all my algorithms must provably terminate is nearly invisible.
There is a very simple general way to ensure that all programs in Coq terminate: augment each recursive function with one additional argument which is a counter which must always count down to 0 (at which point recursion must halt). On the initial call to a recursive function, select a starting value for this counter which is greater than the number of iterations the algorithm requires. (When proving correctness in Coq, you will be asked to prove that this starting value is in fact suitably high.)
I have kept this requirement in mind when designing programs in other languages, and I have found it trivial to meet in all cases. Turing-completeness is not a feature I need in a programming language.
"I'm not writing these posts to try to get developers to use DSLs more often. I'm writing them to try to get language designers to design DSLs more often."
I was actually baiting you to blast you given I guessed the answer to number 1. However, I decided against it since you have the more reasonable position of identifying the problem then pushing responsibility onto language designers rather than users. Many posts seem other way around. So, that's refreshing. :)
"There are rare occasions I see DSLs used to fulfill these roles, and it's to great benefit."
I agree. iMatix, REBOL/RED, and LISP community (esp Racket) have shown it consistently in terms of productivity and reliability with performance being better sometimes. True even for general-purpose, systems programming: see Ivory langauge from Galois.
"SMC rather than hand-coding stuff as C++ boilerplate. Same with using a process coordinator like Erlang's OTP (which isn't a DSL but is close enough)."
OTP as a coordinator DSL I haven't thought about. Might look into it to see if there's design wisdom to learn for a future DSL outside of Erlang.
"I'm writing them to try to get language designers to design DSLs more often."
I know there's been some good books and essays on how to best design DSL's. Most of us don't know which ones are the good ones. Do you have any links for me or others following along on that? I figure the old wisdom augmented with lessons learned from LANGSEC stuff will be a good start.
" This is exactly as it should be, since Coq is focused precisely on the domain of functional algorithms (and proofs). And, I have found that, within that domain, the restriction that all my algorithms must provably terminate is nearly invisible."
Wait, are we discussing avoiding Turing completeness or requirements for a termination proof? Termination proofs are done on code in Turing complete languages many times a year in academia. Harder than Coq model of an algorithm for sure but way more useful. I'll just let you elaborate on why you focused on that point.
So, trying to cut through the fog that remains, would you say a good approach is using a language like Haskell or ML that compiles to System F (Turing incomplete) with DSL's where possible? Aside from your DSL point, is building on or alternatives to things like System F what you would like to see?
" Turing-completeness is not a feature I need in a programming language."
Curious, how do you handle I/O and other OS interaction without making it so complex a new programmer must learn Haskell, monads, etc? Is there a simple route to that part of things?
This is a truism.
For example, I can imagine a language that you can analyse statically to determine not just whether it halts, but how long it will take to do so. As an extreme example, imagine a language with only the four arithmetic operators. No functions, no classes, no references / pointers, no recursion or loops or goto, nothing else. You can statically analyse a program in this language to determine, for example, that if each arithmetic operator takes 1ms to execute, then a program containing 100 of them in its source file will be done in 100ms.
Now, what can you add to this language while still keeping it analysable? Maybe functions, as long as they are not recursive. You'd add only things you can mathematically prove have a bounded runtime. Maybe fixed-length arrays, like int[10], not int[n]. Maybe loops that run a constant number of times, like:
repeat (10) { ... }
not:
repeat(n) { ... }
What good would such a language be? Maybe for document macros, where I won't be able to hang or crash your document viewer or editor with a macro that runs forever? Maybe there are better uses for such a language than this made up example.
Or maybe bounded running time is not the most useful subset of Turing-completeness one can design. Maybe a language that's immune from side-channel attacks, if that's possible? Or something entirely different?
But think about it: how often do you need to perform some complicated algorithm on a function, or store a dynamically generated function in a data structure? Almost always, first-class functions are simply used as a means of genericizing or parameterizing code. (e.g. as arguments to `map` or `fold`, or currying.) (Languages like Haskell and Coq that are deeply rooted in the lambda calculus are a notable exception to this; it's common to play interesting games with functions in these languages.)
You can get the same capabilities by making functions second-class objects, with a suitable simple (non-Turing-complete) language with which to manipulate them. That language can even be a subset of the language for first-class objects: the programmer is none-the-wiser unless he/she tries to do something "odd" with a function outside the bounds of its restricted language. Generally, there is a clearer, more efficient way to express whatever it is they are trying to do.
There is some precedent for this. In Haskell, typeclass parameters live alongside normal parameters, but aren't permitted to be used in normal value expressions. In OCaml, modules live a separate second-class life but can be manipulated in module expressions alongside normal code. In Coq, type expressions can be passed as arguments, but live at a different strata and have certain restrictions placed on them.
Unfortunately designing languages like this is hard. It's easy to just say "well, functions are a kind of value in the interpreter I just wrote; let's make them a kind of value in the language". This is the thinking behind highly dynamic languages like JavaScript, Python, and Elixir: the language is modeled after what is easy to do in the interpreter without further restriction. The end result is a language that is difficult to optimize and analyze.
It's a lot more work to plan out "well, I ought to stratify modules, types, functions, heterogenous compounds, homogeneous componds, and scalars, because it will permit optimizations someday". But these are the languages that move entire industries.
For instance, Haskell's type language (without compiler-specific extensions) has no beta reduction, because only type constructors can appear not fully applied in type expressions anyway. This reduces its “raw expressive power” (w.r.t. System F-ω, whose type level is the simply typed lambda calculus), but it makes inference decidable, which helps machines help humans. It also helps humans in a more direct fashion: Haskellers often infer a lot about their programs just from their type signatures, that is, without having to compute at all. It's no wonder that Haskellers love reasoning with types - it's literally less work than reasoning with values.
So I agree with you that “first-class everything” isn't a good approach to take. A language with first-class everything is a semantic minefield: Nothing is guaranteed, sometimes not even the runtime system's stability. (I'm looking at you, Lisp!)
---
But, on the specific topic of first-class functions, you'll pry them from my cold dead hands. Solving large problems by composing solutions to smaller, more tractable problems, is a style that's greatly facilitated by first-class functions, and I'm not ready to give it up.
My only claim is that you don't need a Turing-complete language to compose functions. As an thought-exercise, consider replacing all uses of first-class function compositions in any given OCaml program with top-level module compositions. You lose nothing: no-one (who designs maintainable software) uses the full expressive power of a Turing-complete language to manipulate first-class functions.
(Of course there are more user-friendly ways to implement second-class functions, which make using them less a burden, but no extant language has such a system.)
And, no, an ML-like module system wouldn't make up for the lack of first-class functions. Standard ML doesn't allow any module recursion whatsoever, and, while OCaml allows it, it's quite unwieldy to use in practice. It would be literally impossible to define a function like `fix`, which requires the ability to call its function argument (let's call it `f`), supplying an expression containing `fix` itself as argument.
How often have you had to define `fix`? It's not even possible in certain strongly typed functional languages.
I stand by my assertion. First-class function manipulation is not a thing 99% of programs need to do.
Indeed, most of the arguments made are the same that were used as justifications for DSLs a while back:
> The problem with this kind of freedom is that every bit of power you insist on have when writing in the language corresponds to power you must give up at other points of the process — when ‘consuming’ what you have written.
A good DSL lets you create, for example, business rules that are easy to maintain (for those familiar with the business) but don't prevent me from swapping out the implementation that drives them for something better on down the line.
http://www.ccs.neu.edu/home/matthias/manifesto/sec_pl-pl.htm...
Kernel is like Scheme, except environments are first class objects which you can mutate and pass around. A typical use for such environment is for the second argument of $remote-eval, to limit the bindings available to the evaluator. If you treat an eDSL as a set of symbols representing it's vocabulary and grammar, then you bind them to a new environment with $bindings-environment, then passing a piece of code X to eval with this resulting argument as the second environment, will ensure X can only access those bindings (and built-in symbols which result from parsing Kernel, such as numbers), and nothing else.
There's a function make-kernel-standard-environment for easy self-hosting.
Trivial examples:
($define! x 1)
($remote-eval (+ x 2) (make-kernel-standard-environment))
> error: unbound symbol: x
($define! y 2)
($remote-eval (+ x y) (make-environment ($bindings->environment (x 1))
(make-kernel-standard-environment)))
> error: unbound symbol: y
($remote-eval (+ x y) ($bindings->environment (x 1) (y 2) (+ +)))
> 3
($define! x 1)
($define! y 2)
($remote-eval (+ x y) (get-current-environment))
> 3
($define! $remote-eval ($vau (o e) d (eval o (eval e d))))
And contrary to the opinions in the article, Kernel is the most expressively powerful language I know, and I'd recommend everyone learn it. You have a powerful but small and simple core language, from which you can derive your DSLs with as little or much power as you want them to have.
The ability to restrict the environment is key. Creating a DSL in standard Lisp would create a more powerful environment: the DSL would include all Lisp features plus the domain features.
In short, this is, to me, one of the most important principles of programming.
Purity lets you easily and arbitrarily constrain a function or program -- easing the burden for the reader who is free to assume so much more about what they read.
Case in point: every build system targeting the JVM. Most build systems in general.
I am not surprised. Langsec could really be a gamechanger in many aspects of IT security, yet many people in the Infosec community are not aware of these concepts.
If you're curious, this older talk by Meredith Patterson is still worth watching a lot: https://www.youtube.com/watch?v=3kEfedtQVOY
There is no benefit to restricting yourself arbitrarily based on what someone who isn't even involved in your project thinks. I think that most people have difficulty enforcing a coding standard. This is because they don't have a cohesive team. This is a people problem, not a technical problem. If you try to fix the people problem with technical solution, the problem will just pop up some place else.
The DSL languages are in a special category here. You start with powerful host languages such as LIPS/Racket or REBOL. The language is usually simple enough to understand plus has something like macros for extension. Each domain that we run into a lot and/or has plenty boilerplate can have a DSL that lets us concisely express those operations both for productivity and clarity. The underlying language is powerful and still comprehensible for times when the DSL's aren't enough.
So, the author is a bit off. What we need isn't less powerful languages. It's powerful languages maintaining clarity and avoiding the everything but the kitchen sink philosophy.
Note: HTML/CSS wasn't the best choice to support author's point. It was good for static pages. Yet, people needed more power. So, they added a crappy client-side language, all kinds of server frameworks, then all kinds of client frameworks, and so on and on. Languages like Curl and Opa handle the situation with a unified, powerful approach that was more clear than the web ecosystem in general. Also had less security and maintenance issues. So, author's example works against them in practice.
http://blog.higher-order.com/blog/2014/12/21/maximally-power...
In the `urlpatterns` example,
urlpatterns = [
url([m('kitten/'), views.list_kittens, name='kittens_list_kittens'),
url([m('kitten/'), c(int)], views.show_kitten, name="kittens_show_kitten"),
]
The lesson I gain is "figure out the implicit invariants from your design decisions, make them explicit and try to keep as many of them as possible, only letting go once you have no other choice."
(I'm working on a blog post about this.)
Many new features are variations on old features. The incremental cost of implementing one so that I can cut and paste the next half dozen is small. The cost of refactoring to a generalization after that is also small. But the cost of poorly generalizing early and having to reimplement feature one and everything it touches in order to make feature two possible is quite painful.
This becomes more obvious when you have different codebases with similar features that might want to share from time to time. A formally generalized solution becomes a big dependency, as it relies on a heavily specified problem. A slightly too primitive solution flows between codebases freely - plug in new primitives and it runs.
Of course you end up building your own little standard library over time, but then you've got to import it into any given project.. Just an irritating extra bit of friction. And reading someone else's code means learning their standard library... (is it a standard library when everyone has their own?)
Tcl's flexible, to the degree of being able to define new language constructs or redefine built-ins (e.g. you can redefine the basic if-then if you want), which is great in some ways.
But it always seemed to me like the language culture is so deep in the mentality of "if you need, implement it yourself," that the flexibility becomes a crutch for the language developers.
What I mean is.. well, the basic data structure is the list, and you get built-ins for adding to a list (lappend), reversing a list (lreverse), but removing an element from a list (what would be 'lremove')? You've got to code it yourself.
Hey, the man page for 'lreplace' even gives an example! http://www.tcl.tk/man/tcl/TclCmd/lreplace.htm
Why isn't that built in? The common answers are, it's so simple you can code it yourself, or you might want an lremove with different behavior than me. To the second answer, realistically, if the language had a built-in lremove, most people would just express their programs to use it, and those who needed a different one could still write it. To the first, yes, it's simple, but it's just a little bit of sand in the underwear--unpleasant friction you'd like to ignore but keeps popping up.
Lack of built-in OO was the same way until relatively recently in the language history. Lots of people made home-brew ones, and there were several common OO systems, but what's a newbie to do? Research all the different OO systems to make a decision first? More sand in the underwear. Run into enough of that stuff and it just drives you crazy and makes you want to stop using the language.
http://www.cs.virginia.edu/~evans/cs655/readings/steele.pdf
In a commentary on the essay, Scott Rosenberg says:
Plainly, Steele started out intending to show just how painful the words-of-one-syllable limit is — and, by extension, how desperately programmers need the ability to expand the vocabularies of languages that feel impoverished (“a thought that seems like a primitive in our minds turns out not to be a primitive in a programming language, and in each new program we must define it once more”). Yet what’s fascinating about “Growing a Language” is that the limits Steele imposed on his expression make the paper far more accessible, precise and clear than most computer-science writing.
I'm a huge fan of less is more. I review a lot of code.
They don't describe an algorithm or a behavior, they just describe data. For that purpose, it's preferable to have a format that's as simple to read, process and analyze as possible. On the other hand, actual programming languages are meant to describe an algorithm, and "power" usually means either high-level expressiveness (Scala) or low-level adaptability (C), both of which are very important in some contexts.
I mean sure, technically that's true, but it's also not interesting.
The interesting thing about regexp is that it's really damned close to the mathematical notations used for describing regular grammars, and therefore it's a good, tight way to specify pattern matches on regular grammars.
No one ever really answers with "I've been programming the regular grammar parser". Sure, maybe technically that's what they've been doing, but the context isn't really right.
You're totally missing my point. A regular expression describes a computation - the series of state transitions a nondeterministic finite state automaton undergoes in order to decide whether an input string belongs to a language. (The fact that this computation is typically optimized by regex compilers is another matter.)
A programming language doesn't need to be general-purpose to be a programming language.
> The interesting thing about regexp is that it's really damned close to the mathematical notations used for describing regular grammars, and therefore it's a good, tight way to specify pattern matches on regular grammars.
Are you saying that what matters most about a language is its surface syntax? Seriously? Wow.
So does the flashing light, it also has the cool advantage of being parameterized by time.
> Are you saying that what matters most about a language is its surface syntax? Seriously? Wow.
If that's what you took from that then there's a knowledge gap and it will do us no good to continue this conversation.
Sounds like a job for free applicative functors, which are strictly less powerful than free monads (which you can imagine as the set of interpretable languages). From John De Goes's ScalaWorld 2015 presentation:
Free Functors: Programs that Change Values
Free Applicatives: Programs that Build Data
Free Monads: Programs that Build Programs
D is one of the most powerful languages available, and it compiles faster than about any.
In any case, in the hands of an expert, using a powerful language results in simpler user code. This is because the program can be crafted to look like the problem being solved. In fact, it's why we have programming languages in the first place rather than writing assembler in hex bytes.
An example might be saying something like "Things you do in a spreadsheet will never benefit from object orientation to such a degree as to justify implementing that ability for the sake of the business's bottom line."
As a former analyst in a financial firm, I encountered this idea all the time. You can use Excel's "slope" to do regressions. You can even do matrix arithmetic if you're willing to deal with the syntax. You can program basic functions.
In a strictly time-to-market sense (the time from some ad hoc financial analysis in a spreadsheet to 'market' -- either producing a report for your boss, a strategy implementation, or some other deliverable based on what your spreadsheet calculations showed you), throwing stuff in Excel, copy/pasting code from templates, etc., can't be beaten.
But this is a very narrow view. For example, data provenance is extremely difficult if your work is a directory of spreadsheet files. Even if they are version controlled, there is no automatic way to understand how the spreadsheet programmer intends to copy/paste some functionality out of one template and into another at the moment a new analysis is performed. The dependencies, if documented at all, are documented only in natural language descriptions, rather than overt 'import' or 'include' style statements, or static analysis of what gets used where.
Unit testing is often ignored in a spreadsheet environment and it's a huge pain to do in the rare cases when someone actually tries to do it. The focus on superficial aspects of 'time-to-market' also puts pressure to avoid version control, even if the spreadsheet paradigm as a whole doesn't necessarily have to reduce use of version control.
But far beyond any of these items, there are (for example) tools like Pandas, or even xlwings, in Python, or HFrame/HMatrix in Haskell, and I'm sure many other things in many other languages, which presents you with what is effectively a spreadsheet as an abstract data type.
You can programmatically perform the spreadsheet interactions that would otherwise have been manually reproduced, and you can include more advanced software techniques, like logging or unit testing, since you are simply working in a full-featured programming environment, rather than an environment where the feature set is purposely limited.
In this case, I've never heard any compelling argument for why a direct spreadsheet is better than a spreadsheet-like abstraction in a full programming language.
The arguments I have heard to justify continued direct use of a spreadsheet are:
(a) the people using it don't know how to write code and the business doesn't believe it's worthwhile to hire/train programming talent for such a role.
(b) someone who makes the decisions is using a hyperbolic discounting function when they assess the future returns of compounding automation through proper software design vs. just-get-it-done-today-by-copy/pasting-in-a-spreadsheet-if-you-have-to attitude.
(c) "Programming" is a low-status activity, and so business/marketing/financial analysts must perform some activity that is plausibly different than "programming" so that "programming" doesn't experience a status rise within the organization, and potentially affect people's raises/bonuses/promotions or project allocation.
Let's take a step back from this example of spreadsheets. What is the general phenomenon going on? I would argue that it is about automation and productivity. But then, general purpose computing is also about automation and productivity. There can be counterexamples for sure, but most often newer layers of abstraction are introduced because they create genuine value in terms of making it easier to automate something or making a more generalized lever that lifts whole categories of heavy things instead of just lifting this or that specific heavy thing.
No matter what limited-scope tool you start out with, over time lots of the tasks performed within the scope of that tool will be repetitive and/or will agglomerate into clumps of similar work that can be factored out into an abstraction.
When the tool also has general purpose computing abilities, you can take advantage of these opportunities to automate or factor out clumps of work and write generic, reusable solutions. The process of doing this almost always has huge positive effects of productivity, especially as it accrues and compounds over time. It's overwhelmingly worth it to pay generally modest short term costs to work in this manner, rather than trying to hack in ways of coping with bottlenecks in a tool that can't support general computing.
Part of the problem, though, is that many middle-managers and up within any given organization do not understand how this works. The syntax of their brains only manipulates the "programming language" of the business. Deliver X to Y; ship Z by Friday; give me a forecast of W. They don't unpackage "Deliver X to Y" down into its atomistic components and ask to what extent general programming can help, and whether or not it will have returns on "Deliver X1 to Y1" next week and "Deliver X2 to Y2" the week after that.
When programmers throw an exception within the business's programming language, Exception('We can't ship Z by Friday if we also write sufficient unit tests this week.'), what happens? Let's just use unit tests as the example of a "general computing" behavior that might sometimes be axed in favor of justifying a narrow-scope tool environment for business reasons.
In some organizations, this is considered very carefully, and a lot of attention is paid to the engineering assessment. The managers may ultimately come back and say, "You know, we really looked it over and did some careful thinking, and we still must ship Z by Friday, so skip the tests." In that case, probably the managers and engineers alike both agree that you don't want a limited programming environment for the task. Unit tests mattered to the engineers, and they also mattered to the managers even though they had valid reasons to skip it this time. But all parties probably agree that, in principle, unit testing would have been better and should be at least a possibility.
In other organizations (a lot), the Exception is simply caught and never handled. Managers don't like hearing about something that sounds like a whiny and low-status issue ("programmers want unit tests"), and so they mandate that such things be skipped, and support using tooling environments where such things are not even a possibility. And at the end of the day, they justify this as a necessary business reality, when it's pretty questionable whether they truly ran any numbers to decide if the longer term gains from unit testing more than offset any short term slowdown to write the tests.
In practice, it's always some gray-area mixture of all of these things, and sometimes there definitely is a legitimate reason to forego general computing options for business reasons.
But I am very skeptical of the more far-reaching claim that it helps bottom-line business productivity to reduce the possibility of automation and software productivity practices enabled by general computing capabilities.
I don't personally see any reason to believe working for a start-up could possibly be a good idea otherwise. (And since most start-ups only pay lip service to supporting best practices, while really supporting only short term incremental business gains like any other kind of organization, this is generally a good reason to default to believing that it's a bad deal to work at any given start-up.)
For me, this also has a lot to do with vision and consistency. If you create a start-up and your only goal is to sell it (or, more realistically, you are expected to do this because it is the goal of the VC firm you got into bed with), then you're not only not going to focus on best practices, but you're also going to jettison any part of your mission or vision any time it's not convenient to some incremental short-term growth opportunity. By the time you reach a point to sell the business, it may be completely unrecognizeable from the vision you started with.
If your only goal is to make money, this might not be a problem. But probably not very many people who agreed to work with you will share the feeling -- and they certainly won't stand to make meaningful amounts of money -- and so they are unlikely to be happy workers through most of the process. That means you haven't been getting their best effort all along, and the product is probably shoddy.
Generally (but not always), good engineers don't want to work for something that is explicitly a hype machine where maybe later quality will be hacked back into it, but probably not. So start-ups that fundamentally take an approach of short term product-market fit are pretty much by definition staffed by bad and/or unhappy engineers.
Contrast this with something like 37Signals/Basecamp. Of course they had to make engineering sacrifices along the way and didn't always do everything in some pedantically best-practices-adherent way. But their ethos/vision was to be much more about best practices than about unreasonable growth or winning an acquisition lottery. Yes, they were driven by succeeding with product/market fit, but they didn't ever become a slave to it or turn into zombies about it.
Given that there's such a poor success rate among start-ups (a large base-rate bias towards failure mode), it's hard to draw much from outliers of any kind, whether they are like 37Signals/Basecamp, or they are like some always-pivoting shop that always favored short term business gains but still succeeded despite it. But my perspective is that it's way, way better to be on the side that pedantically clings to an engineering vision and has a general ethos of turning down short-term business gains as a means of investing in longer-term best-practices-focused infrastructure.
These are machines we're programming. They have instruction sets and memory architectures. We need more languages like Jai, IMHO, that give more power to the programmer to exploit the target platform underneath and provide interesting patterns for controlling resources and transformations.
I generally prefer simple constructs that require multiple lines over complex one-liners. Although it depends on who is on the team. I prefer to write code that any polyglot programmer with a modest level of mastery in any specific language can understand.
There is a massive confounding factor: popularity, in the sense of what is familiar to the largest number of people. And this, in turn, is largely an accident of history, politics, and marketing.
Which is to say, sure, more people will understand a "for" loop written in the familiar C/Java style than a "map" one liner. But that does not mean the "for" loop is easier to understand, in the sense of expressing a concept naturally and declaratively.
Or worse yet, when someone is trying to remove items from a list that is self resizing, it is very easy to make bugs (by not adjusting the counter in the for loop when one is removed).
Sure, you have to draw the line somewhere, but refusing to use Python/CoffeeScript comprehensions because not every language has them, or preferring to use your own utility function for removing doubles from an array instead of underscore.js's _.uniq to me feels like optimizing not for the average programmer but for the mentally challenged.
Sometimes you do get to have your cake and eat it too: simple one-liners instead of confusing and verbose line noise.
IMHO, we need simplicity and clarity in our data structures, APIs, JSONs and powerful (but simple, still) tools to modify that data. If you're using Clojure this library has been great for me recently https://github.com/nathanmarz/specter .
Unless the argument is that we need another big wave of even weaker tools, it feels unnecessary to give this sermon in 2015.
(I invite people who disagree to express something concrete in reply.)
A finite state machine can easily be made basically un-debuggable by human beings.
It's just ... maybe it's the crowd I hang out with, but I feel like these ideas have already totally won.
BER does have "constructed values" which are most 'obviously' parsed recursively. OpenSSL handles this via C macros that unroll the parsing 7 levels deep, but no more.
(What does it do when given nested constructed values? Well-- when its expecting an integer it goes and traverses into the constructed values in a particular order, pulls out the primitive types at the bottom, concatenates their bytes, and treats the result as an unsigned integer. ... Why? Because this is precisely the kind of madness that happens when you deploy a 'too powerful' language! :) )
oh well. I'll just have to write some Lisp tonight. :-)
Things were saner in 1990. Arguments about whether C was "better" or "worse" than Lisp, without giving any application space, tended to fall on bored ears.
Nonetheless, one doesn't have to be a LISP weenie to see how messed up things are. LISP, REBOL/Red, the 4GL's on top of BASIC, etc... these all showed that a simple core language could be combined with macro-like features to keep expression clear, performance high, integration problems low, and boiler-plate low to non-existent. The opposite of the Java ecosystem.
A maintainability win is concise code that gets the job done and fast. Mainstream languages aren't it. They just want too many features to do too many things. Go is the exception is it follows the Oberon tradition. It's too simple imho. Julia is the most interesting language of recent times in balancing productivity, performance, macro's, safety, legacy compatibility, and maintenance. And, of course, it's Lisp underneath. :P
2. Do we really wish HTML were Turing complete? LaTex gotta be Turing complete? The author isn't really saying everything has to be underpowered and take 2000 lines to do what 100 lines of Python can do.
3. "Right tool for the job" is the slogan that could defuse about 90% of the language wars. Instead of saying "HTML is better than Lisp" and trying to start a flame war, I could notice that sometimes I just want to mark some stuff up. I don't always want to debug my text.
Irrelevant to the claim I made. You pointed out Java as a straight win for both variant of Pascal philosophy and maintainability. I countered that this wasn't true given Java was inefficient for over a decade, very hard to port, bloated, hard to understand, and anything but easy to maintain. I gave counter-example of Modula line, esp Modula-3, that was easy to totally understand, consistent, fast to compile, easy to port, cpu/space efficient, and supported most features industry needed for programming in the large. Java got popular due to social and political reasons on top of massive financial push by Sun. It's technically crap. Latest Modula-like language is Go and most Java programmers experimenting with it write like it's a breath of fresh air. That's due to Java problems more than Go strengths but Go embodying Pascal line's good traits helps.
Although, there's irony in you citing Java while critiquing LISP: Java co-author, Guy Steele, said one of Java's successes was dragging some C++ developers half-way to the capabilities of LISP. Implies he knew both technically inferior to LISP even when designing Java to replace C++. ;)
"Do we really wish HTML were Turing complete? LaTex gotta be Turing complete? The author isn't really saying everything has to be underpowered and take 2000 lines to do what 100 lines of Python can do."
I'm sorry to deliver the news but the fools did it anyway:
http://lemire.me/blog/2011/03/08/breaking-news-htmlcss-is-tu...
Also, representing HTML actually takes less in Scheme due to HTML's complexity and redundancy. Same for XML. See here:
https://en.wikipedia.org/wiki/SXML
That's not your point, though, so I'll focus on it. We have two major approaches we take here: a limited language like HTML that's not Turing complete and which we extend all around with Turing complete languages; a Turing complete language with a subset, even equivalent to HTML, that's not Turing complete but lets you have a great, highly-integrated & client/server-consistent language when you need that. Simplest version of the 2nd category is one of the Scheme web application systems that leverage a form of HTML with ability to transform and present it only limited by Scheme's power. Use simple version if you can then use Scheme otherwise. Can do something similar with function calls in other languages that correspond to HTML, building up an AST to output static or dynamic content.
""Right tool for the job" is the slogan that could defuse about 90% of the language wars. Instead of saying "HTML is better than Lisp" and trying to start a flame war, I could notice that sometimes I just want to mark some stuff up. I don't always want to debug my text."
I try to avoid them and convey information instead. My point the whole Turing complete vs incomplete is kind of a false dichotomy where we can't have both. HTML can be a DSL and is in many real works. LISP isn't required there although macros & easy parsing are a great aid in any language for DSL's. Rebol, Forth, and Julia can do it that I'm aware of. That almost nobody uses pure HTML w/ no server-side anymore shows it wasn't adequate. Led to proliferation of worst stack of Turing complete crap I've ever seen with a few exceptions sprinkled in.
So, what to learn from the mess and how to do it better next time? A powerful client- and server-side language with easy syntax + HTML DSL seems like it would've been the best option. Was done by certain groups in fact while delivering on its goals. Let's one default on safe & simple while incrementally adding on with consistency, performance, and power [where necessary]. Or an integration of a bunch of Turing incomplete DSL's for the crowd that loves those.
That is, the set of fundamental concepts exposed is no greater than the bare language, except the higher abstraction allows controlling which subset of the power is exposed so that type inference can be kept, and various syntactic sugar can be used.
Haskell (with virtually all extensions used) doesn't have that many fundamental concepts, and a small minority of extensions actually are fundamental at all.
Basically, YAGNI.
> I also need a word about definitions. What do we mean by “more powerful” or “less powerful” languages? In this article, I mean something roughly like this: “the freedom and ability to do whatever you want to do”, seen mainly from the perspective of the human author entering data or code into the system. This roughly aligns with the concept of “expressiveness”, though not perhaps with a formal definition. (More formally, many languages have equivalent expressiveness in that they are all Turing complete, but we still recognize that some are more powerful in that they allow a certain outcome to be produced with fewer words or in multiple ways, with greater freedoms for the author).
I understand that adding
- automatic memory management - exception handling - object oriented facilities (polymorphism, inheritance)
make a language "more powerful" by allowing the programmer to think at a higher level. For me, this means the language gets out of the way and I can focus on the problem at hand, but everyone seems to have a different idea of what "getting out of the way" means and what constitutes the ideal "default toolbox" for a programming language.
For decades it was true that higher level meant more powerful meant a better default toolkit for the developer, but I resonate with the sentiment that a plateau has been reached in terms of designing the basic toolbox a languages makes available for the programmer.
A less powerful language means easier and more readily available tools to parse and manipulate programs written in that language, which leads me to...
2.
> This happened to me recently, and set me off thinking just how ridiculously bad our toolsets are. Why on earth are we treating our highly structured code as a bunch of lines of text? I can't believe that we are still programming like this, it is insane.
With all due respect, JavaScript as implemented in node.js or in Chrome is a brilliant language and is a fine environment for implementing a static code analyzer and transformer. The same is true of Python, Scheme, C, C++, Java, and many other modern programming languages. It takes relatively little code to to implement powerful tools to parse and manipulate source code when relatively few syntactic forms are accepted. See for example my autoclave.js project below.
The one Wikipedia article I come back to again and again is the "Additional Example 1+1" of https://en.wikipedia.org/wiki/LR_parser
3.
There is a parallel between the instruction set architecture (ISA) of a chip and the syntactic forms that a programming language recognizes. One measure of the "size" of a language is the number of productions in the grammar definition. So I think of a "highly expressive" language with a complicated grammar as a "CISC language" and as you say a "less powerful language" as a "RISC language". All other things being equal, it is easier and faster to implement a toolchain for a smaller language than for a larger one. This is why it is (or was, at any rate) reasonable to ask a first-year college student to implement a meta-circular interpreter in Scheme (MIT 6.001). This is not possible for most other languages (and it is certainly not possible in Python).
This project is just one attempt to carve out such a usable "RISC language" as a subset of JavaScript. If you have the time, I'd appreciate your feedback, thoughts, and suggestions. Thanks!