Hacker News Reader

It's kind of depressing, but I tend to agree. What works for absolute beginners does not and will not work for experts because it's inherently limiting. A lot of the "let's make programming more accesible for beginners" experiments have already been performed and they didn't cause more beginners to become experts (nor did they empower experts). IMO[0] we really need to move closer to mathematics and precision to actually achieve software that you can rely on. That means foundational work in PL semantics, type theory, PL pedagogy, etc.

[0] And this is just my opinion because I have no actual rigorous research to back this up. At least I can admit that, contrary to most of the people who are agitating for approach A or B to software development.

EDIT: s/studies/rigorous research/. "Study" can mean anything from "I though about it for 5 minutes" to "rigourous randomised trials", so I shouldn't use that term.

we really need to move closer to mathematics and precision to actually achieve software that you can rely on.

What's meant by reliability?

That's a good question. Speaking for myself... I'd settle for some comibined metric of "fewer security bugs" and "fewer unintentional behaviors" (per spec). You could argue about UX and such, but most of the real damage that's caused by defective software is basically about security bugs. The "fewer unintentional behaviors" bit is triciker, but math and precision (about what you want) can help with that. Usually a (mathematical) spec is shorter and more concise than the program implementing it.

HN couldn't have been designed via spec.

Huh? I'm pretty sure it started out as an (informal) spec in somebody's head, which was then modified to account for the inflow of noobs like me :)

At the very least the modding system required some abstract analysis to account for trolls and such.

I think the distinction is between agile (we'll start with a vague spec, and flush it out as we go) and waterfall (we'll start with the perfect spec before we start coding). Obviously, there are many points in between.

Nice concise mathematical specs are rare in practice, not because programmers are incapable of expressing such specs (and most probably aren't), but most programming problems are not expressible rigorously via math; they might not even be amenable to much analysis (so called "wicked" problems are common in programming).

I'll take a guess that something like pure functions lend themselves to formal, concise specs. Pure functions are, in the end, just input and output, not something complex like a program that outputs some complex bitmap every 30 milliseconds. Haskell has found pure functions to be very practical to have as 'distinguished citizens'; there are enough problems which can to a great degree be expressed with pure code.

Someone might say "but Haskell itself doesn't solve practical programming problems", or whatever. The point isn't Haskell or pure functional programming per se; it's just an example of things that easily lend themselves to formal specs. Whether more complicated, perhaps effectful programs lend themselves to formal specs (I think ATS might be able to express such specifications) is more hard to answer.

HN was written in a language where functional style is the norm, and yet HN couldn't have been designed via spec. I say this as someone who has thoroughly studied HN's codebase. If you spend time examining it, it becomes evident that most of the features were extemporaneous. There is little which was thought out in advance. The recipe for HN was: do the most important thing each day; repeat for 2000 days. Such a recipe is fundamentally anti-spec, because by day 5 your spec is outdated.

Spec xor exploratory programming. The codebase is the spec.

The problem is that once you get to defining your specs as programs themselves, they can have bugs, and the closer those specs get to maths, the less the people who actually define the specs in the first place - usually neither programmers nor mathematicians - can understand them, reducing the likelihood that anyone will notice the deviations between the spec and what users actually want.

Additionally, frankly, "reliable software" isn't something that's needed at least 99 times out of 100. Software that more-or-less gets some specific task done is. Those tasks can change from day to day, and generally require a few tries before the person who gave the task is even sure what they want, so RAD is more of a necessity than spec-based development.

Basically, we now have this entire class of information workers, and very few tools to give them that are any better than an Excel spreadsheet for the most part. I don't see the article as trying to give traditional programmers new tools, it's trying to give information workers new tools, and probably traditional programmers will find some of those useful.

> I don't see the article as trying to give traditional programmers new tools, it's trying to give information workers new tools, and probably traditional programmers will find some of those useful.

That puts it much more clearly than I managed. Thanks :)

Eventually though, software has to solve a real problem for a human being, and real problems (and humans) tend to be messy. A spec should model the problem and its solution, and will inevitably be sort of messy as a consequence. I also doubt that any spec survives implementation. Solutions must be adapted as the understanding of a problem and its appropriate solutions grow, and it is almost impossible to gain that understanding without experimentation (partial implementation).

Dijkstra (the one who has a graph algorithm named after him, aside from being a really influential computer scientist in many ways) argued more or less exactly what you're arguing: that formal methods (that is, formally proving, mathematically, that your code will behave as it should) are the appropriate way to teach computer science/programming. His argument is basically that computer science is very hard applied math, and should be treated as such.

You might find https://www.cs.utexas.edu/users/EWD/transcriptions/EWD10xx/E... an interesting read, if you haven't read it before. There's been some interesting discussion of it on HN in the past, too.

Dijkstra also complained about lazy students who wasted valuable computing time on assemblers instead of just writing out the machine code by hand.

I am strongly in favour of mathematical methods but that doesn't preclude exploratory work or visualisation. The idea that mathematicians spend all their time constructing rigorous theories and proofs is a complete illusion. It is common practice (outside of the Bourbaki school, at least) to alternate between building intuition through examples and solidifying intuition through rigorous proofs. See eg http://terrytao.wordpress.com/career-advice/there%E2%80%99s-... (by Terence Tao, one of the most talented mathematicians alive).

By comparison, we are extraordinarily lucky in programming in that everything we deal with is computable and representable. It is far easier for us to display examples and visualise systems. So while I agree that more understanding of mathematics and proof would aid in programming complex systems and I have certainly found that my degree in maths was a good preparation for programming, I don't see why we shouldn't also take advantage of the power of computers to help build the intuition that is necessary for rigour.

  > Anon the intern needs to be able to open up Programmingâ„¢ and click 'New Web Form'
  
  are adequate for people who usually don't program, and extremely inadequate
  for people who usually do.

Let's change the example to "Start Web Server" instead.

I think the best solution for this is the way Go does it. It takes the best of both worlds.

  http.ListenAndServe(":8080", nil)

That's one line to start a web server on port 8080. It's as easy as pressing a 'Start Web Server' button, yet it scales to dl.google.com.

The problem is when you press 'Start Web Server' or 'New Web Form' and it generates 300+ lines of boilerplate code that you now have to look at, maintain, and waste your attention on. Advanced users don't want that, and novice users aren't happy about it either.

Instead, take high level things and make them available via high level APIs, that are also extensible and flexible to allow tweaking of what you want [1], if/when you need to.

[1] https://github.com/golang/gddo/pull/143#issuecomment-3194318...

It's remarkable (or depressing) how many of the points in the article that are addressed by Smalltalk -- and yet out of reach because in most environments Smalltalk is just another program, rather than used as the environment in which we do everything.

It includes things like a source browser that allows looking up ways to deal with eg: email, if you have the/some source code, you can easily run it, a built in debugger and a tight write/test/fix-loop etc.

Most of what was Smalltalk has been adopted into modern IDEs (edit and continue, class browser, graphical debuggers, object-oriented programming). Now we are beginning to move beyond Smalltalk; e.g. Smalltalk never provided very powerful search facilities, real live programming, or speculative execution.

How did Smalltalk not provide "real" live programming?

Change your code in smalltalk, say from "w color: black" to "w color: red". And even though the code of the program is "changed," nothing happens because the modified statement is not automatically re-executed. In a live programming system, you get useful feedback about your code edits, not just "fix and continue," and so you would immediately see affected widgets go from being black to red.

Chris Hancock's dissertation [1] is really good at making this point:

> And yet, if making a live programming environment were simply a matter of adding "continuous feedback," there would surely be many more live programming languages than we now have. As a thought experiment, imagine taking Logo (or Java, or BASIC, or LISP) as is, and attempting to build a live programming environment around it. Some parts of the problem are hard but solvable, e.g. keeping track of which pieces of code have been successfully parsed and are ready to run, and which ones haven’t. Others just don’t make sense: what does it mean for a line of Java or C, say a = a + 1, to start working as soon as I put it in? Perhaps I could test the line right away—many programming environments allow that. But the line doesn't make much sense in isolation, and in the context of the whole program, it probably isn't time to run it. In a sense, even if the programming environment is live, the code itself is not

i.e. Smalltalk code simply isn't live.

[1] http://llk.media.mit.edu/papers/ch-phd.pdf

Thanks for clarifying -- I'm still not sure which environments are much "more" live than Smalltalk. If you do make changes, usually that code will be executed on the next call?

Or are you talking about some strict old Smalltalk-80?

While you could change a parameter like color in some text-representation of the live code, wouldn't it be more natural (in Smalltalk) to simply change the object [ed: object instance] in question (and see the immediate change)?

I'm considering things like:

http://pharo.gforge.inria.fr/PBE1/PBE1ch7.html#x32-111006r22...

If Smalltalk allows editing values of variables of running code, and hot code loading -- what is missing? Is it the fact that other systems define the source code as "the truth", and consider the byte/compiled code to be a "secondary artefact" that causes (my) confusion?

Because as far as I can tell, Smalltalk does allow you to manipulate the objects that make up your system live?

Direct manipulation of the program's state (as opposed to writing/editing code that changes the state) is always live of course in any language. So ya, Smalltalk objects are "live", but Smalltalk code is not. So when Hancock invented this term called "live programming" back in 2003, he was focusing on the code (keep in mind, this work exists in a world where Smalltalk is well known, the Morphic paper came out in 1995, and Seymour Papert of Mindstorms/Logo fame is Hancock's adviser).

The problem with editing the program's objects directly is that it is not very powerful in its capability: fine if you want w to always be red, but not fine if their are a hundred w's to edit individually, or if w is red because of some kind of active condition. In any case, hot code replacement doesn't really help either; its useful for sure, but the next step is to change the program's state retroactively [1] to match the new code. Smalltalk never went there, and probably couldn't given the hardware constraints of the time. But now, I think we can go there.

The Pharo folks are claiming that their system supports live programming capabilities, but it is just the same old live object/fix-and-continue features that Smalltalk has had for ages. I suspect that they just really don't know what live programming means, and are latching onto the word because its "trendy."

[1] http://dynamicaspects.org/blog/2012/08/15/changing-the-past-...

Hm. I still think it's a rather thin distinction, and one more concerned with what programs, data and objects are (and their relation to source code), more than with what is possible in Smalltalk.

If we look at what is possible, namely changing data and instructions in ram -- it seems to me that Smalltalk already provides that.

I suppose there is some value to viewing a program as a kind of differential equation over time (where we assume the future has not yet happened, and all input has been given at certain points in time) -- and so be able to change the state at a given present, by changing the program, and accounting for how that change would translate to a different state in the present, given the same inputs... But we cannot stop time, nor the real world -- so most real systems can only be approximated in this way.

The best we can do (with any real system that isn't isolated from the outside) is to take a snapshot at a given time t, make changes to see how that state could have been at t with our new code -- but we then would have to bring our system back into the present -- either by playing back recorded input or by some other means...

I don't think your points about w manage to put weight behind (or communicate) your point: in Smalltalk you could either programaticaly alter all ws, or if you changed the condition on which w depended, all ws would update (in a GUI, or other system with an event loop...). I get that that's not your point though.

The distinction is massive, and why all those Bret Victor demos have nothing to do with Smalltalk. You can record a bit of your program's execution during development.

Anyways, this an active research topic...see:

http://research.microsoft.com/pubs/211297/managedtime.pdf

If you haven't yet.

Interesting, hadn't seen that (have to read it in more detail later). Does sort of remind me of the approach taken to synchronization/managing time for Croquet[1]: Tea time[2] (although the aim there had nothing to do with live programming). Seems like a different need for managing time, but a not entirely dissimilar approach.

[1] http://www.opencobalt.org/about/history

[2] I think the paper I read is this one -- but I can't verify it from here:

"Designing croquet's TeaTime: a real-time, temporal environment for active object cooperation", David P. Reed http://dl.acm.org/citation.cfm?id=1094861

Thank you for the references. I've seen some of these before, but connecting the dots can be difficult (meaning, I have to keep going back to them later). A lot of seems to have gone into Kay's new project @ VRPI.

An earlier paper by Reed also seems relevant:

"Implementing atomic actions on decentralized data" http://dl.acm.org/citation.cfm?id=357355

Again, the goal here is not live programming, but the "boxing" of blocks of code into "atomic actions" feels very similar to the MS paper.

[edit: Some interesting parallels with the earlier story on Soundcloud's Roshi system too https://news.ycombinator.com/item?id=7732696 ]

Thanks, I'll take a look! Look like I'm going to make many changes for the camera copy.

Somewhat tangential (again thanks for the link, and the comments on "live programming"):

"In the Smalltalk model, for instance, state is persistent and code changes don't affect data. In the Clojure model, code is "mostly functional", with a small amount of carefully-managed state. Either model could be a starting point for a system where continuous code changes can be seen as continuous effects." -- Bret Victor http://worrydream.com/LearnableProgramming/

Right, but no one knows yet which will work (my hunch is a combination). My point was that the experiences he showed weren't Smalltalk experiences; we have finally got a lot of great ideas what to do post smalltalk.

The problem is: live programming does not scale and the very few examples are also not convincing.

i.e. live programming isn't programming. It's just a toy now.

It's a dream and a big risk. And I'm sure the destination will not be as envisioned, but we can finally make some real progress in interactive programming over Smalltalk/lisp.

It's too late for an edit, but looking at the article again, especially the conclusion, I wonder even more about the seeming dismissal of Smalltalk:

> * storing code in a networked database with version control and realtime sync

This sounds a lot like Monticello and/or what Lively Kernel does over Webdav?

> * a structured editor to enable rich ASTs with unique UUIDs

I'd say something similar is achieved by Smalltalk with it's view that there is no "source code" -- there is only the compiled code that is also viewable/editable as text (this is not so much about code, but a feature of a truly object oriented system).

> * managing environments declaratively so that evaluating code is always safe

I don't see how "evaluating code" can "always" be safe? Or is this similar to having the ability to deploy a separate dev-image in Smalltalk terms?

> * a uniform (logical) data model where every piece of state is globally addressable

One example of this would be an object graph?

> * a model for change that tracks history and causality

I take it this is the "live programming" bit discussed alongside here. If all input and output is via messages, it would seem feasible to simply log messages (possibly collapse some messages (ie: +1, -1, +1, -1, +1 becomes just "+1").

> * a powerful query language that can be used for querying code, runtime state, causal graphs, profiling data etc

I'm not entirely convinced we need a "powerful" query language. But something to specify context (show me all counters in the date widget) would be good -- but it would be fine to express it as "in:someDateModule var:count". Maybe I just read "powerful query language" differently than the author (intended).

> * composable gui tools with transparent guts

We seem to be reinventing these forever.

> * a smooth interface to the old world so we don't end up sharing a grave with smalltalk

I really didn't want to go all "Smalltalk did that" (I'm not even sure it did I only know new-ish Smalltalks) -- but I think the eco-system is in the process of proving the death-by-different wrong: We use Apps on phones and through the web -- and the fact that eg: Google spreadsheets has pretty a pretty crappy interface with "the old world" doesn't seem to bother anyone. I mean, yeah, we know that we probably need to interface with some form of external file system -- but now we can pretty much just throw http/dav at the problem.

I guess I'm just grumpy -- arguably there's no difference between running a javascript vm on bare metal vs running a Smalltalk system on bare metal (except for the language part, but we can implement the language(s) we need on top of js anyway…). I just can't help but feel we might do better with a system better designed to be a system, rather than the web browser that accidentally became a vm and virtualized display driver. It doesn't strike me as likely that we'll see a proper security model that actually works for javascript in the near future, for example.

What you glossed over was the 'model for change' part and the implications that follow from there.

It's a shame that it's not being said more clearly and explicitly in the post; but that 'model for change' implies immutable/persistent data structures and all that comes along with that.

The foundational assumption that data is never deleted or updated but just garbage collected does really change a lot.

Essentially (I'm hoping!) they're just saying: let's build another Smalltalk but with persistent data structures and reactive programming instead of Smalltalk's MVC model.

But yeah; it does sound like they need to study the Smalltalk ecosystem a bit better to figure out that apart from those two things Smalltalk did already do pretty much everything they want to be doing here; or find a marketing angle that's less off putting to those that know Smalltalk well...

Sure, also see:

https://news.ycombinator.com/item?id=7762535

> They led to the development of Visual Basic & co

Visual Basic was phenomenally successful as an end-user programming tool. There are entire businesses that still run off VB6. It worked because they nailed the early learning curve. It doesn't matter if your language is faster or more maintainable or less bug-ridden and insane - the majority of people will always gravitate towards the tool that is easy to get started with. This is why VB6 and PHP are so widespread.

The moral of the story is that if you want people to program in a sane and scalable way you must make it the path of least resistance. No amount of advertising or education has ever been able to overcome the fact that most people don't give a shit about programming and are just trying to get stuff done.

> if Anon the intern's problem is he doesn't know how to Google for libraries

Anon the intern's problem is that the information returned by google is incomplete and requires manual effort. I don't want to replace stackoverflow, I want to make it executable.

> I do think that the real resolution to this problem is writing simpler programs whose state is easier to track.

Well, that's where the managed state part comes in. We are trying to separate essential state which is actually part of the program logic from accidental state which is just an implementation detail (an idea expounded in http://shaffner.us/cs/papers/tarpit.pdf). That's a whole other post though.

> I'd much rather prefer being good enough a programmer to avoid the need for a debugger that can tell me why foo is not 8 despite the fact that I fucking said foo = 8 ten lines above, than being a programmer with good enough tools to help me when I'm stupid.

There is an implicit assertion here that not having a debugger makes you a smarter programmer. Would removing other feedback print statements, unit tests and the repl help too? Why even execute code at all, just print it out and think hard.

Thinking hard doesn't work if part of your mental model is wrong. Being human (unfortunately) my mental model is often wrong and a quick glance at a stack trace does more to enable correct thinking than hours of working the problem out by hand with pen and paper.

The whole point of having a computer is that it can do some of the thinking for you.

> maybe he ought to start with something other than web programming

The goal is not for Anon to learn to program, it's to build lunch_app. The web sucks but it is the only good distribution platform we have. Either we use that or we build something easier on top of it.

* The goal is not for Anon to learn to program, it's to build lunch_app.

If anon wants to build software then learning how to should not be optional. The obvious alternative is to buy software, just as we buy most other things (cars, books, bread, etc).

My point is that its not enough to point to an easier environment if that environment does not allow them to accomplish the original goal. 'Sure, just go away and study a completely different ecosystem for several months' is not an appropriate learning curve for a simple task.

I guess my point is that creating software is not a simple task.

Simplifying the parts you mention does nothing to simplify the important things in software development, which, as VB has demonstrated, leads to two things: crappy software and people who know how to create a barely working program, but not how to structure it to be maintainable or to prevent data loss in somewhat unexpected situations.

> Visual Basic was phenomenally successful as an end-user programming tool.

Was? So what happenened, where did it go, why isn't it still filing the role you are aiming at?

Another example, held in somewhat better regard by programmers, of a phenomenally successful tool that let end-users build working programs, is Hypercard. Which definitely isn't around anymore.

Looking at the prior art of tools that have succeeded at letting end-users build programs is probably a good bet. What they did right, what they did wrong, why they don't exist anymore even though they were succesful, if they don't exist anymore.

> Was? So what happenened, where did it go, why isn't it still filing the role you are aiming at?

The last proper version of Visual Basic was released in 1998 -- Microsoft discontinued it in favor of VB.NET. If they continued to develop VB, I think it would still be a heavily used platform.

Hypercard is not the same thing. People didn't build real applications in Hypercard -- it was a toy. I spent a lot of time playing with Hypercard and it was very VB-like when programming for it but the UI was more like Flash. It's best use was making simple games.

The real problem with simple tools is that they solve simple problems. Once you have to start doing something outside of the tools range, you're stuck. This happens with every tool but more complex tools have a bigger range.

> People didn't build real applications in Hypercard.

Remarkably, they did! Wikipedia:

>A number of commercial software products were created in HyperCard, most notably the original version of the interactive game narrative Myst, the Voyager Company's Expanded Books, and multimedia CD-ROMs of Beethoven's Ninth Symphony CD-ROM, the Beatles' A Hard Day's Night, and the Voyager MacBeth.

It's true that Hypercard was not an ideal platform for development. Apparently, Myst developers had trouble making the high resolution images load quickly on machines of the time.

As I said, Hypercard was like Flash -- so it's not surprising that all these commercial products are sort of point-n-click multimedia projects. But you couldn't, for example, build an email application in Hypercard. You could, however, build one in Visual Basic.

Many people/companies built Hypercard applications that talked to backend databases. It wouldn't be difficult to build an email application.

Sure, you could build an email application in HyperCard. I'm pretty sure there actually were several. TCP/IP was available via HyperCard plugins or extensions or whatever they were called. HyperCard's card/stack data model would work fine for email. Making the list view would kind of suck, as lists always did in HyperCard, but you could use a plugin like WindowScript.

I wrote several commercial software products in HyperCard and SuperCard back then. It provided awesome tooling leverage. I worked on an EDI application that was written in SuperCard (with the EDI processing running in a C plugin) which was working, but later the company's president decided we needed to re-write the whole thing in C++ with the MacApp framework (http://en.wikipedia.org/wiki/MacApp) because C++ was a "real" programming language. This was the cfront era -- build times were long, debugging meant manually de-mangling names, etc. Crazy slow. We spent a long time on it but the schedule slipped long past the end of my time there.

Had no idea that the original version of Myst was Hypercard, but it makes sense in hindsight.

The reason HyperCard couldn't support applications that advanced wasn't due to its concept or scope, but that Apple never really supported it well, and the last real major update to it came in 1990, before the internet exploded. HyperCard actually had decent support for AppleTalk networking (I wrote some network games and a network email client/server system in HyperCard as a kid, all solely over AppleTalk). I'm convinced that if Apple kept supporting HyperCard that it would support TCP/IP and anything else required to be competitive.

The common knowledge about why Apple never supported HyperCard whole-heartedly is that they didn't know what it was - it wasn't a programming environment, it wasn't a database, it wasn't a paint program, it wasn't a multimedia environment. Yet it was still all of the above.

FWIW, a lot of people were very angry when Microsoft pulled Visual Basic and replaced it with "Visual Fred" (aka, Visual Basic .NET).

> So what happenened

MS killed it in favour of a 'real' programming language. See eg http://msdn.microsoft.com/en-us/magazine/jj133828.aspx

> Looking at the prior art of tools that have succeeded at letting end-users build programs is probably a good bet

Indeed. We have a 50 odd page google doc full of exactly that :)

Why do you think MS killed and stopped supporting such a successful project? (I am not being rhetorical or sarcastic!)

The article I linked explains it pretty well. A small but vocal minority of users wanted more power. MS gave it to them in VB.NET, and with that power came complexity. The huge majority of people who just wanted to do simple CRUD stuff stuck with VB6 (which is supported to this day).

Link to that google doc? Would love to read it.

> Was? So what happenened, where did it go, why isn't it still filing the role you are aiming at?

VB6 still runs an astounding amount of apps that are used by millions of people every day.

I know you're not implying anything, but, there's several hundred billions of dollars done in the US annually with just VB6, excel macros, and shitty Access DBs. It's incredible, really.

I worked for a Fortune 500 company that still relies heavily on VB6 programs for some of its branches.

Another humungous company (think 10.000 employees) has its core system in VB6 (and it works REALLY WELL).

The company I work for has > 80% of its programs in VB6.

It didn't go anywhere. I'm not sure what the spiritual successor is, though.

I hope my comparison with Visual Basic didn't come out as derogatory. Because, at the risk of no sane programming talking to me ever again, I will admit that I really do admire it! It wasn't built in a period that would have allowed it to live on today (not without significant effort from Microsoft...) but hey, it was nice for its time.

I made this lengthy introduction because I really like LightTable. What you folks are doing is impressive! I'm not particularly fond of the tech stack behind it but it's the first significant, organized kick in the tools' butt I've seen for many years now that isn't confined to academia.

> There is an implicit assertion here that not having a debugger makes you a smarter programmer.

The implicit assumption I'm making if that I were a smart (and careful!) enough programmer, I wouldn't need a debugger :-).

So yeah, if I had the choice between being so good that I don't need a debugger and having a really, really good debugger, I'd pick the former any time. Sadly, that's not... quite a choice, so I'll grumpily let gdb tag along.

> The whole point of having a computer is that it can do some of the thinking for you.

This is something to be enthusiastic about, certainly! I was glancing over Moseley and Marks' paper (I read it a while ago, but not too carefully) and, now that I think about it, the mantra about writing simple programs is held in highest esteem by communities that traditionally had to struggle with their tools (Unix, because 1970s, and more recently the web folks, because they were building complex web applications long before JavaScript had anything that would even resemble a reasonable debugger). Much of this was done so as to eschew the limitations of the platforms themselves. PDP-11/20 had 64 KB of addressable RAM; the fact that you could write and compile programs on it alone was remarkable. Letting the computer take some of the thinking load was quite certainly impossible.

I'm still debating with myself whether allowing the computer to do some of the thinking for me would be good for the quality of my programs. I'm a very clingy father :).

> The goal is not for Anon to learn to program, it's to build lunch_app.

I don't wish to sound rude, but this sounds like a lump made by marketing types :-). Is lunch_app a program? If so, Anon has to learn at least the basics of programming, much like someone who builds a desk has to learn at least the basics of carpentry or someone who wants to learn to play a musical instrument has to learn at least the basics of music.

More helpful tools will definitely help Anon, but there are things no tools will, or should, do for him: he will still have to understand how web applications are served, how they are hosted on the same server along with other applications and so on. Not knowing these things will be a lot more detrimental to everyone using his application than spending a couple of weeks learning all this boring stuff will be to Anon.

> I don't wish to sound rude, but this sounds like a lump made by marketing types :-). Is lunch_app a program? If so, Anon has to learn at least the basics of programming, much like someone who builds a desk has to learn at least the basics of carpentry or someone who wants to learn to play a musical instrument has to learn at least the basics of music.

Isn't this the destiny of most software☨, and another step along the craft to utility curve [0]? Software of this type is needed to do a chore, and honestly if the tools are in place, it could be re-created in an afternoon rather than extended when the time comes.

I would use the analogy of printing; we used to need a typesetter and go to a printer; now we select a font, type and hit 'print' and - if the damned printer on our desk cooperates - we get our work back. Websites are reaching this level of utility; there's no reason simple web apps or desktop software won't too.

[0] This graph http://1.bp.blogspot.com/-lrxB3ntRzKQ/U05ClSIXwGI/AAAAAAAAEy... taken from http://blog.gardeviance.org/2012/06/pioneers-settlers-and-to.... The graph matches what I see in the web industry; his blog is a fun read

☨ Because thinking of all software ever written, including scripting for automation, the really complicated and large and needing well-engineered software is far less common than we sometimes think, and not what someone like Anon is considering building with these tools.

> Software of this type is needed to do a chore, and honestly if the tools are in place, it could be re-created in an afternoon rather than extended when the time comes.

I agree that it's supposed to do a chore, so it should be as easy to churn it out as possible. However, if we were to follow your printer analogy, the proper way to do that with a printer would be to just build a machine that makes it very easy to create a custom printer that types out exactly what you want it to type out! What we have instead is a very general typesetter that can be convinced to print a particular picture (by feeding it an automatically-generated program in e.g. PS!).

The application in question is pretty much a simple CRUD that generates a report in the end so that whoever is ordering knows what to order and from where. IMHO, just like printing doesn't require the user to know PS, this should not involve programming. It's not that programming should be easy, it's just that the CRUD model is so easily understood and so orthogonal that building one should not require anything more than a wizard asking you about the data it should collect.

Or, to use the desk analogy: if you just want a desk, you generally don't learn the basics of carpentry; you just go to IKEA (or a furniture store).

(Which is a great thing for carpenters and furniture store owners. We're those people in the analogy, so why get uppity about people needing our services eventually?)

Actually, the very basics of carpentry are the things you use to put together the IKEA desks!

No, seriously. IKEA didn't have a store open around here when I was a kid; some of the less pretentious furniture in my parents' house was made by my father, and since I was a kid, he had no choice but to need my assistance. Putting wooden things together with screws based on drawings was literally the first thing I learned about carpentry, just before sawing and all that.

The fact that it's so easy should be no surprise. Patching together some stuff in BASIC is exactly as easy.

Yeah, it's fucking disgusting that today's "personal computers" are such impersonally cluttered, patchy, uselessly complex designs that even experienced programmers can barely wield them.

I think what we are missing is smarter tools. I think that's the argument here. What we really need is the ability to say something like "I want a data structure D holding objects of type Q, with an ordering using this key-accessor(A) called Alpha, and an index with this key-accessor(B) called Beta" and then later in the code say "I want to observe D for the following conditions ..." and let the compiler figure out how to store that data in memory, how to most efficiently build a data structure to trigger those conditions, etc. (Instead of saying List<> or IList<> or writing a custom data structure, etc).

And then if you really care (and this is where the non-text/denser ASTs come in; as well as a deeper understanding of computer science and programming) you can add annotations to your description of the data structure like "I want reverse traversal of ordering Alpha to be O(n)", annotations could even cause a compile error like, "I want to insert via index Beta in O(1) time, with an O(n) in order traversal time of Beta". And even fine grained annotations like "For index Beta use a red-black tree" or more specifically "To implement index Beta use module 'foo.bar.baz' by 'FancyBeansCorp', using class 'AwesomeEnterpriseIndex'"

Let the tools do the heavy lifting! Let data structure and systems programming experts write data structure implementation strategies. Hell, I bet you could build a marketplace of code just for optimized data structures, if it was easy to swap them in and out. Heck those sorts of dependencies should be at the company level, without even touching the project in question (a company level code policy to use a specific library of data structure implementation strategies, regardless of project being build (unless the project or user overrides it)). Let alone a market place of implementations for algorithms, general code patterns (conditionals, whiles), AI, networking, UI, etc.

Compilers should take a long time. A very long time. Sure we can short circuit them for debugging, and just use the best workable code we got in a few seconds, so we can test (and besides, with constant compilation, it should have already solved most of the problems reasonably well by the time you click run). Our compilers should be smart enough they can change their output for different machines not just at an assembly optimization level (better intrinsic, etc), but for different cache sizes, for different memory performance characteristics (changing between intensive memory data algorithm to intensive processor algorithm) as necessary. And yes, I expect it to be deterministic for every platform.

Absolutely. This is actually part of our plan. We're writing mid-level-ish specifications in a logic language and then compiling them into an incremental dataflow network. Our current prototype just uses the same implementation for each rule but later we intend to be able to pick and choose different data structures and join algorithms per index/rule.

I have a lot to say about separating specification from implementation and trying to better capture programmer intent (eg http://www.freelists.org/post/luajit/Ramblings-on-languages-...) but the blog post was already pretty long.

To me it seems the only way to smarter tools is static analysis, but there's only so much one can do in a dynamic language.

There seems to be this dissonance between dynamic languages, which tend to be popular with new programmers for a variety of reasons, and better tools enabled by static analysis of (usually) static langues.

The static languages seem to present this "hump" that a lot of people are averse to, and if we really want these smarter tools, we're going to have to find a way to get programmers to use more static techniques; we need to get them over that "hump", possibly then these smarter tools would ease things in the long run, but doesn't have quite the same immediate-gratification aspect of dynamic languages due to the up-front effort.

You can do a lot of static analysis of dynamic languages. Check what PyLint/PyFlakes does for example

It's probably not the dynamic language that is the problem, but its constructs.

There is also so much one can do with a static language.

Dynamic languages are preferred by those who don't want the conservative verbose static type system to get in the way of writing code. You can design a static type system that is less verbose (e.g. via more inference), but then it can become more conservative (H&M's inability to deal very well with semi-unification in the form of subtyping and assignment). That is the "hump."

Smarter "more magical" compilers are a nice idea in theory, but hard to realize in practice. There are even limits to the kind of analysis we can do dynamically, but they are a bit less constrained then what we can do statically.

So, basically you want a uniform way of defining data structures that can use different kind of indices on different fields. Let me add a few more desiderata to your list:

- It would be cool if these data structures could be transparently saved to disk, because data usually lives longer than code;

- It would be cool if they could be used by multiple applications, with some sort of guarantee that each application sees a consistent view of the data at any particular time;

- It would also be cool to have a declarative language for accessing several of these data structures at once, "joining" them on the values of certain fields and dynamically picking the best indices to use. Since you want a marketplace of implementations, it would make sense to standardize that language, so different data structure libraries can support it. A good name for it would be Structured Query Language, or SQL for short.

Tl;dr: You are completely right that network databases (object graphs and special-purpose data structures) are inferior to relational databases (declarative general-purpose data structures that separate querying and indexing, and allow a host of other general-purpose functionality). That idea has the potential to transform the whole software industry and lead to billion-dollar profits, as it has amply demonstrated since the 1970s. But since today's programmers are a forgetful lot, perhaps you could rebrand the idea and sell it to them as "Big Data 2.0" or something.

I understand how SQL ties into what I want, but there are a few points you might have overlooked for what I wanted:

* Micro-optimizations. The implementations should be granular, not monolithic like SQL servers (let alone the fact that optimized SQL rarely translates between the implementations, as most optimization commands are not cross platform).

* Embeddable DSL, this one is honestly a problem with programming languages in general, but I've yet to see a static programming language which can tell me when my query is malformed at compile time (or hell, even provide decent syntax highlighting). Let alone importable, custom, extensions to SQL, or embedding lazy functions within the data structure.

* Intent vs Exact. The ability to describe partial intent, rather than exactly how to organize, the data (e.g. describing the operations expected to be executed across it, rather than the indexes to build). As well as having information available from the compiler about how the data structure is used (so that without the explicit declarations, the intent can still be optimized for).

SQL is a fine direction to approach the big picture problem from, but it isn't anywhere near the goal I care about. I'm not interested in dealing with big data. I'm just tired of having to know the difference between arcane implementation names for ordered sequences [1], I want to be able to describe what I want my ordered sequence to do, and let the compiler figure out the best implementation available.

[1] http://www.programcreek.com/2013/03/arraylist-vs-linkedlist-...

I would argue that tools solve specific category's of problems well. Photoshop allows non programmers to do all sorts of wonderful things, perhaps not send the first panda to the moon but hey you can fake it reasonably well.

The core issue is programming does not really focus on any one problem which is why there are no great general solutions. People forget that just because you call something a database does not mean SQL and noSQL solve interchangeable problems there just local maxima. For all lisps power and simplicity people don't actually use it to write a lot of embodied code.

PS: It might be helpful to consider to note in general case compression does not actually work. But, 7zip exists because we don't live in a generic world we live in this one. So, don't hand people a steam shovel and ask them to bake you a cake.