Why MIT switched from Scheme to Python
wisdomandwonder.com
wisdomandwonder.com
Python (and, frankly, a number of the scripting languages-turned-mainstream) combines this clarity of computer science with a practicality that Scheme never had. If you can convey 95% of the basic ideas in Python, and you can also open the door to learning how to deal with 3rd party code, you'd be a fool not to. It was never about programming purity anyway, so there's no reason to mourn the passing of Scheme. It's progress.
But you cannot.
Some of the ideas in SICP can nearly be conveyed in Python (like lexical scoping, closures, data types, streams). You'll use built-in language features in Python (classes, generators) instead of implementing the functionality. I think that implementing these things is important if you want to really understand them.
An example, cons, car and cdr from nothing at all:
(define (cons a b) (lambda (f) (f a b)))
(define (car p) (p (lambda (a b) a)))
(define (cdr p) (p (lambda (a b) b)))
Other ideas, like interpreters, require a lot of effort. Try writing a metacircular interpreter!
def cons(a, b):
return lambda f: f(a, b)
def car(p):
return p(lambda a,b: a)
def cdr(p):
return p(lambda a,b: b)
Aside from lack of tail safety and performance issues with writing scheme-ish code I don't see why you couldn't implement those things in Python. The main difference is that typically in a real world situation you wouldn't because there's a library or a language feature
This may not be important if you teach programming to people who want use programming languages, but it is important for people who want to advance the state of the art in programming (i.e. CS students).
Maybe it's better if you teach Scheme in a seperate class class for CS students.
It took me a while to find a Lisp that I liked, and it certainly wasn't a decision I was prepared to make while I was still trying to learn the basics. (I eventually settled on Chicken Scheme.)
foo = (1,2,3)
bar = (4,5,6)
print cons(foo, bar)
print car(foo)
<function <lambda> at 0x00B3F430>
Traceback (most recent call last):
File "./test.py", line 14, in <module>
print car(foo)
File "./test.py", line 5, in car
return p(lambda a,b: a)
TypeError: 'tuple' object is not callableTry instead:
car(cons(1,2))
-> 1
cdr(cons(1,2))
-> 2
car(cons('a', cons(2, 3)))
-> 'a'I ask myself, "how would I teach my son to program?", and for all my copious academic and practical experience, "start with re-implementing car and cdr from lambda functions" isn't even remotely in the running. Sure, I'd introduce such things earlier than most people and SICP is still firmly in the running, but for that to be an introduction in the general case? Not so sure.
There's definitely an argument to be made that robotics is in many ways a much more realistic starting point. While some people may find the idea that we should actually start out by making a case to the students that their major is cool and important and gives them the ability to affect things in the real world vaguely repulsive, I'd say that it's an idea at least worth considering. The student that can leap from lambda, car, and cdr, to realizing that right in the introductory course is so exceptional that you're not going to hold them back anyhow.
On the other hand that was when I was 9. By 12 I was learning Lisp, and by college age I'd written my own little Lisp compiler.
I've always had this image of MIT students that doesn't quite mesh with them needing a gentle introduction to programming, but maybe that's not really the case?
If it leads to more (non-CS) students taking a programming class, I'd call it a big win. Other fields can benefit greatly from programming skills. I'd almost go as far as calling it a new form of literacy.
6.001 from my experience does not in any way imply a student's focus on computer programming or software development. I've heard horror stories about people in completely unrelated majors ending up in that class and having their brains thoroughly melted. In my personal opinion, Python will be considerably more humane in that department.
I'd assume most, if not all, computer science students have, but there's still a difference from learning to bang out some code in high school that does neat things, vs. starting back at the beginning and approaching it as, well, computer science.
The way my university covered it was doing Pascal for the intro courses (this was the mid-90s) and then in third-semester "Programming Languages" we worked mostly in Scheme implementing language fundamentals. That seemed to be a reasonable balance.
Reminds me of this post that came up on HN not too long ago:
Rather than complaining about how everyone else is a bonehead for not using Lisp, write some libraries to make it useful for more things in the real world.
Also, Python does pretty well for real-time control. At Anybots we use it to run walking balance feedback loops at 100 Hz in the same process as a GUI and logging and it never misses a tick. PLT Scheme (used on HN) frequently pauses for 10+ seconds to GC. While there are theoretical concurrent GC systems for Lisp, none of them seem usable in real implementations.
What made them popular was the design of the core languages. I started using Ruby because it was faster to type and easier to remember how to do things than Perl, not because it had more libraries, which it didn't.
The problem with Scheme is that the core language isn't terse enough, and so it gives the impression that it will be hard to do things. And that's correct. Scheme requires much more typing and is harder to remember than Ruby.
Consider:
Ruby:
h = {}
h[:foo] = "hi"
h[:foo] #=> "hi"
h[:bar] #=> nil
(define h (make-hash-table))
(hash-table-set! h 'foo "hi")
(hash-table-ref h 'foo) ;; "hi"
(hash-table-ref/default h 'bar #f) ;; #f
This isn't a Lisp problem, it's a Scheme one. A Lisp with a better core and enough users would get lots of libraries eventually, just like any language.
I think Scheme is good for teaching though, and would rather learn it in a course than Python.
user> (def h {:foo "hi"})
#'user/h
user> (h :foo)
"hi"
user> (h :bar)
nil
You say libraries aren't the issue, but the discussion is about a blog pst that said: "And why Python, then? Well, said Sussman, it probably just had a library already implemented for the robotics interface, that was all."
I'm sure your experience is grounded in a general reality. But the post you disagreed with was about a specific case: i.e. the one described in the blog post.
The problem, at MIT and elsewhere, is that hackers don't like hacking Scheme enough. Python has more libraries than Scheme now, but that's not the main thing that made more people use it. People use it because they prefer the language. This is something that Sussman and others at MIT should be trying to understand, but it doesn't strike me that they are trying to understand it. They're just saying "Python has a better robotics library -- oh well!" as if this were something completely random that they have no control over. That doesn't seem like the sort of attitude that produced Scheme in the first place.
My point of contention with the comment I replied to is the idea that fixing the problem is a question of writing more libraries for Scheme. Writing or interfacing to libraries for numerics, 3D graphics, etc. is a trivial task compared to making a language hackers love.
Now, maybe hackers' distaste for Scheme is Scheme's fault, or maybe it's hackers'; but in any case, the issue is with the core language, not the libraries. And based on my personal experience with Scheme I would say that enough of the fault lies on Scheme's side that it's worth fixing.
It is no accident that it survived almost exclusively in non-saltmine sanctuaries like MIT, where industry herdthink held less sway. In the linked article, Sussman directly admits that he is unable or unwilling to fight back against the intellectual decay that has consumed non-academic programming.
He has decided to "go with the flow" - namely, the flow of industrial sewage into students' heads.
Scheme (PLT)
(require scheme/foreign)
(unsafe!)
(define my-c-lib (ffi-lib "libmylib")) ;; loads libmylib.so
(define my-c-func (get-ffi-obj 'my_c_func my-c-lib (_fun _long -> _long)))
;; The C code does not need to take Scheme into account--
;; Existing C libraries can be interfaced with Scheme
;; with a trivial amount of effort.
#include <Python.h>
#include <netinet/in.h>
static PyObject *
py_htonl(PyObject *self, PyObject *args) {
long my_arg;
if(!PyArgParseTuple(args, "l", &my_arg))
return NULL;
return PyBuildValue("l", htonl(my_arg));
}
static PyMethodDef methods[] = {
{"htonl", py_htonl, METH_VARARGS,
"Demonstrate Python's C interface\n"}
};
static char module_doc[] = "Demonstration";
PyMODINIT_FUNC
initfunc(void) {
PyInitModule3("mylib", methods, module_doc);
}Thank you. Cannot stress this enough. (Making said libraries portable helps, too!)
Most of the 3D stuff is performance-critical, so I typically prototype in Python and then convert to C++, which for graphics code isn't too painful. The Boost.Python interface makes it pretty easy.
Also, the Cairo graphics interface is wonderful for drawing charts, animated robot stick figures, and many other visualizations. It has a full Postscript rendering model with alpha blending.
It looks like PLT Scheme has an OpenGL binding too, though I haven't seen it used. It looks very low-level, for instance it has separate methods named: gl.Vertex3d gl.Vertex3dv gl.Vertex3f gl.Vertex3fv gl.Vertex3i gl.Vertex3iv gl.Vertex3s gl.Vertex3sv
Also, if you haven't used it yet, check out Py++. It takes a lot of the pain out of hand-authoring a Boost.Python code file. I'm using Py++ to automatically generate Boost.Python bindings for the C++ library FCollada. (FCollada is a library for accessing 3D modeling data exported into the .dae COLLADA format.)
Thanks for the tip about Cairo.
Also, glVertex3d, 3dv, 3f, etc are part of the standard GL API. The 'd' suffix indicates 'double', 'f' indicates 'float', 'i' indicates 'int', 's' indicates 'short', and the 'v' suffix indicates that the function accepts a pointer rather than 3 separate components passed by value. Of course, the glVertex* API has been deprecated because VBOs are far more efficient.
Just curious again, why do you use OpenGL for Anybots? Do you create simulations before building new robot prototypes?
We use OpenGL to show real-time renderings of the robot, derived from the gyros and joint angle sensors. We have an analysis tool that shows, either in real time or post-run analysis, renderings synchronized with high-speed video of the robot and scrolling strip charts of every important variable over time. I always see much more looking at slow-motion videos & renderings afterwards than looking at the real robot.
OpenGL rendering also turned out to be useful for extracting data from CAD models. Our CAD software (SolidWorks) can export STL files (a list of polygons) and we need to extract center of mass and moment of inertia and such. Doing that without visualizing it is a nightmare.
We have done dynamics simulations (using ODE), but they haven't had much predictive value.
Are you computing center of mass / moment of inertia yourself, or using a 3rd party library to do it? PhysX is free, and I believe it can do those calculations for you fairly automatically. (In PhysX, a 'scene' is composed of 'actors', and each actor is described by a set of shape descriptors. A shape descriptor can represent a box, a sphere, a capsule, a convex hull, etc. So a laptop actor could crudely be approximated with two box shape descriptors (one for the screen, one for the body) connected by a joint. Given those shape descriptors, the library calculates the actor's center of mass / moment of inertia / etc. If your CAD software simply exports a list of polygons, you'll need to decompose those into a set of convex hulls using convex hull decomposition techniques: http://codesuppository.blogspot.com/2006/04/approximate-conv... ) You also might have better luck running dynamics simulations with PhysX than with ODE. It comes with a large number of code examples to get you started: http://developer.nvidia.com/object/physx_downloads.html
http://www.cs.berkeley.edu/~jfc/mirtich/massProps.html
Maybe 80 LOC. I'm not sure why anything more complex is needed.
PhysX does seem promising.
The other reason I mentioned convex hull computation is because PhysX can only simulate dynamic actors that are composed of boxes, spheres, capsules, and convex hulls, IIRC. Arbitrary polygon meshes aren't supported for dynamic actors, because they are very difficult to simulate in realtime. So if you wanted to use PhysX for dynamics simulations, you would probably need to decompose your model into a set of convex hulls.
Good luck -- if you need any assistance with PhysX, feel free to contact me at shawnpresser@gmail.com.
(I'm just brainstorming ideas in the rest of this post; I'm not sure whether they'll work, but it's a fun problem to think about.)
Perhaps it would be useful to write your own simulator that incorporates data about how the components of the robot behave in the real world, to get a more accurate simulation. By that I mean, the problem is that the robot model has so much complexity that it's difficult to simulate accurately. So instead of creating a mathematical model for how a compressed-air actuator behaves, for example, gather real-world data about how it actually behaves in the robot, then incorporate that data into the simulator. I mean, if the robot is walking slowly on a flat surface, then its individual components behave in the same ways each time the robot takes a step forward, right? Therefore it seems like the behavior of the whole robot can be reproduced and predicted in a simulator, since the behavior of each component is known (under controlled conditions, like when the robot is walking slowly). Brute force the problem by using large amounts of real-world data, is what I'm getting at.
"Brute force the problem by using large amounts of real-world data, is what I'm getting at."
Absolutely, that is Trevor's viewpoint, and I agree. Turns out to be somewhat difficult in practice. It's hard to even know what all the parameters should be, much less do all the work necessary to tune them. And all the while, the robot itself is changing. Joints get loose; servos get replaced. It's a constant battle.
There is a feeling among hard-core robotics folks that this level of simulation is either impossible or not worth the trouble. I tend to disagree. I think we got pretty far with the Anybots simulator; we did manage to evolve a neural network that balances on the simulator, and also balances the real robot. It's definitely a 'hard' problem to get a simulator good enough to be usefully predictive.
Depending on exactly what you're trying to do, of course...
People defending what's right can say one should do it because it's right. But people defending something they know is basically wrong can't say one should do it because it's wrong. So they look around for another word, and "practical" is usually the first they find.
I think "practical" often means choosing your battles. Sussman's choosing to fight the battle of explaining some syntax to his students rather than the battle of implementing a real-time robot control library from scratch in a language where no implementation supports real-time very well. Sun-Tzu would approve.
It's not evident from Andy Wingo's blog post, but just to clarify, Gerry didn't choose Python over Scheme: a committee did. Gerry said that when they had their revelation about the state of computer programming in 1997, he and Hal Abelson decided to stop teaching 6.001, and they eventually stepped away from the course altogether. (I believe Gerry returned to teach the class for the final term it was offered, but he didn't get into that last night; I could be wrong.)
Gerry went on to say that 6.001 essentially languished until sometime around 2007, when a committee got together and redesigned the introductory EECS curriculum to be what is now 6.0[012]. He didn't say whether he was on that committee, but I got the impression he was not. He appeared to be speculating when he offered the reason why Python was chosen. It was an offhand remark, something to the effect of (and I'm paraphrasing here), "And why Python? I suppose there were already good libraries for controlling these robots."
It sounded like Gerry explicitly chose not to fight any battles at all on this issue, and simply walked away resigned to the fact that computer programming, and engineering in general, are much different now than they were when he, Hal and Julie wrote SICP.
Two more things:
* I don't recall Gerry using the word "practical" even once while reciting his short anecdote.
* Gerry continues to teach with Scheme, not Python, in 6.945, his advanced topics course.
He wanted to do a course about robots, and according to him, the interface to the robots was Python, which was otherwise an adequate language.
If you want to make a case to the contrary, it should be rooted in the specifics of the situation.
(define (deriv f)
(define (f-prime x)
(/ (- (f (+ x dx)) (f x))
dx))
f-prime)
dx = 0.00001
def deriv(f):
return lambda x: \
(f(x + dx) - f(x))/dx
In [11]: def cube(x):
....: return x*x*x
....:
In [12]: deriv(cube)(5)
Out[12]: 75.000149996640175
I love scheme (alot more than I do python), but the only example that seems "impossible" to do is to implement a large subset of <language> in <language>.
You could obviously implement, a scheme-ish lisp in python, but there are alot of advantages to the host and guest languages being similar (or so that berkley professor would have me believe) that you would miss out on.
(You would also need a scheme expression reader in python... which I'm just itching to write now ... )
the example is from sicp: http://tinyurl.com/cubk49
I think that realizing that universities can't effectively teach programming should be at the root of choosing which programming languages are used in computer science instruction. In college you get loaded up with the right concepts that will later facilitate you as a programmer, and I think languages chosen should be languages that are geared to that end.
Perhaps the most instructive thing I've seen for student programmers, and which pays to boot, is Google's Summer of Code. That throws students in with a bunch of brutally honest and usually seasoned hackers and makes them fare for their own for a summer.
Also, 6.01 actually assumes that you have some programming experience already. When I TAed, at least, we assumed working knowledge of basic Python from week 1 and told people to go read a Python tutorial/book if they didn't have such. If you don't know how to program at all, you're supposed to take 6.00 first, which is the real 'Introduction to Programming' course. Except that most people skip it because they already know how to program or think they can learn on their own (and because it's kind of seen as a remedial course, which is especially unfortunate for people who really don't know how to program already--6.01 is pretty harsh if you've never programmed before).
I think a lot of people don't understand the goals of this course. 6.01 is not meant to be a crash-course in writing code. (6.005, know as "java deathlab" is for that.) Likewise, it's not designed to reveal the deep inner workings of computers. Instead, it's mean to be an introduction in how to /think/ with computers. It's conceptually more high-level, and often focuses on the "why" rather than the "what." We learn how to approach problems and look at the best ways to solve them using logic and code.
Python is visually clear, has great documentation, is well maintained, and has an active community of developers. I don't understand the outrage in switching. I didn't learn to drive in an obscure European sportscar; I learned in my mom's minivan. Python is like my mom's minivan, but with better documentation.
I don't think the caliber of CS grads coming out of MIT in the next few years will change because they know Python, although it'd be an interesting thing to look at.
#
Here's the course website, if you'd like to look closer: http://mit.edu/6.01/mercurial/spring09/www/index.html
I can confirm this is true. The Python Robotics module (Pyro) inspired MIT's internal library, SoaR, which was developed at first by a highly motivated TA.
What does SoaR stand for? Snakes on a Robot, of course.
I believe the main reason why robotics has progressed so slowly in the last 30 years is that getting the damn hardware and firmware working together takes 90% of the available time and students barely get beyond that. So libraries that abstract away most of that and enable plug-in real-time control modules are going to give the field a huge kick in the pants.
http://wingolog.org/archives/2009/03/24/international-lisp-c...
Sussman's rationale for the change in 6.001's focus and language is more interesting to me than anything else in the 'international lisp conference -- day 2' post, and is hidden near the bottom of that post. So, the pull quote blog item adds finding/filtering value for me, which is the opposite of spam. Neither the blogger nor submitter deserves the implication they are a spammer.
Let me just tell you.. I've never seen so many bearded people in one place!
So I suppose beards either mean lit gurus or lisp hackers.
I had the privilege of taking an SICP-based course at the U(C). The brain-stretching is critical, isn't it? Yes, there's a lot of head-slamming that goes into modern development, but that's not really the same thing.
Python may not stuff its academic credentials in your face, but I think it has one of the better balances between what you can use it to teach, and what you must teach to use it. (Referencing the common and IMHO justified complaint that the simplest possible Java program requires rather a lot of stuff to be taught before you fully comprehend it.)
Teach Scheme later... or perhaps, fully distinguish between academic and practical by starting with Python to get you going then transitioning to an ML or Haskell later. (That approach has a lot of appeal to me personally, and has the benefit of the student coming out with at least one useful-in-the-industry language (Python) without compromising academic utility very much.)
Lua has much more Scheme influence than Python does (while being otherwise pretty similar). Its design assumes proficiency with C, however, so, while very practical, it's probably not as good a language for people new to programming.
Now it means having immutable values, tail recursion, easy lambdas, powerful type systems, and so on. There's a clear historical progression from past to present, but there's still been changes in what the term means over time which means that different people mean different things by "functional programming" if they don't clarify.
I believe both of these to be true. From my point of view the new 6.01 sacrifises depth for breadth. This is part of much bigger change of the ciriculum in MIT. That's why I hurried to take all nice classes before they get destroyed.
P.S. I didn't know python untill 2 weeks ago. But I learned in need very fast - definitely having background in different programming languages helps a lot.
The glorious MIT of the 1980s and prior appears to be dead. Erased, in fact, without a trace. Consider the following:
http://mitpress.mit.edu/catalog/browse/browse.asp?btype=2
These are the latest releases from MIT press. Among them you will find nothing remotely like SICP, but plenty of postmodernist/related pablum.
"You have to do basic science on your libraries to see how they work, trying out different inputs and seeing how the code reacts." Is this not an atrocity, to be fought to the last bullet? Is this not an argument for declaring all existing software obsolete and starting from scratch? Why are we - even luminaries like Sussman - so willing to give up on the future without a fight?
As argued in some of PG's essays, in order for creativity to bear fruit it is not enough for a given individual to possess a creative mind. The culture has to be hospitable to creativity. And there are processes which come close to stamping out creativity without any overt or deliberate suppression involved. Specifically, a transition to a medium where one cannot get reliable, understandable "small parts" having predictable behavior.
Take note: http://xkcd.com/298/
Programmers have indeed created a universe where "science doesn't work." Learning where the permanent bugs and workarounds are inside a phonebook-length API teaches you nothing. It is anti-knowledge. Where your mind could have instead held something lastingly useful or truly beautiful, there is now garbage.
There is no longer a "what you see is all there is" Commodore 64 for kids to play with:
No one is working on, or sees the need for a replacement. This is just as well, considering as one would have to start from bare silicon - and avoid the use of design tools built on the decaying crud of the past. Only a raving lunatic would dare contemplate it...
Outside of programming: Linus Pauling developed an interest in chemistry by experimenting incessantly throughout his childhood. Today, he would have been imprisoned:
http://memepunks.blogspot.com/2006/06/americas-war-on-scienc...
The new generation will have no Linus Pauling, unless the Third World - where one might still purchase a beaker without a license - supplies us with one. Likewise, we will see very few truly creative programmers. What creativity can there be when your medium is shit mixed with sawdust?
I think it's an acceptance of reality. There is a phase change when systems become too complex for a single human to ever understand them in their entirety. Dealing with systems at that level is a different beast than dealing with simpler systems where "what you see is all there is".
"What creativity can there be when your medium is shit mixed with sawdust?"
The same sort of creativity artists possess, who work with media that are idiosyncratic. It's a different mindset. I see the reddit-gen programmers talk about things on a completely meta-level. To them, MySql is the hardware. Is it a bad thing? Not necessarily. Some of them hopefully will dig down the stack and be the low-level heroes. But that should (and probably can) only be a small percentage.
Now I guess one could argue that those sorts of heroes are what MIT is supposed to produce, but as has been mentioned, this course is not just for CS students. So the real question is, can the CS heroes of tomorrow survive an introductory course in Python? Well, consider that they have probably been modding games since 10, hacking PhP at 12, realizing at 14 they need to learn a 'real' language (C#, Ruby, Python), by 15 I bet they've abstracted out the fact that languages are not all that important; they're just the tip of the iceberg of complexity.
So yes, I think they will survive.
All of technological progress consists of refusing to roll over and "accept reality" - instead, attacking it and modifying it to suit our needs.
>There is a phase change when systems become too complex for a single human to ever understand them in their entirety.
This is how innovation dies. The disease is preventable:
http://naggum.no/erik/complexity.html
> To them, MySql is the hardware. Is it a bad thing?
Yes! It is. A programmer's finite mental capacity for dealing with complexity is consumed by an ever-greater load of accidental complexity - inconsistencies in the medium learning about which teaches you nothing. They are holes in reality. The thing that made modern technology possible is that physics, chemistry, etc. do not have "bugs." A correctly set up experiment will yield the same results when done a week from now, in Australia, etc. And in the end, there are unifying principles governing it, which will explain everything that happened in a way that teaches you something permanent. A medium with "holes", like a buggy programming system, is a universe where "science does not work":
> Some of them hopefully will dig down the stack
Wading in sewage.
> languages are not all that important
This statement is true when the languages are all equally crippled - as they increasingly are, through their reliance on an ever-sinking foundation of 35-year-old rotting OS.
[edit:] I believe the xkcd strip can be read two ways. Unknowable complexity equals magic (a riff on Arthur C Clarke). I strongly believe that exactly what's been wrong about software is our clinging to the idea that it is reducible and manageable. We don't think that way when we work in the physical world; we do rely on science, but it's empirical science, with statistically useful models at appropriate levels of abstraction. We don't rely on quantum mechanics to build bridges.
I would be, if I actually believed that it is possible given current technology. However, I believe that the problem is technological rather than fundamental, and that we can arrive at a solution. Let me give you an example of a problem that was once thought to be fundamental and inescapable yet ceased to be viewed as such. Before the advent of high-quality structural steel, no building (save a pyramid) could exceed five stories or so in height. Towards the end of the 19th century, metallurgy advanced, and the limit vanished. I believe that we are facing a similar situation with knowledge. The conventional wisdom is that the era of "encyclopedic intellects" (and now, human-comprehensible software) is gone forever due to the brain's limited capacity for handling complexity. Yet I believe that the computer could do for mental power what steam engines did for muscle power. I am not alone in this belief - here are some pointers to recommended reading, by notables such as Doug Engelbart:
http://en.wikipedia.org/wiki/Intelligence_amplification
Additionally: why do you assume that any system built to date necessarily must contain 100M LOC? Does this not in itself suggest that we are doing something wrong?
> clinging to the idea that it is reducible and manageable
All of technological civilization rests on the assumption that the universe is ultimately "reducible" to knowable facts and "manageable" in the sense of being amenable to manipulation (or at least, understanding). The alternative is the capricious and anthropomorphic universe imagined by primitive tribes, run according to the whim of the gods rather than physical law.
> We don't rely on quantum mechanics to build bridges
I must disagree here. QM is entirely relevant to the science of material strength, which is ultimately a branch of metallurgy - arguably a discipline of applied chemistry. In any legitimate science, there really are "turtles all the way down." And when there is a discontinuity in the hierarchy of understanding - say, the situation in physics with the need to reconcile general relativity and QM - it is viewed as a situation to be urgently dealt with, rather than a dismal fate to accept and live with forever.
A computer, unlike the planet's atmosphere, is a fully deterministic system. It is an epic surrender to stupidity if we start treating programming the way we treat weather prediction - as something that is not actually expected to reliably work in any reasonable sense of the word, or to obey any solid mathematical principles. It is a disaster, to be fought.
In the practical world of software development today, we do indeed have tens of millions of lines of code 'under' the application level most of the time. Even on Linux, where every one of those lines of code is theoretically accessible as source, the practical fact is nobody has the resources to read, comprehend, and manage all of that code during the lifetime of a typical project. Therefore, you are necessarily in a position where you have to deal with most of that codebase as black boxes, which have a statistical likelihood of doing what they are intended to do. You have to program defensively, hoping for the best but assuming the worst. You have to rely on your pattern recognition skills and your intuition to guide you. You have to treat the chaotic complexity you sit on top of like a surfer riding a wave. You cannot afford to take a fully reductionist approach. It's just not possible anymore, full stop.
The exception to this would be a small, embedded system with a minimal operating core and limited memory. But wait -- that system today is almost sure to be networked up to the bigger, chaotic systems that populate the metaverse. So again, at the level of communications, you have to treat the components you communicate with as black boxes.
If you disagree, give me a feasible scenario in which you could in fact take control of the situation in a reductionist fashion. I propose that any such scenario is a toy problem, that will not scale to real-world deployment.
The cat is already out of the bag.
The modern computer is the first time mankind has built a system which is actually capable of perfection. You are more likely to have a spontaneous heart attack and die at any given moment than the CPU is to perform a multiplication incorrectly. All of the "hairiness" which your computer's behavior exhibits is purely a result of programmer stupidity. It is theoretically possible for the entire stack to work exactly as expected, and for each element to seem as inevitable and obvious-in-retrospect as Maxwell's equations.
It is possible to build a system where the black boxes aren't fully black and can be forced to state their actual function on a whim. It is possible to expose the entire state of the machine to programmer manipulation and comprehension in real time:
http://groups.google.com/group/comp.lang.lisp/msg/b2c0190dc3...
http://www.sts.tu-harburg.de/~r.f.moeller/symbolics-info/dev...
>The cat is already out of the bag.
The Lisp Machines existed, and were capable of everything modern, "hairy" systems are capable of, plus much more. You cannot un-create them. They are proof that the entire software-engineering universe is currently "smoking crack" - and I sincerely wish that it would stop doing so.
"It is theoretically possible for the entire stack to work exactly as expected, and for each element to seem as inevitable and obvious-in-retrospect as Maxwell's equations."
Yes, and it is theoretically possible according to the laws of physics for broken glasses to spontaneously reassemble. But they don't.
This is basically an issue of entropy.
Lucene was mentioned earlier - it, at least, is very good.
I have the same impression about the numerical computation and machine learning libs too.
However, the JVM libraries are well-supported and generally can be trusted to work. For a lot of the wheels I'd rather not reimplement, they're quite useful. I don't think that the Clojure of 2020 will be as reliant on Java, but for now, the JVM libraries are very useful.
I asked: "why is this Java component in the Windows client app?" The answer was that the server was also implemented in Java and the SOAP libraries were compatible, whereas Microsoft's and Java's were not (at that time).
I nearly choked: they chose a "standard" like SOAP and then re-standardized on a particular library (client and server components of it) "for compatibility."
So to bring this digression back to relevance, I'll say this: if you already have Java code around, it will be easiest to talk to it from Java. And since Java is the new COBOL, there is a bunch of Java floating around. (but if you need a Java client to talk to a Java server using a "standard" protocol then something is seriously awry somewhere)
I'm not saying they are all are high quality, or relevant to your task, or that there are no C++ or Python equivalents - just that there is a high quantity of Java projects. Why not have a quick search? Maybe there is a project there that will convince you.
AWT is about the most prejudiced example imaginable - it was rushed out, disowned by its authors, and quickly replaced.
I think one of the key aspects of the java ecosystem is the idea of free reference-implementation libraries, but you can pay ridiculous amount of money for corporate versions with better implementations. For example, the BEA Systems implementation of Java itself and related tools seems pretty good, well-documented and so on.
I find Java itself and its OO style is pretty horrible for writing prototype code that can be changed quickly and arbitrarily - so, probably, any revolutionary ideas that require that flexibility of interfaces will be hard to iterate in java, and therefore we probably won't find many such projects written in java.
Ahh yes -- replaced with Swing. Do I need to spell it out?
{
"x": [
{
"y": "a",
"z": 23,
"q": [
54,
32,
45
]
}
],
"r": 43
}The other difference is that there are JSON parsers and generators for more languages and they're not programmable.
{ ("x" ( { ("y" "a") ("z" 23) ("q" ( 54 32 45 )) } )) ("r" 43) }
SBCL:
(defun f (x y) (let ((z (gethash x :a))) (+ z y))
Clojure
(defn f [x y] (let [z (get x :a)] (+ x y)))
Having syntax for maps is a huge win. Like ML's pattern matching, it's one of those things that changes your coding style entirely for the better, in a way that you wouldn't predict just by looking at the feature.
The only think I miss from CL when I work in Clojure is keyword arguments to functions. There, the Clojure way is a bit worse:
(defun f (x &key y z) (list x y z)) (f 2 :z 3) => (2 nil 3)
(defn f [x {y :y z :z}] (list x y z)) (f 2 {:z 3}) => (2 nil 3)
vs
> (defn f [x y] (let [z (get x :a)] (+ x y)))
The CL defn of f is a function that is called with two arguments. That function is called like "(f a b)" If the Clojure definition is comparable, that is, the call looks like "(f a b)", why are []s used in the definition and the let?
> This slight decrease in regularity makes a lot of functionality more visible.
What functionality? x,y aren't part of a vector and neither is z. (x,y may come from a vector in the caller, but I'll assume that the defn works if they don't, so that shouldn't matter.)
(I actually prefer Python's style of using indentation to indicate blocks, but I recognize generating such code is harder than code with tokens.)
def indent(code): return code.replace('\n', '\n ')
The code I generate has no indents and no newlines. Not having to keep track of these makes life simpler. To make my generated code more legible, I just run indent on it.
Again, I see no point in making efforts to generate clean looking code when utilities like indent exist.
surely this is unambiguous?
x:
y: "a",
z: "23",
q: 54,
32,
45
,
r: 43(defn sum-vec [vec] (let [vec-len (count vec)] (loop [idx 0 acc 0] (if (< idx vec-len) (recur (inc idx) (+ acc (vec idx))) acc))))
If I have Javascript then I already have mobile code.
If I'm using json in a situation where I don't have a Javascript interpreter, then I probably don't have a Lisp interpreter around either.
My experience is that it is as little as and possibly less initial development and maintenance as a custom flat-file format or a really robust csv parser.
Lucene is sort of Doug Cutting's Java version of Text Database (TDB), which he and Jan Pedersen developed at Xerox PARC, and which, to complete the circle, was written in Common Lisp (see "An Object-Oriented Architecture for Text Retrieval").
It's a great language, but there is nothing to gain here by claiming it somehow is mature because Lisp is 50.