Hacker News Reader

Comment by jacquesm | Hacker News Reader

Python: interactive glue language between high performance C libraries. It's funny how we have converged on this solution, I use it daily so I'm really not complaining but I really didn't see this as one of the logical outcomes of the various programming language streams.

But when you think about it it is kind of logical: inner loops and low level code tend to be fairly static and are often re-used so it pays of to write them in a language that maximizes for that use-case, but structure can be executed much slower and re-use is relatively low when going from one application to the next so it pays of to write that in another language.

So we get this split roughly halfway down the application stack where we switch from interactive and interpreted high level language to compiled low level language.

It's a very interesting optimum.

View on HN

It's actually a fairly common pattern: think high performance filters glued together with the shell. Or "steering" of supercomputer codes via Tcl, Squeak Smalltalk doing high performance multi-media (at the time) using a few well-placed primitives and an otherwise run-of-the-mill bytecode interpreter.

A very similar split goes right through the middle of Objective-C. Although it is compiled, the "Objective" part always felt pretty close to interactive and is pretty high-level, and C is, well, C. Same concept, less mismatch between the parts.

One of Objective-Smalltalk's goals is to invert the Objective-C trade-off: make the interactive, high-level part primary, while still allowing a smooth transition to the low-level parts without resorting to making that transition automatic by either a heldencompiler or super-smart JIT.

Yes, it is a very interesting optimum, though I think it is one that can be improved on.

I've long heralded this as some of the most attractive things about interpreted languages like Python and friends, and what really draws me to them in the first place.

Take parsing a config file - it doesn't need to be fast unless it's truly in your hot path and you're doing millions of them. I like not having to worry about memory management when it comes to them. On the other hand if I have a hot path/slow operation having the ability to compile a library to do the heavy lifting and gluing it together in a higher-level language means that you spend a lot less time fighting your language and a lot more time solving your problems.

Fighting your language to get performance is a special trait of C/C++. Languages like D, Rust or Go are trying to fix this.

I use this pattern a lot. For example, in https://github.com/kovidgoyal/kitty where the UI layer is in python and the performance critical code is in C. I find this pattern gives me an optimum in performance + maintainability.

You REALLY didn't know about http://kitty.9bis.net/ before starting this?

Yep. Similar to how we used to write the high performance code in ASM and the business logic in C. Now we write the high performance code in C (or equiv. low-level language) and the high performance code in a slow but expressive dynamic language. My favorite is writing high-level code in some kind of Lisp (previously Clojure, lately Racket) because REPL-driven development makes it incredibly fast to iterate and experiment and then cement/polish it into production code all while having the same state in memory.

This is one of the reasons I enjoy using Nim; it's that exact split, but within the same language. It's quite a mind-bender to begin with, but damn is it powerful once you've got the hang of it.

For example, today I was doing some homomorphic encryption experiments, and leveraged the GNU GMP library for some integer calculations. Crossing that "boundary" in Nim was the easiest thing I've ever done. Take a high level "ref" type (`Int`) which is actually `type Int = ref mpz_t`: This means I can write nice high-level procs over the top of the low-level GMP types, but also just de-reference any `Int` to pass it to the "real" GMP functions as needed!

I'd love to see other languages explore this, too, as I think it's really powerful. It's my biggest gripe with Node.js: FFI is a bit "ick" the last time I tried it!

Also known as Ousterhout's Dichotomy

http://graydon2.dreamwidth.org/3186.html

And it's super great when your C extension has a memory leak and you can't figure out where in Python you are creating circular references... happened to me once. Two days of my life...

But yes as long as you write good C, it's a pretty awesome pattern.

That's why people are starting writting the Python extension in rust.

You can have memory leaks in Rust as well. Especially if you are interfacing with a program that expects a C interface (i.e. CPython).

This architecture has a name: http://wiki.c2.com/?AlternateHardAndSoftLayers

Reply on news.ycombinator.com