Why do we need modules at all? (2011)

Eliminating modules and namespaces is one of the problems we solved early on in the Burpit project. Modules and namespaces just add unnecessary cruft to the program text, are still prone to collisions, and are not globally unique -- hacks which do not actually solve the problem in the manner of Java's com.xyz.whatever.SomeClass notation notwithstanding.

In the Burpit programming language, Poon, all functions are named according to a globally unique, 1024-bit hash of their representation in the Kock VM, which has been transliterated into a human-pronounceable string of CVC units, such as %pip-yod-rat-bag-hej-mig-wab-lik-sac-duf-top-kek. This is a bit hard to remember, but for the low introductory price of 100 BTC you may reserve a "quark", or block of 32-bit function names such as %pip-yod-rat. We've eliminated the need for tools like git too: all publicly available Poon functions live in the Burpit network itself in a distributed blockchain database indexed by these hash names.

This means you have to give up human-meaningful function names, but in practice this has been shown not to be a problem: the Burpit developers have no problem remembering what %zoq-fot-pik does, and you can search on a function's description in the global database.

There are some functions, called traps in Burpit, which are the equivalent of system calls and do not follow this convention. For example, the trap %sporc creates what is roughly the Burpit equivalent of a process (called a kite in Burpit).

A major reason for modules in OO languages is to manage mutability: There's some hidden state, and some conventions surrounding that state, that the module tries to abstract away from us. But the price is very high: We have to make sure our executables are built with the right versions of modules, and avoid incompatibilities: What we know as dependency hell.

But why do we need dependency hell at all? For instance, in Scala, why do I have to do gymnastics to use a modern version of Shapeless along with spray-routing? Because the JVM is not happy with multiple, incompatible versions of shapeless in the same classloader.

Under Joe's plan, all of that disappears. It's just a question of how much we lose in exchange for sidestepping a problem like this. Given that any libraries I write nowadays tend to have just a few hundred lines of production code, I am very tempted.

Instead, the idiom has emerged to prefix every function name with its "package name". I'd take a deep look at the discussions in the Emacs Lisp community, in order to see the matter from the other side.

Modules can then be views and I think they would still be useful. I think the coincidence we currently have where modules are files makes it seem like mutual exclusivity and tree-like hierarchy is a defining feature of modules. We can throw that out. I think a lot of Joe Armstrong's complaints go away now.

Modules are still useful for human purposes. They are useful units of responsibility, accountability, maintenance, learning, and presentation. In fact, these facets of modules are enhanced if modules can intersect as views into a codebase: Overlap and dependence of code can be modeled as intersections and joins of modules-as-views. That these can be computed from the structure of code rather than implicitly understood on top of the directory hierarchy of a file system, or worse, forcibly mangled to fit into a tree-like hierarchy like files is a tantalizing possibility.

There would be a function name rush in open source (similar to domain name rush) to claim cool, short function names.

Erlang is simple-minded about this: frameworks get a special place in Erlang, and EDSLs are hard to come by since you can't declare, or override syntactic structures.

Taking modules out (in any language), you remove the programmer's option to semantically structure what they deliver into a holistic "thing". This may be a good thing: you can avoid monolithic modules.

So I think both options would be nice.

Hierarchical namespaces can certainly be overdone, but they do allow for quick, local decision-making.

I always thought about it like about dimensions.

1. "Logic/test dimension" I can have functions doing actual logic and for tests. Do I put it in one module (to make it easy to change both if I need to) or in separate module (to make business logic more easy to follow).

2. "Data structure dimension" I can have functions operating on different data structures like lists and sets.

3. "Operating dimension" I can have functions operating on "Enumerables". Do I put map in the Lists module or Enumberable? I need to know, where someone else put it when debugging. Database operating on meta data would solve that problem.

4. There can be metadata for time/space complexity, so I can easily make tradeoffs between functions that do the same thing in different ways.

5. "prod/stg/dev dimension" is another one. Maybe I want to use completely different logging mechanism for stg and prod, because I pay per logline...

6. "Quality dimension" could show, what is the code coverage for the function or if it follows naming conventions/practices.

7. "Popularity dimension" could show, how often is given function referenced, which would show most important functions and where to focus on optimizing.

Some of this problems are solved in different ways. IDEs can jump from function usage to its implementation. If you follow conventions, you can jump to a test code for given function. In Elixir, you can jump to protocol implementation. I can use inversion of control for switching implementations between environments. Those problems would have single solution, if there was a central database for functions.

There are many, many more dimensions and even relations between them.

It does not seem possible in mainstream languages where the OOP dogma tells you to obfuscate your sets, lists and maps and turn them into non-reusable classes.

Anyway, you end up with packages named "map" and "filter" or "toString" that export functions of the same name and are all very easy to require and consume/understand/test. Then on top of these you can build your libraries or apps.

"the only place where modules seem useful is to hide a letrec." - JS already has this via require you can easily choose what you want to export.

"The unique names bit is interesting - is this a good idea. Qualified names (ie names like xxx:foo/2) or (a.b.c.foo/2) sounds like a good idea but but when I'm programming I have to invent the xxx or the a.b.c which is very difficult." - I agree 100% with Joe here, with a few developers it works beautifully without any namespacing for the sake of it, I can attest to this. You have a canonical "isTextNode" function for example which you can use when you want to know if a dom node is a text node, its simple and its much easier than remembering the whole namespacing of things like org.example.subrouter.MyRidicClass...

So in summary I really like this idea, I've really liked it for a while now in js, I just don't know how well it would work at a larger scale where "anyone?" (or who exactly) can plop funcs into the DB.

Then realize that the mini-trend not so long ago to make NPM modules a single function was a freaking nightmare, scratch the whole idea, and remember that modules were invented for a reason.

If two, seemingly unrelated modules contain an equivalent function, does that not suggest the need for a third module? And if that module happens to only contain that one function, aren't we just at the same point as the suggestion of uniquely naming all functions, yet not stuck with it as the only way to invoke functions?

But seriously, this thread title needs a "(2011)" on it. It's been 4 years. If someone thinks this is a good idea, just freaking try it already and see what happens. Lead by example.

Is there a progamming language where one can construct/negotiate contexts in a clever way? Something more powerful than Python's 'import' or Javascript's 'with'?

Less constructively: what is described gives me flashbacks to pre-OO PHP4, endlessly looking up function docs. Maybe there is a a better way, but the proposed solution doesn't seem to provide fundamentally different/improved paradigms from a Big List Of Functions.

I think that Joe ignores the major rationale for modularization.

Modules provide the way to cut out the fragment of a complex project and apply local reasoning to it.

In other words a module is the way to "divide and conquer" the complexity that crosses the border of the single mind's capacity.

As such a module is not about code compilation, distribution and upgrades. It's about human cognitive abilities.

If you keep going down that path, eventually even smart people run into a cognitive wall, because the sad reality is that programming is defined by difficult to parse textual code in a bunch of text files.

Good luck trying to break through that wall.

http://node-machine.org