Hacker News Reader

Comment by urxvtcd | Hacker News Reader

Any results from your research? That seems interesting.

Research is a bit too strong a word for it but I'll summarize my conclusions.

Most people I talked with seemed used to and very happy with the Python model: modules are distinct namespaces for symbols and their referenced objects. Pretty much everyone I talked to had no difficulty understanding modules, it's always the package management that makes things complex no matter which language it is.

Something I spent quite a bit of time thinking about was whether it was worth reifying that model into the language as first class objects. In other words, should import be a special keyword that makes the interpreter magically bind symbols or a function which returns a regular everyday normal object? At the end of the day, Python's modules are just dictionaries, just like Javascript modules. Should I make that fact apparent or hide it behind language keywords? Interestingly, both languages went in the opposite directions: Python went from special import syntax to allowing you to access modules as a dictionary, while Javascript went from a require function that returns an object to an import statement. I ultimately chose a somewhat weird mix of both, powered by lisp's flexibility.

I also tried to figure out how compiled languages approached modules. So I dug up literature on Modula, Modula-2 and Oberon and tried to figure out how they represented modules. This ended up having a significant influence in my design in the form of isolated modules with export control. Each module contains its own table of symbols and their references. Importing is just setting a local symbol to the value of the other module's symbol, and only symbols in the exports list can be imported. I also liked the qualified and unqualified names: added the option to prefix the module name to the local symbol.

I also thought a lot about how to map modules to the file system. The idea of program folders from Windows has been an inspiration for a long time now. The idea is if the module itself is reachable then all of its submodules are also reachable by the loader. I also think it's important that no file escapes the package directory. Python and Ruby frequently have thing.{rb,py} scripts and thing/ directories side-by-side, I sought to eliminate that for the module's root directory only which results in a main file like thing/thing.{rb,py}.

To enable module-oriented development, I've found the most important feature of the modules and packaging system is editable libraries. Like pip's editable installs and npm link. When I develop a project, I often end up with several supporting libraries. Languages should support linking these local versions to the main project so they can be developed simultaneously.

Have you looked into the "first class modules" system used by ocaml? AFAIK it's the only language of its maturity to use something like it. It's very powerful without adding mandatory complexity. But because basically no one comes to it with prior experience in a similar approach, it's hard to gauge how powerful, or when the added complexity of using it fully is actually worth it.

I have! I didn't reflect much on it though because I'm focused on creating a dynamic language. Perhaps I did not understand it?

Another interesting concept I ran into while researching modules is parameterized modules. Essentially, modules that take their dependencies as arguments. The modular equivalent to dependency injection I suppose. Instead of a module importing by symbol a specific library as a dependency and some package manager resolving it to actual files in the load path later on, the programmer explicitly loads the library and passes it to the module as an argument.

Instead of the symbolic imports we're all used to:

  (import lib); lib imports lib2 internally

Module importing becomes analogous to function calls which construct an instance of the module given its dependencies:

  (import (lib2))
  (import (lib lib2))

This is really elegant and more or less reifies package management into the language. However, it presents serious ergonomics issues because it forces the programmer to deal with all these package management and library loading details. The truth is we want to sweep all that ugly stuff under the rug, not deal with it every single time we import a module.

The main benefit, the loose coupling that stems from the ability to substitute dependencies without having to change the importing module, can be accomplished in a declarative manner via package managers. Arch Linux packages for example may have a "provides" variable which allows multiple packages to implement an interface of sorts and be used interchangeably to satisfy dependencies. So I think parameterized modules imposed significant costs for little benefit.

Thanks for response. Some of these thoughts floated around in my head as well, though less ordered and well articulated.

One thought regarding the quantification element. I’ve grown to appreciate the uniformity of Go’s solution, where you always import an entire namespace, optionally aliasing it to avoid conflicts.

This small restriction makes some naming decisions simpler (for example sticking to config.Parse, and not the stuttering config.ParseConfig). Additionally it sprinkles the namespaces over the code so you get at least _some_ feel for them.

On another note, on some occasions I have thought to myself that it’s easy for the import section to get to little scrutiny on reviews, and wondered if there’s something that could make us wonder „should x depend on y?” more often.

> I’ve grown to appreciate the uniformity of Go’s solution, where you always import an entire namespace, optionally aliasing it to avoid conflicts.

That's a very good solution in general which makes everything uniform and consistent. It's only due to my personal tastes that I didn't implement it that way.

I'm obsessed with symbol management. I'm so obsessed with this I wrote my language in freestanding C just so there would be no libc and compiler cruft in the resulting ELF binary. I'd rather deal with complexity than see weird doubly underscored stuff in readelf output.

Names are everything in computer science. I have some kind of psychological need to have clean names. For that I need clean namespaces that I can shape to my will. So I absolutely wanted the ability to import only the symbols I needed in order to minimize the pollution of the namespaces. I support just importing everything as a convenience but I personally never use that feature. I also made sure I had the ability to rename imported symbols to anything I wanted just in case other programmers aren't as obsessed with names as I am.

I went so far with this I implemented basic control flow as a library of lisp macros. There are no reserved keywords or special cases, they're just normal functions that get imported like all the others. Thus they can be renamed or avoided entirely. In fact I made it so only two symbols are present in every namespace: import and export. And those can be overridden too after the programmer is done with them.

I probably have more than few screws loose or something. Go's approach is totally reasonable. Instead of importing N symbols from a module, it imports 1 symbol and nests all N symbols under it, and if there's a clash you only need to rename the module prefix. It's nice and creates single points of truth.

... Now that I think about it, the only reason I didn't implement it the Go way is I didn't want to add special syntax for nested symbols to my language. In other words, in my language "config.Parse" is a single symbol instead of a "config" + "Parse" pair. I feel like it just wouldn't be lisp anymore if I added syntax to decompose the former into the latter.

> sticking to config.Parse, and not the stuttering config.ParseConfig

Yes. I find the stuttering repetition in your latter example to be profoundly irritating and a symptom of a bad modules system. In my language I tried to prevent that by making it easy to add or remove module prefixes to the imported symbols. I also made it easy to rename symbols for good measure just in case people did it anyway.

Reply on news.ycombinator.com