Hacker News Reader

Comment by dpark | Hacker News Reader

This screams “toy projects” to me. Separation of concerns is a hard-won insight. We call code without this “spaghetti” because it eventually becomes nearly impossible trace or to reason about. Even if you have all your engineers working all over the whole codebase, the codebase itself still needs separation of concerns to simplify reasoning about behavior. This is considered a good engineering practice because the alternative is that you need to hold the whole system in your head to understand any behaviors. I’ve seen codebases like this. It’s not pretty. Use of abstractions to simplify cognitive overhead is not a weakness.

The idea that projects should take source copies instead of library dependencies is just kind of nuts, at least for large libraries. Yes, the proliferation of tiny libraries is obnoxious and adds a ton of pointless overhead. But for large libraries, taking a source copy is absolutely crazy. Imagine taking a source clone of Chromium into your project because you need a rendering engine. Yes, you can probably cut a bunch of code out as a result, but you will also now need to deal with security concerns and incorporation of new features until the end of time. And this is now way more expensive and you can’t just do a straightforward merge because you changed the code. 5 line package? Sure, just copy the code in (assuming licensing is in the clear). 500 dev-year codebase? Cloning that code is by far not a win.

View on HN

The thing I tell people when I’m critiquing their tests is that nobody gives a shit about your tests until they break, at which point they’re interacting with your code when they’re already in a bad mood.

They will associate that feeling with you. So do you want to make their bad day worse by writing abstruse code? Or should they be simple?

I know I’m “done” with a test when I know why it’s red from what’s in the console. I shouldn’t even have to look in the source code. Until then, my tests are a work in progress.

We could apply this philosophy a lot more widely.

Totally agreed!

Here's a prototype from a few years ago where I tried to make this easier: https://github.com/akkartik/mu1#readme (read just the first few paragraphs)

I still think the full answer lies in this direction.

"Separation of concerns is a hard-won insight."

Absolutely. I'm arguing for separating just concerns, without entangling them with considerations of people.

It's certainly reasonable to consider my projects toy. I consider them research:

* https://github.com/akkartik/mu

* https://github.com/akkartik/teliva

"The idea that projects should take source copies instead of library dependencies is just kind of nuts..."

The idea that projects should take copies seems about symmetric to me with taking pointers. Call by value vs call by reference. We just haven't had 50 years of tooling to support copies. Where would we be by now if we had devoted equal resources to both branches?

"...at least for large libraries."

How are these large libraries going for ya? Log4j wasn't exactly a shining example of the human race at its best. We're trying to run before we can walk.

    I'm arguing for separating just concerns, without entangling them with considerations of people.

Conway’s Law says you’re going to be disappointed a lot. These impedance mismatches lead to throughput problems and worse.

Can you elaborate on this comment? Based on your comments elsewhere I think we're likely to be largely in agreement.

I tend to have a love/hate relationship with Conway's Law, spending half my time thinking about how to design stuff assuming it, and half my time trying to design stuff (unsuccessfully) to make it go away.

A lot of pain comes from taking a “concern” that doesn’t align with the org chart, making simple tasks take weeks to accomplish because I need to get my boss to tell your boss to do it anyway or we don’t make payroll. People tend to get precious about things they own. See also the legal department or the financial department thinking they are the company, as if they had any revenue without all of those “cost centers” or legal liabilities.

Yeah, I agree with this. There's a larger context here of capitalism evolving firms because they're more efficient, even though they dilute the price-communication value of markets to some extent. As a consequence your boss and my boss are talking about something that might affect millions of people. That results in an effect akin to regulatory capture. So one way to view my case against division of labor is as a case for pushing intelligence to the edges, as close as possible to the owner of the computer running the software.

On this larger level my suggestions are not actually that radical. Capitalism has had a long trend to push intelligence down to the edges. Kings used to use hierarchy to push decisions down. Blitzkrieg was largely about creating an organization that pushes decision-making down. And we've been getting better at it. I think we (i.e. society) can be even better at it by pushing decision-making out of the org and towards the people using it.

> Absolutely. I'm arguing for separating just concerns, without entangling them with considerations of people.

The separation of people is largely an artifact of scale. At some point it doesn’t make sense to have everyone touching the whole code base. It’s untenable to have, e.g., everyone who works on Windows be an expert in every part of the code. It’s also generally unnecessary because in a large code base features are often nearly completely contained to one area of code.

Some teams are really dysfunctional and won’t let people ever do work outside “their” area. This is absolutely an antipattern.

> The idea that projects should take copies seems about symmetric to me with taking pointers. Call by value vs call by reference. We just haven't had 50 years of tooling to support copies. Where would we be by now if we had devoted equal resources to both branches?

We have probably invested dev-millennia into managing copies. This is exactly what source control does. This is not a new area of investment. Merging is a giant pain in the ass and very possibly always will be. Accepting merge pain better come with some huge benefits.

> How are these large libraries going for ya? Log4j wasn't exactly a shining example of the human race at its best. We're trying to run before we can walk.

Can you clarify what you see as the alternative? Implementing everything from scratch seems absurd and so costly that there’s no point in considering this an actual option. Plus there’s no particular reason to believe that you wouldn’t care your own security bugs. Taking a source drop wouldn’t protect you if you happened to take one with the bug. And you’re forever bound to that version unless you want to pay a very high cost to reintegrate a new one. What viable option do you see that I don’t? (Picking on log4j right now is easy but are you seriously implementing your own SSH, for example?)

For the record, these large libraries are going great. I’ve built numerous products that simply would not have ever been created without, say, SQL. The cost would simply have been too high. I’ve built others with libraries that the project outgrew, but it was still a big win to start with the library.

> It’s untenable to have

Absolutely this. There's a wide ranging of programming activities that are domain-agnostic, but most of what programmers do requires deeper knowledge in a specific domain to be useful.

I question the need for scale in 90% of the places where the tech industry has cargo-culted it. Clearly I'm still failing to articulate this. Perhaps https://news.ycombinator.com/item?id=30019146#30040616 will help triangulate on what I mean.

> Can you clarify what you see as the alternative? Implementing everything from scratch seems absurd and so costly that there’s no point in considering this an actual option.

Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context. I have a day job in tech and have to deal with some cognitive dissonance every day between my day job and my open source research. The one thing I have found valuable to take to my scale-obsessed tech job is to constantly be suspicious of dependencies and constantly ask if the operational burdens justify some new feature. Just switching mindset that way from software as asset to software as liability has, I'd like to believe, helped my org's decision-making.

> We have probably invested dev-millennia into managing copies. This is exactly what source control does. This is not a new area of investment. Merging is a giant pain in the ass and very possibly always will be. Accepting merge pain better come with some huge benefits.

Not all copying is the same. We've learned to copy the letter 'e' so well in our writings that we don't even think about it. In this context, even if I made a tool to make copying easier and merges more reliable, that would just cause people to take on more dependencies which defeats the whole point of understanding dependencies. So tooling would be counter-productive in that direction. The direction I want to focus on is: how can we help people understand the software they've copied into their applications? _That_ is the place where I want tooling to focus. Copying is just an implementation detail, a first, imperfect, heuristic coping mechanism for going from the world we have today to the world I want to move to that has 1000x more forks and 1000x more eyeballs looking at source code. You can see some (very toy) efforts in this direction at https://github.com/akkartik/teliva

> It’s untenable to have, e.g., everyone who works on Windows be an expert in every part of the code.

It's frustrating to say one thing in response to counter-argument A and have someone then bring up counter-argument B because I didn't talk about it right there in the response to counter-argument A. I think this is what Plato was talking about when he ranted about the problems with the newfangled technology of writing: https://newlearningonline.com/literacies/chapter-1/socrates-.... I'm not saying everyone needs to be an expert in everything. I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context. My Linux laptop doesn't need to be optimized to the hilt the way Google's server farms do. Using the same scheduling algo or whatever in my laptop imposes real costs on my ability to understand my computer, without giving me the benefits Google gets from the algo.

> I question the need for scale in 90% of the places where the tech industry has cargo-culted it.

Certainly our industry does a lot of premature scaling. Sometimes this is the best bet, because it can be so difficult to scale later. But certainly sometimes it’s a huge waste and creates unnecessary complexity.

> Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context.

Yeah. This again makes sense if you don’t really need the library. If you’re only using some tiny piece of it.

Otherwise this looks a lot like Not Invented Here. Reimplementing huge pieces of functionality is rarely a win. Copying can be better but is again rarely a win. It works for cases like Chromium where a massive team was created to maintain the fork, but in general firing code comes with huge cost.

Reimplementing and forking are generally not value adds. Building software is today about creating/adding value.

Certainly it pays to be suspicious of new dependencies and ask if they are necessary. I’ve seen unnecessary dependencies cause significant pain. I have also seen variants of not invented here waste years.

> I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context.

I don’t understand this criticism. If I write some code, I’m realistically the expert. The person who code reviewed for me is likely the next most knowledgeable. It’s not about wanting to silo expertise. It’s about the reality that certain people will understand certain subsystems better because they’ve worked there. If I need to get something done, I’m going to first try to get the experts to do it, not because someone who typically works in another area can’t, but because it will probably take the “outsider” twice as long. It’s not a statement of what’s right or wrong, just what’s efficient. Certainly I’d knowledge becomes too silo’d, it’s a massive risk. But some siloing can be significantly more efficient, at least on large or complex systems.

> If I write some code, I’m realistically the expert.

Ha, disagree. This is overly general and reductionist.

See, I know the C strcpy() function is a bad idea. In this I know more than Ken Thompson or whatever Unix God did at the time he wrote it. I also know more than everybody who wrote (at the time they wrote) every other library that's on my computer that uses strcpy(). Why the fuck is this still in my system? Division of labor accumulates bad ideas along with good ones.

> Division of labor accumulates bad ideas along with good ones.

Agreed. To take an example from my area of specialization, accessibility, both Windows accessibility APIs (yes, as is often the case with Windows, there's more than one) have serious design flaws IMO. But they still allow disabled people to do many things we would otherwise be unable to do. I worry about what will happen if programmers truly follow your proposal from another subthread: "build/reuse less. Do without." In this case, doing without would make computers much less useful for many people, and possibly erect new barriers to employment, education, and just living one's life.

It's bad now. It might not have been bad when it was originally written. Back when Unix was first being developed, you had machines with 64K or 128K of RAM, which is not much, and because of that, software was much simpler, and such design compromises could be kept in mind. And when C was standardized in 1989, it might have been a bad idea at that time, but the standards committee didn't want to obsolete all the existing code (a major concern at the time). But by all means, get a time machine and plead your case.

As an example of coding for size, I recently wrote a copy of strcpy() with similar semantics. I had to, because of constraints on memory. I wrote an MC6809 (an 8-bit CPU) disassembler in MC6809 assembly code, and the prime consideration was code size. Strings ending with a NUL byte? My GOD! that will waste 80 bytes! Waste a byte on size? That's another 80 (or 160 if you insist on 16 bits for size) bytes wasted! Instead, since all the text is ASCII, set the high bit on the last character. That more than makes up for the bit of oddness in my implementation of strcpy() to deal with the last byte, and helped contribute to the code being 2K in size.

What makes strcpy() a better compromise than strncpy() or strlcpy() with 64K or 128K of RAM?

strncpy() is for a different context, and it won't necessarily NUL terminate the destination [1]. It's not a compromise for strcpy(). strlcopy() will NUL terminate the destination, but it first appeared in 1998, nine years after C was standardized, so again, time machine etc. Should Ken Thompson & co. included strlcpy()? It's an argument one could make, but I'm sure they didn't see a need for it for what they were doing. Oversight? Not even thinking about it? Yes, but I don't blame them for failing to see the future. Should Intel not do speculative execution because it lead to Specter?

Now, assuming strlcpy() existed in the 70s---is that just as efficient as strcpy()? Should strcpy() have always included the size? Again, I think it comes down to the context of the times. Having worked on systems with 64K (or less! My first computer only had 16K of RAM), and with the experience of hindsight, the idea of implementing malloc() is overkill on such a system (and maybe even 128K systems), but I won't say people were stupid for doing so at the time.

[1] I think strcnpy() was there to handle the original Unix filesystem, where a directory was nothing more than a file of 16-byte entries---14 bytes for the filename, and two bytes for the indode.

> Should Ken Thompson & co. included strlcpy()?

I'm getting a little impatient with all this argument in circles. The point is not what someone should have done with hindsight. The point of this little sub-thread is that saying the author is always the expert is oxymoronic. Expertise is often multifaceted and distributed.

The larger point is also that it should be possible to get rid of things once hindsight has found them wanting.

> The point of this little sub-thread is that saying the author is always the expert is oxymoronic.

Writing the code makes me the expert in that code. Meaning how it works. Why it was built the way it was. How it was intended to be extended. Yeah, it might be shitty code and maybe someone with more domain or even just general expertise would be able to do better. But if I write the code, I’m still the expert in that specific code. Someone else can of course roll in and become and expert in that code, too. Expertise in my code is not a bounded resource.

If you took my comment to mean that writing the code makes me the expert on all possible implementations or choices for the solution, or that I am best qualified to decide what the best solution is, I must have communicated poorly.

(Also, this is pedantic, but that’s not what oxymoronic means.)

> If you took my comment to mean that writing the code makes me the expert on all possible implementations or choices for the solution, or that I am best qualified to decide what the best solution is, I must have communicated poorly.

No, I didn't think that. You're all good there.

> Writing the code makes me the expert in that code. Meaning how it works.

Go back and reread the bottom of https://news.ycombinator.com/item?id=30019146#30040731 that started this sub-thread. Does this definition of 'expert' really help in any way? Ok, so some author wrote some code, they're expert in that code. Big whoop. I don't want to use that code. I wrote a shitty simpler version of what you wrote, now I'm an expert, you're an expert. How is this helpful?

I think the only expertise that matters is domain expertise. Software should be trying to communicate domain expertise to anyone reading it. So that anyone who wants to can import it into their brain. On demand.

The “expert in the code” matters in terms of siloing. That was the context where I mentioned it. The engineers who work in an area all the time and know the code best are typically best equipped to make the next changes there. Hence a silo. Maybe I completely misunderstood your comment about late binding experts?

But stepping back for a moment, I’m increasingly confused about what your thesis here is. You’ve clarified that it’s not actually about abstraction. It’s also apparently not about clearly factoring code (though maybe you think that contributes to problems?). What is the actual thesis?

Engineers should be experts in the entire code base?

Code is in general too hard to understand?

Something else?

It feels like we’re discussing mostly tangents and I’ve missed what you intended to say. If I understood your message, I might have a better mental model for the utility of copying code.

Yeah that's a failure on my part. Like I said, I've spent ten years trying to articulate it and spent a few hours in conversation with you, and I still can't seem to get my actual thesis across. I don't think another attempt right now will help where all else before it has failed :) How about we pause for a bit, and we can pick it up again in a few hours. Perhaps even over email or something, because this back and forth is likely not helping anyone else. My email is in my profile.

I’m good with pausing. Maybe we can chat again another time. :)

Have a good evening (or whatever time of day)!

Here's another attempt, a day and a half later. My thesis, if you will.

Before we invented software, the world has gradually refined the ability to depend on lots of people for services. When we buy a shoe we rely on maybe 10,000 distinct groups of people. When we write software we bring the same social model of organization, but all the vulnerabilities and data breaches we keep having show that it doesn't work. https://flak.tedunangst.com/post/features-are-faults-redux enumerates (in better words than mine) several examples where services made subtly different assumptions, and the disconnects make their combination insecure.

Since we don't know how to combine services as reliably in software as in non-software, we need to be much more selective in the services we depend on. A computer or app just can't depend on 10,000 distinct groups of people with any hope of being secure.

What you’re actually saying is that you have the benefit of hindsight. That in no way makes you more on an expert than Ken Thompson who also has the benefit of hindsight.

I would guess that literally thousands of people have implemented an equivalent to strcpy with exactly the same issues (and likely more). Reimplemention does not imply a better implementation.

Decades of experience using strcpy is what taught us it’s a bad idea, not reimplementing it a thousand times.

Maybe it took you decades. One decade sufficed for most of the world.

This is starting to feel like a pissing contest. Go write your software thinking you know more than your users. Me, I constantly need to talk to mine. They're often more expert on the domain than me.

> This is starting to feel like a pissing contest.

Is there some reason it suddenly went from us discussing a difference of opinion to hostility? I have no idea what this is about.

> Go write your software thinking you know more than your users.

That’s a strange nonsequitur. End customers don’t generally tell me how to craft software. When I work with customers, it’s to understand their domain and their problems, not mine.

Again, not sure where this hostility is coming from.

Perhaps I'm coming down with something. I'm uninterested in the sub-thread past https://news.ycombinator.com/item?id=30019146#30042174. It feels like willful misinterpretation, like you forgot what I said 2 exchanges ago when you respond to any single comment.

I've said my piece in a bunch of different ways, over ten years and again ten times to you. Perhaps this is as close as we're going to get to mutual understanding, if you think after all this that I'm trying to one-up Ken Thompson or something.

I think you're using words like "expert" and "strictly worse" way too bluntly. What's better for the author is often not what's better for the user.

> Reimplementing and forking are generally not value adds. Building software is today about creating/adding value.

Check out the writings of Carlota Perez[1]. She distinguishes between two phases in adoption of a technology: installation and deployment. Analogously, reimplementing adds value because you learn to make something better or faster or simpler. And you learn how to make it and can now take that knowledge elsewhere and use it in unanticipated ways.

It blew my mind a few years ago to learn that the Romans had knights: https://en.wikipedia.org/wiki/Cataphract. WTF, I thought they were a medieval thing. It turns out everyone in the Ancient world understood the idea that an armored person on a horse can outfight someone on foot. What was missing in Ancient Rome was the ability to teach people at scale how to fight on horseback. As a result, most cataphracts would dismount before battle. It took literally a thousand years for society to figure out how to educate people at scale on mounted combat. You don't really know how to do something just by building it once, and you don't unlock all its economic potential just by building it once. Societies have to build it over and over again, thousands of times, to really understand its value. And they gain benefits throughout the process.

[1] https://avc.com/2015/02/the-carlota-perez-framework

> Analogously, reimplementing adds value because you learn to make something better or faster or simpler. And you learn how to make it and can now take that knowledge elsewhere and use it in unanticipated ways.

You know, there’s some truth in this. At the same time, there is only truth to the extent that the person reimplementing is actually striving to do better. I would bet that well over 99% of reimplementations are strictly worse than the thing they are mimicking.

I spent a bunch of time implementing a custom windows framework similar to WTL. It was a great learning experience. I would also say that in no way woks it have been useful to anyone for or to appear in an actual product. In most ways it was doubtless inferior to WTL. I used it for toy projects and that’s what it was.

> My Linux laptop doesn't need to be optimized to the hilt the way Google's server farms do. Using the same scheduling algo or whatever in my laptop imposes real costs on my ability to understand my computer, without giving me the benefits Google gets from the algo.

Perhaps I don't fully understand what you're arguing. But here's a scenario, involving a hypothetical developer using similar logic, that often makes me worry. Suppose a developer reasons that the GUI for their application doesn't need to have all of the complexity of, say, a web rendering engine. So they develop their own GUI toolkit that meets their needs and aesthetic. This toolkit lacks some features that are important to some users, but not to this developer, such as accessibility with a screen reader. An application developed using this toolkit becomes popular, and companies start integrating it into their workflows, perhaps even requiring other companies that they work with to use it. Oops, now these companies can't hire an otherwise qualified blind person, or perhaps a formerly productive blind person loses their job [1]. If the original developer had accepted the current division of labor inherent in building high atop the web stack, and yes, the complexity and bloat that come with it, their application might have been fully accessible without them having to deeply understand accessibility.

[1]: This actually happened, though I'm not sure why the application was inaccessible. https://blindaccessjournal.com/2006/02/my-job-lost-due-to-in... https://blindaccessjournal.com/2006/02/torn-from-the-collect... https://blindaccessjournal.com/2006/02/the-cold-equations/

Yeah, that's a really good example. In general, accessibility and internationalization seem like really good reasons to specialize. We can't all know all possible languages.

It’s not about specialization, though. It’s about getting those benefits from standard and existing tooling. You don’t need to be an expert in accessibility to build accessible software (though it helps of course). But you do need to build accessibility support, which is a lot more work when you do it from scratch.

The longer I work in the industry the more I realize the cost of “doing it from scratch”. If you ask a brand new software engineer to estimate the time for a project, they’ll basically give you an estimate for a prototype. A slightly more experienced engineer will include time to get through code review and maybe shake out the egregious bugs. A more advanced one will remember to cost for tests. I have found you have to get a pretty senior engineer if you want them to accurately account for things like telemetry and logging, deployment, live site management, accessibility, localization, etc.

I’ve seen engineers build a “simple” system and then spend 4x as long as they planned to make it actually supportable because they “didn’t need” the existing tooling. So they rewrote it all but worse.

> For the record, these large libraries are going great. I’ve built numerous products that simply would not have ever been created without, say, SQL. The cost would simply have been too high. I’ve built others with libraries that the project outgrew, but it was still a big win to start with the library.

We definitely have great libraries out there. But they're the exception rather than the rule. I consider Redis and Sqlite in particular to be quite exemplary.

SQL I'm more ambivalent about. Or rather, SQL the abstract ideal is great, but I'm more ambivalent about implementations like MySQL or PostgreSQL. As I've hopefully made clear already, I consider just raw speedup for an initial implementation of an app as a poor and incomplete metric for the quality of a library. At best it's a good quality for something you can use in prototypes, not the final version you build after you throw the prototype away. (Because we all agree you should throw the prototype away, right? :) For non-prototypes you also have to take into account the total cost of ownership. I think the ever expanding wavefront of "specialization" in roles like DBAs is a sign that MySQL and PostgreSQL are a bad trade for society. People being deployed to fill in for all the numerous gotchas arising from a crappy interface. MySQL and PostgreSQL make things more efficient in the short term, but they make us less resilient in the long term.

> I think the ever expanding wavefront of "specialization" in roles like DBAs is a sign that MySQL and PostgreSQL are a bad trade for society.

That’s a sign of a maturing field. Does the proliferation of medical specialties indicate a problem? I would argue that it simply arises from the amount of specialized knowledge that one can possess and utilize efficiently.

The existence of SQL implementations allows for faster and more maintainable implementations. The set of engineers who could do what SQL allows without it is quite small. If SQL disappeared tomorrow, a thousand equivalent implementations would appear tomorrow because it delivers real value. Hand rolling queries in a general purpose programming language is a very poorly trade off. I’ve seen this sort of stuff before. It’s always a bug farm and every other engineer asks “why the hell didn’t you use SQL”?

Give it enough time and you’ll see “Redis expert” show up as a specialty.

> Does the proliferation of medical specialties indicate a problem?

Ivan Illich certainly thought so: http://akkartik.name/illich.pdf. And he said a lot of useful stuff in my opinion.

But I don't know enough to have an opinion of medicine. I find it useful in my experience. I wasn't saying the connection between my observation and conclusion is ironclad in all situations. But I do hold the heuristic opinion in the one case I mentioned.

> The set of engineers who could do what SQL allows without it is quite small. If SQL disappeared tomorrow, a thousand equivalent implementations would appear tomorrow because it delivers real value.

How much of this argument is about idealized SQL and how much about MySQL or its equivalents? I was careful to draw a distinction. SQL the language/mindset is incredibly valuable. EF Codd thoroughly deserves his Turing Award.

You don't need to persuade me that MySQL lets us do things faster. What would really persuade me is an argument that it helps us do the right things. I think it makes it easy to scale past our level of competence.

> Ivan Illich certainly thought so

*Shrug*. When I go to the doctor, I love finding someone who does nothing but the one thing I need. Maybe this person is a generally “less useful” doctor, but if I break my ACL I want the surgeon who does nothing but ACLs all day every day to work on mine.

> I think it makes it easy to scale past our level of competence.

This sounds like an amazing win. This is frankly what software is supposed to do.

> Give it enough time and you’ll see “Redis expert” show up as a specialty.

No opinion on what happens to Redis after Antirez left it (http://antirez.com/news/133). But I have high hopes that the versions he created will be fairly timeless and usable even a decade from now.

I wasn’t aware he stepped back. I don’t follow it closely because I don’t use it directly, though I have been very impressed by the way it’s managed to become so ubiquitous so rapidly.

Lisp machines showed the way decades ago, computing took a different path. Here's the system's UDP receive-ip-packet method: https://twitter.com/RainerJoswig/status/1215728406823886854

The copies vs dependencies thing doesn't seem like a useful framework to me -- in order to do anything with software that has dependencies, such dependencies need to be copied in some form that can be integrated into the final build. So the copies are there regardless. Different ecosystems differ on how easy it is to work with them. If all you get is a .so, then even if it's open source you're going to have to find and build your own to make changes. In Java you'll get jars, which may or may not contain java files and class files -- in the past I've unzipped jars, recompiled a single java file to replace a class file, rezipped it back up, and used that jar instead of the upstream jar. Not too bad. Some log4j 'fixes' involved stripping out some class files from jars, again not too bad, just don't accidentally clobber your changes later. Some JS projects will have additional instructions (or a bash script) that after npm downloads everything into the local node_modules, you need to go modify a particular JS file in there. It's easy for the changes to get clobbered later, but..

Lastly in Lisp land, we have quicklisp, so you point to a library dependency like normal and it downloads it locally elsewhere (so it only need be downloaded once and not for each project). When I'm writing a program using library X, and I'm curious how some function works, I can just jump-to-source and it'll take me to that local copy. If I want to change it, I can, I just edit it, and because Lisp has "compile", "compile-file", and "load" all as standard runtime functions, not separate programs, I can compile and load my changes into the running program and immediately test them. Maybe such changes are upstreamable, maybe not. You can maintain the changes in the original files, maybe going so far as a full source copy locally instead of using the ones quicklisp brought down, or just make your own local 'patch' file of redefined functions or extended classes that you load after loading the original and the patch simply recompiles and replaces things you've changed. It's also rather fun to do this to inspect and learn about the Lisp implementation you're using, what it's doing or how it's implementing things, changing stuff to see what happens/fix an edge case bug, whatever.

Part of it I think is supported by a notion Thinking Forth calls a 'lexicon'. See its early section on 'Component Programming' but in short 'lexicon' refers to words (symbols) of a component that are used outside of a component. "A component may also contain definitions written solely to support the externally visible lexicon. We'll call the supporting definitions 'internal' words." Of course, without special effort, in Forth as in Lisp even those internal words are not really inaccessibly internal. In Lisp it's the difference of referring to an exported symbol by namespace:symbol and an un-exported one namespace::other-symbol. The other supporting notion here is a that of globally available "components", which are just a product of decomposition, not some formal thing like separate processes or libraries or what have you. "It's important to understand that a lexicon can be used by any and all of the components at higher levels. Each successive component does not bury its supporting components, as is often the case with layered approaches to design. Instead, each lexicon is free to use all of the commands beneath it. ... An important result of this approach is that the entire application employs a single syntax, which makes it easy to learn and maintain. This is why I use the term 'lexicon' and not 'language.' Languages have unique syntaxes."

I'm not sure I agree about the 'easy to learn and maintain' bit but at least not making things painfully inaccessible can help, especially with re-use. There used to be OS concepts where theoretically if I wanted to play a music file, and knew I had a popular music player application installed, I could just reach in and call a function in that application, rather than writing my own or finding some "do one thing..." library. All programs on the OS were simultaneously programs and DLLs able to be used by other programs.

I don't have many arguments in this space, it's just fun to think about, but good luck on your research. You might enjoy this quote:

"When I see a top-down description of a system or language that has infinite libraries described by layers and layers, all I just see is a morass. I can't get a feel for it. I can't understand how the pieces fit; I can't understand something presented to me that's very complex." --Ken Thompson

Thank you! My ideas are definitely greatly influenced by my time in the Lisp world. Here's the first time I articulated them, back in 2011 (so before the first entry in OP): http://arclanguage.org/item?id=13263.

> in Lisp land, we have quicklisp, so you point to a library dependency like normal and it downloads it locally elsewhere (so it only need be downloaded once and not for each project). When I'm writing a program using library X, and I'm curious how some function works, I can just jump-to-source and it'll take me to that local copy. If I want to change it, I can, I just edit it, and because Lisp has "compile", "compile-file", and "load" all as standard runtime functions, not separate programs, I can compile and load my changes into the running program and immediately test them. Maybe such changes are upstreamable, maybe not. You can maintain the changes in the original files, maybe going so far as a full source copy locally instead of using the ones quicklisp brought down, or just make your own local 'patch' file of redefined functions or extended classes that you load after loading the original and the patch simply recompiles and replaces things you've changed.

Is there an easy/idiomatic way to save your edits, or do they get blown away the next time you download the library? All my comments are in the context of not having tooling for this sort of thing. I wouldn't be surprised to learn the Gods at Common Lisp figured all this out years ago. (In which case, why is the whole asdf fracas happening?!)

Heh, that thread brings to mind a bunch of things... One advantage of CL that helps with a 'borrowing' aspect that give people the jeebies is that the unit of compilation is much smaller than the file, so you can also borrow much less. Another is that methods are decoupled from classes so there's a lot of room for extensibility. (The LLGPL interestingly provides an incentive for the open-closed principle, i.e. you extend objects you're fine, but if you modify the library's then you are subject to the LGPL.) If you haven't read Gabriel's Patterns of Software book (free pdf on his site), I think you'd enjoy it.

Your edits won't get blown away, at least by default, since quicklisp doesn't redownload a system that it knows about or check for 'corruption'. The way quicklisp does its own versioning also means if you update quicklisp's distributions lists and a newer version of the library has come out, it'll download that one into its own new folder and leave the old one alone. There's a cleanup function to clear out old things but I don't know of a case where that gets called hidden from you under the hood.

Maybe there's some magic and interesting stuff related to this for emacs or mezanno OS or in the annals of ancient lisp machines but I'm a vim heretic ;) But in any case if you want to save stuff, you can just save it in a new or existing buffer... So options are basically as I described. To give a specific example, I have a script that fetches and parses some HTML to archive HN comments and I wanted to remove the HTML bits so I'd just have text, making it more markdown-y. There are lots of ways to do that and I'm pretty sure I chose one of the worst ones, but whatever. I was already using the Plump library, and after not achieving full success with its official extension mechanisms, one method I came across and stepped through was https://github.com/Shinmera/plump/blob/master/dom.lisp#L428 and I learned I could hijack it for my purposes. I started by editing and redefining it in place until I got what I wanted, but instead of saving my changes over the original, I simply copied it over to my script file, modifying slightly to account for namespaces e.g. it's "(defmethod plump-dom:text ((node plump:nesting-node))", thus redefining and overwriting the library implementation when my script is loaded and run.

Some possible problems with this approach in general include later trying to integrate that script with other code that needs the default behavior (though CL support for :before/:after/:around auxiliary methods can help here, e.g. if I can't just subclass I could insert a seam with an :around method branching between my custom implementation over the library's without having to overwrite the library's; and in the long-term the SICL implementation will show the way to first-class environments that can allow multiple versions of stuff to co-exist nicely). Or the library could update and change the protocol, breaking my hack when I update to it. Or in other situations there may be more complex changes, like if you modify a macro but want the new modifications to apply to code using the old def, you need to redefine that old code, or if you redefine a class and want/need to specially migrate pre-existing instances you need to write an "update-instance-for-redefined-class" specializer, or if the changes just span across a lot of areas it may be infeasible to cherry-pick them into a 'patch' file/section of your script, so you're faced with choices on how much you want to fork the files of the lib and copy them into your own project to load. But anyway, all those possible problems are on me.

The asdf noise isn't that big of a deal and I think is only a little related here technically since it's a rather special library situation. It's more 'interesting' socially and as a show of possible conflict from having a core piece of infrastructure provided by the implementation, but not owned/maintained by the implementation. An analogous situation would arise if gcc, clang, and visual studio all agreed to use and ship versions of musl for their libc with any other libcs being obsolete and long forgotten. A less analogous situation is the existing one that Linux distributions do -- sometimes they just distribute, sometimes they distribute-and-modify, whether they are the first place to go to report issues or whether they punt it off to upstream depends case-by-case.

>The idea that projects should take source copies instead of library dependencies is just kind of nuts, at least for large libraries.

There's a medium to be found between static linking and dynamic dependencies which I think is one of the most crucial tradeoffs to understand in software engineering. NPM in my opinion goes a bit too far on the modularization of things. Especially in large projects where you can have multiple private repo URLs and different auth tokens needed for each, it can become a nightmare to orchestrate for CI/CD. Something that takes an NPM approach, but with a more fleshed out stdlib to eliminate things like left-pad is probably the way to go.

> But for large libraries, taking a source copy is absolutely crazy.

The solution is, when possible, to use (and write) smaller libraries. Many libraries are far larger and more complex than they need to be.

Depends on what it is you need. If you really have a small need, then taking a giant dependency is very possibly a bad idea. But I have seen the opposite pretty often. “I don’t need that library. I just need this one thing. Okay, this other thing, and this. And this and this and this.” And by the time they’re done they’ve reimplemented 75% of the library in a far worse way.

The most egregious example I ever saw was reimplementation of xml parsing with C++ substrings. They “didn’t need” a real xml parser. In the end, they found endless bugs with their shitty parsing logic and it turns it it was responsible for something like 50% of the (very large scale) system resource use.

Taking source copies is thought to be especially FOR large libraries. You take what you need, leaving the "large" behind...if you can. You probably can. Though, libraries are developed with the assumption that they are to be consumed as one massive binary. Making it more difficult to run off with just the part(s) you need.

Reply on news.ycombinator.com