Hacker News Reader

Comment by ChrisMarshallNY | Hacker News Reader

> Another important reason for software complexity lies in monolithic design, wherein all conceivable features are part of the system's design.

This is really a call for modularity. I'm a believer in modularity, but it is not a cure-all.

Modularity brings a lot of baggage with it, like duplication of code and lifecycle management, along with increased resource usage (ironic, as the article is mainly about reducing resource usage).

But I am still a believer in it. In my experience, it makes for a massive increase in system stability, quality, agility, and speed of development.

View on HN

Maybe I'm misunderstanding, but it sounds like you talk about modularity at the unit of deployments, i.e. separate services.

But you can have modularity within a single codebase and some languages / build-systems allow you to define very clear boundaries between your different modules. In that case, I don't think you have necessary a lot of code duplication (you might have slightly increased consumption due to the use of more indirection). But you might have to code to interfaces and explicitly eliminate coupling etc.

Oh, no. I use it all the time, for code components.

If you look at my portfolio[0], you’ll see it packed with things like widgets and adornments. Most of these, I use in my own projects. I eat my own dog food.

But today’s languages, with their reliance on extensibility and interfaces, makes writing standalone code assets a lot less cumbersome. I write in Swift, and there are some pretty big Swift "extension suites" out there[1]. The Apple Swift Package Manager is nice, too[2].

It's just that when I write a code module, I do it as a standalone product, with a full test suite (usually a harness[3], which often takes the form of a "full-fat" application), and a unique release management cycle.

My modules are pretty independent, so they aren't usually custom-tailored for all of their implementations; acting in a fairly general capacity.

[0] https://stackoverflow.com/story/chrismarshall

[1] https://swifterswift.com

[2] https://swiftpackageindex.com

[3] https://littlegreenviper.com/miscellany/testing-harness-vs-u...

Modularity is, by definition, a feature of the code. No feature is free. Some do have major benefits. More often, some are required by tradeoffs or circumstances of deployment.

They could have just said "distributed systems". It's not like the Node kids suddenly invented "microservices".

If the "modules" are source code you don't know or can't control, then it's a dependency that can create dependency headaches. For example, a new OS or browser version comes out and the existing components are not compatible with the upgrade. Happens all the time. External components are thus not a free lunch.

What's really missing is the willingness to say "no". There's pressure to put fancy or trendy toys into applications to look up to date or to get experience on the latest fads. KISS and YAGNI are often ignored for eye candy or Fear of Falling Behind.

For example, fairly early in the 2000's I had a web form that only did server-side date validation. It worked just fine. But client-side date validation was becoming "expected" because everyone started doing it and the manager asked me to put it in. So I found a JavaScript component on the web, and it worked fine at first. But a few months later a bug was found in it that appeared on a certain date, and it generated a lot of phone calls from users. Those phone calls would have never existed had I gone with my gut and left it out. Few users are going to spend all day making date errors such that client-side date validation is NOT necessary. It's a me-too toy.

Warren Buffett often cites his ability to say "no" as to why he's so wealthy. He looks at the actual tradeoff numbers, not peer pressure.

> Modularity brings a lot of baggage with it, like duplication of code and lifecycle management, along with increased resource usage (ironic, as the article is mainly about reducing resource usage).

I just can't conceive of how modularity, in general, causes duplicate code rather than reduces it. And this is before considering something like generic code which could potentially further reduce duplication. Every time we've been deliberate about using modularity in projects it ends up creating libraries that reduce the total amount of code across projects. Even if it's just in one project, good modularity has been used to reduce a lot of duplication.

As I mentioned above, if the module is expressed as a code extension, then it is, sort of, "free." It is compiled into the main code body.

But, for example, say we are talking about a communication module. One module may use a well-known library as a bundled dependency, and another module may use the same one. They may be smart enough to use dylibs, in which case, you have the potential for "DLL hell," or they may use static libs, in which case you have duplication of code, and maybe resource contention.

Ok, so it's a conflict of meaning.

I would not consider static libraries to be duplication of code. Duplication of assemblies or compilation unit, sure, but not code. The code itself has not been duplicated.

Fair 'nuff.

Others have already pointed to the distinction between code duplication and runtime duplication.

Regarding the development experience:

I think it's a tug and back-and-forth kind of dynamic.

Modularity has exactly the kind of effect you talk about, until it doesn't.

At some point, the cost of communication (between people!) responsible for different modules can increase until it's a net negative to try to adapt a module, or even find out it exists and is suitable.

Of course, a monolithic project of comparable size has likely collapsed from its weight into a ball of mud.

Development tools have come quite a ways, but the science of getting teams to develop software together, perhaps not so much.

> Development tools have come quite a ways, but the science of getting teams to develop software together, perhaps not so much.

Yup. I think I'll put that on a T-shirt.

> I just can't conceive of how modularity, in general, causes duplicate code rather than reduces it.

Yeah, like my node_modules folder doesn't contain tons of duplicate code.

Modularity doesn't cause that. Modularity doesn't inherently reduce duplication either. But modular code offers opportunities for reuse (or clearer opportunities for reuse) that non-modular code does not (or where the reuse potential is so obfuscated that it may as well be non-existent).

On an idea level, i'm all for DDD (domain driven design) and many of the ways to achieve modularity - be it at code level, like in the new versions of Java, be it on a library level, with shared common code, or maybe even having a hexagonal architecture, or microservices.

That said, in reality, doesn't any sort of modularity imply increased complexity?

  - you need to manage your modules and figure out where to put the code
  - you need to figure out mechanisms to share and reuse it, as well as test every single integration (breaking changes will be harder)
  - you need to invest significant resources into the Ops side of things - maintaining multiple deployments and making them talk to one another

It seems like it would lead to trading one set of constraints for another, each having their downsides. In my subjective opinion, monoliths are the only way to go for small teams (~1 pizza) and relatively small systems (~10k MAU), maybe migrating over to modular monoliths as time goes on, and only then branching out to other approaches.

I actually wrote a blog post on modular monoliths a while ago, that mentions using feature flags, however i'm not sure whether the tone/quality is entirely appropriate for HN: https://blog.kronis.dev/articles/modulith-because-we-need-to...

Regardless, it feels like the industry perhaps jumped over the concept of modular monoliths entirely and went straight for microservices.

Of course, if we're talking about modularity purely on the code level, then dynamic loading of code is its own can of worms, as can also be many methods of installing extensions.

I'm not sure why you think that isn't suitable for HN. It's a well-written, thoughtful piece.

Because of the slightly tongue in cheek style. It's something that would fit better in a Medium article, perhaps.

That's not to say that everything on HN needs to be cut and dry, but factual discussions seem a bit more productive, in my experience. To that note, having any stats on the prevalence of modular monoliths would have been awesome, or maybe even more information about feature flags - those are often not discussed all that much!

Oh, heck. I'm screwed, then. I'm a wiseass.

One thing I observed as a data analyst (prev job) was that modularity could make data anlysis pretty difficult.

Example: In a monolistic application you can easily send out telemetrics of different "components", e.g. player hitpoints, coins in inventory, days in game, etc. But in a microservice structure (at least the one I saw), since hitpoint service doesn't care about coins, and coins service doesn't care about days in game, to get all three in one telemetry, developers have to break the barrier and connect them in one way or another.

So essentially you still get to write them into sort of monolithtic, but uglier and needs more arguments (but still have to write them at the end because CXO needs the report).

That's what data warehouses are good for though. So long as you have something you can join on, you don't need to worry about how the application developers structure their systems.

Theoretically it's true (but again the burden is simple transferred to someone else).

But in reality, as a BI Developer (prev as a data analyst), what I really don't want to do is a join by timestamp (and a userId), pretty much the only way to join two "subsystems" that are not interconnected.

Example: Analysts want to know what's the coin balance when player starts purchasing energy bundles. Now that they don't know about each other so the telemetry only has one common field: userId. Eventually either BI developer (who develops DWH/View/etc.) or DA has to write a join by userId and timestamp to get the most accurate coin balance when player purchases an energy bundle.

Microservices are not the only way to modularize. But yes, cross-cutting concerns tend to wreck modularization; in desktop environments it's usually global settings or events that cause this.

(Long ago, the software industry wanted to copy the electronics industry, where people would assemble software from components listed in catalogs. We never quite got there, but you can see the influence of this metaphor all over the place when people talk about "components". It required the development of libre+gratuit Free online libraries to get there, causing an absolute Cambrian explosion and the left-pad problem)

Most things are cross-cutting concerns. For example, database access and security are issues that entity-specific modules have to deal with, and it's best to share those features rather than have every entity reinvent it.

Early in the GUI days, you had UI components that were pretty independent. But now they are expected to match the look and feel of given OS versions or brands such that they can no longer be truly isolated: they are expected to integrate somewhat with the OS. If people had the guts to live with "ugly", a lot of things could be simplified. I discuss saying "no" to the IT Fashion Show nearby. It may read like a git-off-my-lawn rant, but software could be cheaper if you kick fadsters off your lawn. Beauty ain't free.

That's an interesting take. My observation is quite the opposite, with each application seemingly overriding systems style willy nilly. I'd say that application styling has become MORE differentiated.

I mean that is exactly what correlation IDs and joins are for.

It's not that easy. Join by what? In a lot of my work I have to join by common userId (sure) and timestamp which is pretty heavy. Something like this:

SELECT *, ROW_NUMBER() OVER(PARTITION BY userId ORDER BY timestamp DESC) AS rowNum FROM A LEFT JOIN B ON A.timestamp <= B.timestamp and A.date = B.date

And then write another query to take the top row based on timestamp, and for sure this method may lose some data (e.g. what if userId doesn't have data in table B that day?).

(warning: old man rant ahead, apologies in advance)

My original answer was perhaps a bit terse and should have been more along the lines of "that is what a good schema design should fix". Your original position that "microservices make things more difficult for analysis" is absolutely true, but I guess my observation is that while you might get lucky and have everything you need in one place in your raw data, for any sufficiently complex system (especially if there are 3rd party components involved) you're going to need to do some real work to transform that data to build a repository with a flexible schema to support your analyses and make yours and your colleagues work easier and more productive. You are 100% right in that every layer of indirection makes this slightly more difficult, but unfortunately that is something you need to design for in all but the most simple systems - this is true today the same as it was true 20-30 years ago.

Hopefully you're already doing that sort of design, but my experience is that disciplined schema design for effective analysis is incredibly rare, despite it being an old, crufty, well established and documented area ala Kimball and old school Enterprise Data Warehousing and BI. Not all of it translates perfectly to the world of Big Data(TM) but the fundamental concepts of designing such that your data can be turned into easily queryable and useful measures and dimensions still very much applies.

For your example, the correct correlation ID completely depends on the context in which you need to do your analysis, and that should be ideally baked into your telemetry schema where applicable (and where it makes sense from an architectural/performance/data volume perspective), but if not, then baked into your intermediate ETL schema such that it is easily accessible for analysis. Perhaps it's a contextual or global incrementing index that more easily allows you to get the "latest" transaction per context (maybe a session or a match in your game, or an encounter, or outside of games, a shopping cart checkout flow), maybe your client passes more contextual information to the microservice to allow you to more easily hierarchically partition into your context. Maybe if you don't have that level of control downstream (like in a 3rd party service for example) you have an ETL or streaming job generating the global or local context from the raw telemetry that makes it easy to join against.

And lack of context for analysis isn't even just a microservices issue, it can even happen in any kind of modular software - in the game I'm working on right now, for example, the crafting system has no need to know about overall state from the player, hence the crafting module has no (easy) access to player state that might be relevant for downstream analysis - so even in what would be considered a monolith, getting access to player health/armor/xp/etc in the context of crafting analyses (e.g. do players craft medkits at low health more often than at high health?) is difficult and really uglies things up, much like you imply. But making available a playerStateIndex that can be used to join back to that state (managed by the player state system) is doable and introduces no real problematic module coupling, and is a repeatable and extensible pattern to use across other modules and even external services.

I've spent the better part of half of my 20+ years in the industry doing some form of data analytics, and my message to the analysts/data scientists I work with is always "if you have to do crazy gymnastics to do your analyses then our schema design needs work". You don't want to be in the (all too common) world of analysts spending 80-90% of their time doing data transformations and cleaning and 10-20% of their time doing analysis.

>if you have to do crazy gymnastics to do your analyses then our schema design needs work.

Yeah I figured out that if the query is over say 50 lines long and has weird marks everywhere (say DISTINCT with some windows function) then it's pretty much a bad database design.

Then I move to the data side and figured out that it actually also has something to do with the original raw data sent by the app.

So eventually _someone_ has to take the burden and do the gymnatics and frankly it totally depends on who is closer to the CXO. In our company the developers are in favor so they can do whatever they want.

>Hopefully you're already doing that sort of design, but my experience is that disciplined schema design for effective analysis is incredibly rare, despite it being an old, crufty, well established and documented area ala Kimball and old school Enterprise Data Warehousing and BI. Not all of it translates perfectly to the world of Big Data(TM) but the fundamental concepts of designing such that your data can be turned into easily queryable and useful measures and dimensions still very much applies.

EXACTLY what I'm feeling now. We are trying to work out a DWH on top of the raw telemetries (basically some simple transformations and routing done by Spark from HQ). However I really think that Kimball doesn't fit very well with the big data thing. I don't know how to say it clearly because I just moved to the data team for a few months so I'm a total newbie and everything I _feel_ could be 100% BS. I think we should start from business requirement (i.e. from our data analytics lords), and then bend the rules (Kimball will require a LOT of extra joins if what I'm reading it correctly) and keep the DWH tables sort of flattened but in a way that makes more sense than the original raw telemetry tables.

I'm not in control so I don't know what it will result into.

>For your example, the correct correlation ID completely depends on the context in which you need to do your analysis, and that should be ideally baked into your telemetry schema where applicable (and where it makes sense from an architectural/performance/data volume perspective), but if not, then baked into your intermediate ETL schema such that it is easily accessible for analysis. Perhaps it's a contextual or global incrementing index that more easily allows you to get the "latest" transaction per context (maybe a session or a match in your game, or an encounter, or outside of games, a shopping cart checkout flow), maybe your client passes more contextual information to the microservice to allow you to more easily hierarchically partition into your context. Maybe if you don't have that level of control downstream (like in a 3rd party service for example) you have an ETL or streaming job generating the global or local context from the raw telemetry that makes it easy to join against.

It's...not really easy to come up with an intermediate "common context" id sometimes. Session Id doesn't work because each session takes a lot of time and nothing else really are real-time. Since I'm the only guy in the company for requirements taking (I really think developers should have their own business analyst), it's pretty painful that each time I have to fight both sides. I have to ask the DA team to relax their requirements (realtime? No it's not possible, how about same session?) and ask the developers to bestow more connection fields (if you have this field DA will be really happy), thus the vent above :D

This is what api's are for. A telemetry service asks each service for the specific data, combines them and outputs the combined metrics. This doesn't really break a barrier, but it does increase overhead.

Yeah but other teams don't really have control over that.

Keep in mind that modularity is a complex notion: the space of components can be rather non-trivial, in that the dependency graph can have a pretty interesting topology - if you include, in this notion, for instance, inheritance, generics (templates), and all the related usage patterns that have been invented over the past 40 years. (I believe Modula and Oberon both failed to address this complexity, having restricted the programmer's understanding of the concept to a very basic use case.)

> Modularity brings a lot of baggage with it, like duplication of code and lifecycle management

If the operating system and standard libraries provide better facilities for the common parts, that can go a long way to solving these problems.

Exactly. But these days, everyone likes to bundle in dependencies.

Also, for cross-platform SDKs and libraries, the provider may be forced to "reinvent the wheel." That may well be worth it, as developer bandwidth, and time-to-market are important variables.

> But these days, everyone likes to bundle in dependencies.

If the OS and standard libs provide those dependencies, then I don't need to bundle them.

I'm thinking of things like the Go or Python standard libraries, or OTP in Erlang. On iOS you have a rich set of built-in frameworks, etc., etc.

...and you have a huge number of folks that program for iOS in JavaScript, with dozens of dependencies in each project.

This can lead to some rather...overfed...applications, and four- or five-digit bug counts.

That sounds like a problem for those developers and their approach, not with the platform on which they are building.

Yup, but these have become so common, in the industry, that they are now synonymous.

For example, I no longer trust people that describe themselves as "iOS developers" to know the first thing about actually developing for the Apple ecosystem. In some cases, they don't even know how to use Xcode.

It sounds like UNIX.

I'm convinced that the interesting stuff happens at the interface. It's just like for neural networks: the information bottleneck makes each tool distill information inside its output.

1+1 > 2 when multiple people cooperate, thanks to this. I think the same can be applied to software. It really has plenty of advantages: code more audited, swappable parts, common interfaces, standardization on data exchange formats, testability of the components.

This is actually a call for restrained Product Management.

> This is actually a call for restrained Product Management.

I would adjust that, like so:

This is actually a call for restrained, jargon-free, Product Management.

Reply on news.ycombinator.com