Hacker News Reader

Comment by bob1029 | Hacker News Reader

We provide software that runs within very secure financial networks, and have some extreme constraints regarding what sorts of 3rd party code we can pull in. We are having to do a vendor swap on some document parsing code because one of our clients scanned a dependency and found it could be vulnerable in a context that it would never be exposed to in our application. These types of things make it really risky to go out and build your enterprise on top of someone else's idea of a good time.

Virtually everything we do is in-house on top of first party platform libraries - i.e. `System.`, `Microsoft.`, etc. We exclusively use SQLite for persistence in order to reduce attack surface. Our deliverables to our customers consist of a single binary package that is signed and traceable all the way through our devops tool chain, which is also developed in-house for this express purpose of tightly enforcing software release processes.

This approach is certainly slower than vendoring out everything to the 7 winds, but there are many other advantages. Every developer knows how everything works up and down the entire vertical since its all sitting inside one happy solution just an F12 away. Being able to see a true, enterprise-wide reference count above a particular property or method is like a drug to me at this point. We are definitely over the hill and reaping dividends for building our own stack. It did take 3-4 years though. Most organizations cannot afford to do what we did.

View on HN

Couldn’t this end up being disaster with a large codebase and even a small amount of turnover? Are you really advocating that someone should write all their dependencies themselves even if they can afford it?

It depends on the context. Yeah it'd be lovely to have your system connected to the internet - but no, our clients wouldn't give us money if you do that, and we need money. So, maybe yes in this case, start writing some quality, well thought out maintainable libraries (that can include audited third party code), and just bill it. In my case, the cost to the client of a team of devs working on that was less than the cost to them of the risk of a film leaking...

[Edit] - But what I have found through experience is that the code that was written under these constraints seemed to be better, more secure and robust, than without them. YMMV.

I assume if you have the money to be that thorough you have the money to offer some inducements to stick around.

Plus if all you know is this custom stack, where are you gonna go?

Anywhere that hires good software developers, since if you learned this stack it's presumably not hard to get a job somewhere else? Exooglers have a pretty easy time getting hired.

I’m find it amusing that we’ve reached the point where developers can no longer imagine a shop that writes its own software.

The software community has built a lot of extraordinary tools that have been through a lot of battle testing. Pretending those lessons aren’t worth something and thinking you can do it all yourself is a mistake a lot of the time.

If you're taking about foundational dependencies like OpenSSL, Linux, LLVM, or even jQuery and React, sure. Also most stdlibs and DBs, like GP said he uses, also fall into that.

Dependencies in general, like 95% (or more) of the kind we see in modern package managers? No, they're mostly untested liabilities and the majority of them could be rewritten in an afternoon.

This whole discussion is a bit strange. GP clearly uses dependencies, just not as much as everyone else today. I don't understand why the fixation with polarizing the discussion into "use lots of dependencies" vs "write everything from scratch".

The community had produced 1000x as many tools that are garbage. It can sometimes be hard to tell the difference (if the developer cares to look at all).

"The software community" also built npm, which then enabled "extraordinary tools" like leftpad.js... I suspect there's a lot to be learned from "those battle-tested lessons", but pretending everybody is learning them is a mistake.

Dependancies are inevitable.

I'm OK with depending on glibc and _almost_ as OK with depending on openssl. And of course I inevitable rely on gcc/clang/CPU microcode/transistor photolithography...

But those examples are a world away from npm pulling in 12+ levels of random dependancies of unknown origin buried so deep it's all but impossible to audit - and the segment of "the software comm8unity" who built that particular house-of-cards and then promotes it blindly to the segment of the software community who obliviously use it to generate code that runs on expensive production platforms in business critical applications just boggles my mind.

(And makes me alternately weep and drink heavily - having to support exactly that in production because "velocity" and "moving fast and breaking things" are considered more important that quality or security by both management and clients - both of who will happily point the finger of blame elsewhere when the inevitable happens, in spite of having been repeatedly warned... :sigh: )

You definitely have some valid points. On the other hand, I just did a project at my company that had dynamic PDF generation from web-app data as a requirement. Being a shop that in no way has the experience, time, or budget to build a dependency-free in-house tool to do this, I instead was able to find the jspdf library, pull it in from node, and implement the functionality in an afternoon.

Believe me, as someone who cares about code, I far prefer a world where I know how every line of code in a system I'm building works. In fact, in my free time, I do just that. But my job as an in-house software dev for a non-tech company is to solve business problems with technology, and as with every business problem, there is an acceptable level of risk that you need to be OK taking. In our case in this post-COVID world, people's jobs are literally depending on us iterating quickly.

Oh sure. I'm totally not saying that the right point on the velocity/quality spectrum is always right over at the quality end (or even that those two choices are fundamentally orthogonal), and by using the phrase "an acceptable level of risk" you're demonstrating you're the sort of developer that understands there _is_ risk. And I'd assume that means you'd have pushed back if someone had instead "and this PDF generator needs to run on the same servers and have access to the databases with all our PII and financial information on them".

I spend a lot of time talking to "full stack developers" who came out of graphic design into front end web dev and then fell into nodejs backend dev, who's only "architecture" course was about drawing buildings and who's main complaints about security requirements are that bars across windows look ugly. A startlingly large number of them don't even know there are things they don't know. I'd say fewer than half of them could tell you what OWASP was, and fewer than 10% of them could tell you what SQLi or XSS was, and whether their sites need to consider them as attack vectors. Most of them just say "I use $frameworkDeJour, it handles all the security stuff!"

(BTW, last time I needed to supply dynamic PDFs in a web app, one of the good fullstack-via-FE-and-graphic-design devs build the PDF generation in the browser (using some random js/pdf library) so we could just feed it JSON from the backend. Made me sleep better at night doing it that way...)

Out of curiosity, why would you expect 5x more people to know what OWASP is than to know what SQL injection is? I would have thought it would be the other way around.

In my head at least, knowing of the existence of a respected/curated list of problems/solutions in your area of expertise is likely to be more widespread than knowing the details of specific items on that list.

I know _of_ HIPPA regulations, but not being in either the US or in healthcare records, I have very very vague notions of the HIPPA requirements. Same with PCI compliance - I know there are important rules and requirements, which I don't fully understand the details of, because I choose to use 3rd parties like Stripe to handle all my CC processing so that those requirements don't apply to me (with there exception of needing to understand the risks of webapps and problems like XSS in the context of Stripe-powered CC forms).

It is not uncommon to find cases where the whole monstrous library was brought in just because someone needed 5-lines trivial function. And these add up. And then we all get surprised how come this page weights megabytes and barely does anything fancy besides screwing-up scrolling.

I think the problem is mostly cognitive. The codebase should be viewed as the most important investment that a software company could ever hope to possess.

Within the codebase, the infrastructure and tooling is by far the most important aspect in terms of productivity and stability of daily work process.

If you take the time to position yourself accordingly, you can make the leverage (i.e. software that builds the software) work in virtually any way you'd like for it to. If it doesn't feel like you are cheating at the game, you probably didn't dream big enough on the tooling and process. Jenkins, Docker, Kubernetes, GitHub Actions, et. al. are not the pinnacle of software engineering by a long shot.

A company's codebase is a liability, not an asset - it needs to be maintained, and as you point out, it needs money spent on tooling and infrastructure to be most effective.

Unless you happen to be one of the very rare companies that sells source code and not built artefacts, your asset is the built artefact and your code is the expense you take on to get it.

Having less code to get the business outcome only makes sense when you see the code as a cost, not a thing of value itself.

> A company's codebase is a liability, not an asset...

> your asset is the built artefact

Therefore, summarized, you mean that the sourcecode needed to generate the resulting <app/service/whatever> is a liability, but that the result can be an asset (if it does generate external revenue, or internally lowers costs ,etc..)?

I personally never thought about this kind of separation - interesting.

I mean that source code comes with costs, often substantial, but has no direct benefits.

It's easy for us as developers to think that source code is valuable - but this leads to problems like never removing code "in case it's needed", or with an in-house dev team developing systems you could get off the shelf.

If code is an asset, then it makes a lot more sense to write stuff yourself: you not only get the artefact, you also get the source.

If it's a liability, then it makes a lot more sense to let someone else bear the costs of that liability, especially if they have economies of scale, except where you can't get your desired outcome other than writing code.

This is of course technically incorrect. It's perhaps more accurate to say that source code requires upkeep and is expensive to maintain. That tends to draw less interest and discussion, because it's "obvious." Except as an industry we're overall pretty lousy at paying the required upkeep on code.

I think you have a point. But essentially what you're saying boils down to unmaintainable, untestable, unrefactorable, obsolete code being a liability. I don't think anyone disagrees with that and I personally have been involved with a (semi-popular public cloud) service deprecation myself precisely because it was legacy and simply had to be rewritten to be moved to a shiny new home that used new-age systems owned by another team that was doing a stellar job at upkeeping and innovating on those (though one could easily classify their "innovation" as NIH). The code that they wrote wasn't a liability at all, but in fact, it kept seeing active investment from all engineering and product management angles.

Think about a machine in a factory. You have to maintain it and it doesn't generate revenue by itself. It generates the product you then have to sell. The machine is still an asset.

This is misleading and pretty much wrong. It’s like saying the hen is a liability, the only asset is the egg. Or like saying your team is a liability and the only asset is the work they produce..

But these are both true. The problem is only if you assign a derogatory meaning to the word liability. If the company could provide the solution it does without a team, or with a smaller team, it would - the purpose of the company is to provide the solution, not feed team members. If we could get eggs without taking care of hens, we would - they're a pain to take care of and feed.

The moment you want to minimize something while still achieving your goals you know it's a liability. Do you want the same profits or solutions with a smaller team? Then the team is a liability. If team were an asset you'd be trying to hire a bigger team without any work for them to do. If you had the chance to double egg production with constant demand, you'd eat or kill half your hens. They're a liability.

> The problem is only if you assign a derogatory meaning to the word liability

That's only true if you assign an appropriate meaning to the word derogatory.

A company's codebase is a liability, not an asset

Why not just throw it away, then?

If you can, you should.

If you can't, it's (IMO) because you're stuck with the burden of having to write and maintain that code.

If it's an asset, why not just write more?

> If it's an asset, why not just write more?

Companies do write more.

It’s a liability in the sense that a rusty drain pipe covered in duct tape is a liability. You think of it as the source of your problems, but throwing it away would be worse.

> It’s a liability in the sense that a rusty drain pipe covered in duct tape is a liability. You think of it as the source of your problems, but throwing it away would be worse.

So in your analogy, what's the software equivalent of replacing the drain pipe?

> A company's codebase is a liability, not an asset

Its an asset. Like many (virtually all, other than pure financial) assets, it has associated expenses; maintenance, depreciation, and similar expenses are the norm for non-financial assets.

> Unless you happen to be one of the very rare companies that sells source code and not built artefacts, your asset is the built artefact and your code is the expense you take on to get it.

No, things that are instrumental to producing product are still assets, not just the things that you sell. That's true if its machines on your factory floor, if its the actual real estate of the factory, or vehicles that you use to deliver goods. And all of these assets, like a codebase, have associated expenses.

The whole "code is a liability, not an asset" line is something from people who might understand code, but definitely don't understand assets and liabilities.

From a pure accounting perspective, yes, but the whole "code is a liability, not an asset" crowd aren't using the strict definition of it by any means.

Its more just a mental footnote that the contract with the customer is what is valuable, and the code is either supporting that value or destroying it. So if you can have the same contract with the customer for less code that's the outcome to strive for.

I think this perspective comes from the opinion of folks that generally want to cut costs, not leverage their assets to their fullest. If your means of production is an asset that you understand how to leverage over your competitors your mindset won’t be “we really need to minimize this codebase”

No, it comes from wanting to leverage your asset to the fullest. But that's hard to do if it's full of cruft and getting in the way of actually delivering value.

More lines of code is not equal to better. What you need is better lines of code, which usually means less of them.

Let a dev loose on a codebase and they could add value, but they could also be subtracting it. Hmm maybe developers are the liability.

One needs to keep in mind though, that minimizing a code base can have significant cost as well. (1) the cost of doing the minimization itself (2) the cost of runtime or compile time differences if any. Also one needs to keep in mind, that less code does not mean easier to maintain either.

Perhaps the amount of code is the wrong metric to optimize. Perhaps it is readability, simplicity, maintainability that we really want and those are harder to put into numbers than LOC.

I think the saying is drawing on the same idea as Bill Gates' opinion that measuring progress on software by lines of code written is like measuring progress on an airplane by weight.

Airplanes have many functional parts and many not-so-functional parts. There are parts of the airplane that will, if removed, prevent it from working in various critical ways.

But from another perspective, all of those parts, decorative, functional, or essential, are liabilities, dragging your airplane back toward the ground when you want it to stay up in the air. The fact that a particular piece is important doesn't mean it's less of a problem having it; it means you have to suck it up and work around the problem.

Source code is like this. The mere fact of its existence causes problems. Some of it you can't do without. But it's causing problems anyway, and if you can do without it, you want to.

> Source code is like this. The mere fact of its existence causes problems.

So you delete your source code after you ship an app? That doesn't make sense. The source code is one of every software company's greatest assets which is why we have created so many tools to keep track of it like version control, automated testing etc. Otherwise there would be no point in clean code, code documentation etc, just write something that solves the problem and done! No need to even check in the code, just build a binary locally and ship!

I understand the point, but saying that code is a liability and not an asset is false no matter how you look at it. Source code solves a lot of problems you can't solve with binaries, it lets you adapt to change much better. So instead of saying that code is a liability, use the business saying "Focus on your core business", meaning don't write code for things that isn't your core business.

> I understand the point, but saying that code is a liability and not an asset is false no matter how you look at it.

I didn't say that. I said it was a liability. "Don't write code for things that aren't your core business" doesn't tell you that you should try to minimize the amount of code that addresses your core business. But you should.

You misunderstand the analogy. An airplane could be very heavy because it is a very large aircraft with a lot of payload capacity - that would be good. An aircraft could also be very heavy because it is inefficiently built and thus have very little payload capacity. Similarly an aircraft could be light because it is small, but still be inefficiently built, or the reverse. Bill Gates is complaining about evaluating an aircraft by its weight because its weight doesn't on its own tell you about any of the stuff you really care about with an airplane.

Likewise, code could be many lines because it is well formatted and robust. Alternatively it could be long because there's a lot of repetition and bloat. It could be short because it is very targeted to what it needs to be, or it could be very short because the developer used a lot of unreadable code-golf tricks.

In any complex system, you have optimization problems. Very rarely is the answer to an optimization problem simply to minimize an isolated variable. You can not say a lighter airplane is better than a heavier one without asking why it's lighter; likewise a smaller codebase with fewer lines can not be called better unless you know why it is smaller.

Just because a plane could get off the ground without something does not mean that cutting that thing to save weight will move the plane closer to its optimal design. Likewise just because the code base could be reduced does not mean that actually makes it more maintainable and better performing.

It comes from the fact that code depreciates like a car. Yes it does make you a profit (gets you to work) but after 20 years nobody wants your code (car) unless you have done rigorous maintenance. Having the biggest codebase(car) is not a benefit. It's kinda like measuring your software product by how many cups of coffee your developers have drunk.

> It comes from the fact that code depreciates like a car.

If a business owns a car that they use to make a profit, guess where the car is in the balance sheet?

> Having the biggest codebase(car) is not a benefit.

Sure, lines of code is not how you measure the value of code as an asset, just like tons of gross weight isn't how you measure the value of vehicles as an asset.

That doesn't mean code, and vehicles, aren't assets.

“Lines of code are a measure associated with maintenance cost, not asset value” is a reasonable statement. “Code is a liability, not an assset” is not.

> Its an asset. Like many (virtually all, other than pure financial) assets, it has associated expenses; maintenance, depreciation, and similar expenses are the norm for non-financial assets.

To get technical about it, code whose discounted future maintenance costs exceed the discounted revenue it is expected to bring in or helps to bring in is a liability. Some maintenance costs are invisible, such as having team members leave; leading to recruiting expenses to seek a replacement and the associated ramp time when they're hired. Companies are often not equipped to assess the cost side in any way that approaches reality and just stick the cost of the "code" on the balance sheet as capex. Having an asset on the balance sheet after doing this doesn't mean you have an asset in reality.

> The whole "code is a liability, not an asset" line is something from people who might understand code, but definitely don't understand assets and liabilities.

Code can be a net liability. Unless you are looking purely at the asset side of the balance sheet and ignoring liabilities, which tends not to be very useful.

I think many folks forget that nobody wants to use your software. They want something that maybe your software can help them get maybe. But if the problem could be solved without your software, they'd try to remove your software ten times or more if they could.

It's still an asset, it's just that most assets require maintenance.

It always comes down to resources. In my experience an underdeveloped or funded NIH project happens far more often than most tech teams would admit. That said, when it’s only lean on vendors and no invention is happening then it’s basically an expanded IT department and will be eventually treated as such.

If a company isn't competent enough to even check their dependencies for vulnerabilities, as GP's company is, how could they be competent enough to write and check their own alternative versions?

I think this is simply the recognition that our expectations of what team of developers can do in a given time has surpassed what that team can actually write themselves. We know understand that teams have to lean on open source, or buy libraries, in order to achieve what we expect within a reasonable time.

The reason is very obvious - the most expensive part of most projects is the people. Why pay them to write code that you can just download for free? That's a tremendous waste of money.

This is exactly it.

You should raise that point to all the clueless shareholders who relentlessly pulled even technical control out of the hands of the techies.

Most of us just do what we're told and we're in no position to question the "company's priorities" -- nevermind the fact that very often those priorities actually align with the techie's vision.

Let's start a car company by first building an iron smelter? Using available tools is not some kind of crazy idea that software engineers came up with in the last ten years.

Original commenter advocates for writing your own, presumably "from scratch", and mentions high-risk targets. Even if you don't take those for granted, though... Let's assume lower risk than a studio, and you relax the conditions from developed-in-house to maintained-in-house (e.g., a library exists, so you go grab it, and by the power of open source, internally it's now "yours"—a fork that you take full responsibility for and have total control over, just like if you had developed in-house, except you're shortcutting the process by cribbing from code that already exists.)

Here's an unrecognized truth:

The cost of forking a third-party library and maintaining it in-house solely for your own use is no higher than the cost of relying on the third-party's unforked version. Depending on specifics, it can actually be lower.

Note that this is a truth; the only real variable is whether it's acknowledged to be true or not. Anyone who disputes it has their thumb on the scale in one way or another, consciously or unconsciously.

You're really claiming this is a universal truth? Do you think it applies to OpenSSL? Chromium?

For cryptography one should just be able to rely on their OS' library, and depending on a full browser with high amounts of code churn, no compatibility for implementation code and a large dependency graph of its own is not really seen as a good thing in this context at all.

Let's be more specific: do you think Brave would be better off if they hard forked Chromium?

Chromium ships with Windows and is maintained by Microsoft. Use the OS crypto library.

libjpeg?

Specifically, I think hard forking is a bad idea for any sort of library that needs to be regularly updated for compatibility or security reasons.

That's possibly true if you don't have headcount for doing that maintenance. If you have appropriately planned for it however, it's just more software that you're writing to do the work you need done.

If you're depending on some random person on the internet to update software which underlies your whole stack, then when the next imagetragick drops you can't update until they get around to fixing it. Since you won't have developers familiar with the code, fixing it won't likely be feasible for you. That's a lot of risk.

I didn't say it was universal. OpenSSL and Chromium are outside the scope of "CDNs for external JavaScript libraries".

(I actually included the appropriate hedging to clarify that my comments are scope-locked to that topic and to prevent digressions like this, but I edited it out because it made the comment too hard to read. Goes to show...)

Reading your "the cost of forking a third-party library and maintaining it in-house solely for your own use is no higher than the cost of relying on the third-party's unforked version" I didn't realize at all that you were trying to only talk about JavaScript that runs client side. I think your deleted clarification would have been helpful!

> I didn't realize at all that you were trying to only talk about JavaScript that runs client side

Well, not just client-side JS; server-side, too, or anywhere that NPM is used, but even more than that: e.g. other package managers that were influenced by or work similarly to NPM and encourage a similar package-driven development style, e.g. Rust's crates.io or the Go community's comfortability with importing by URL. It applies for many of those cases, too, it just wasn't the focus of my comment.

> Note that this is a truth; the only real variable is whether it's acknowledged to be true or not. Anyone who disputes it has their thumb on the scale in one way or another, consciously or unconsciously.

Could you elaborate on the actual argument for why this is the case. On the surface it seems like the opposite of what you are claiming can just as easily be true depending on the situation. For example take a widely used utility lib such as lodash or jQuery. In your scenario there are two options:

1. Use lodash via a package manager and rely on the lodash team to fix bugs, write tests, and add new useful utilities over time.

2. Fork lodash and take on the maintenance burden yourself. You are responsible for keeping up with security vulnerabilities and making patches. You are responsible for writing high quality tests for the parts of your code that diverge for the original.

Think about how much ramp up time is required for new hires to become familiar with a company's codebase. Why would you ever want to devote that amount of time to maintaining code that for the vast majority of businesses is already "good enough". For some companies the engineers working on the open source project may even be more competent than the resources available in house. Sure there may be edge cases were performance or security is absolutely paramount, and in those this approach may make sense, but not for the majority of generic CRUD apps.

I would have a very tough time trying convince a competent manager that it is beneficial to devote so many man hours to this task rather than to business logic, new app features, etc.

I'm not really saying you should write anything from scratch that you don't have to, just that you should treat a dependency as something that you did. Therefore, review it, check compatibility, and have some named team responsible for its maintenance and availability.

Even without turnover, I've no idea how you're supposed to compete with the quality of top open source software. Maybe no one finds the bugs in your software, but they're definitely there.

We don't try to compete with the quality of top open source software. Our stack fundamentally consists of:

C# 8.0 / .NET Core 3.x / AspNetCore / Blazor / SQLite

Of these, SQLite is arguably the most stellar example of what open source software can provide to the world.

Everything else in our stack consists of in-house primitives built upon these foundational components.

It's funny to me the number of developers who have effectively forgotten that Microsoft exists, and that it's possible to have your entire stack be provided by one company who directly sells it's software for profit.

That's really interesting. My team's stack is the same except we use Azure SQl server instead of SQLite.

I'd love to understand why you chose that.

Feel free to hit my up at the address in my profile if you don't want to talk here.

Perhaps the parent's situation ("software that runs within very secure financial networks" cf https://news.ycombinator.com/item?id=24747781) prevents them to rely on an external service?

We cannot rely on anything outside the client's secure network. There are a few exceptional items that are allowed to talk to the internet, but our system's data persistence layer is not one of them.

Effectively, our client's operations cannot rely on cloud services and all of the related last mile connectivity into their infrastructure. If AWS/Azure/et.al. go down, many of our customers are still able to continue operating without difficulty.

In the frontend space?

When I go looking for a dependency, I check the license and I have a quick read of the code.

Last time I did that was for autocomplete. I checked the most popular six options; in the space of 2 hours, I found obvious-from-reading-code bugs in all 6.

None of them had a CLA signed by contributors, so there's really no evidence their code is genuinely available under the license they claim to offer.

I wrote my own. It took about 3 hours initially plus 2-3 hours ironing out edge cases found over the following weeks. It only added 700 bytes to my bundle.

Total time spent: 1 days work. Smaller code, loads fast, free from license issues, does exactly what I want.

A lot of time when you are doing high critical environments you tend to be less cutting edge and rely more on old proven technologies that is boring, but dependable.

Reply on news.ycombinator.com