Hacker News Reader

Comment by lkrubner | Hacker News Reader

I wish I could give you more upvotes. You are describing a somewhat hidden psychology, which I think provides a rational basis for much of "Not Invented Here" psychology. We tend to think that "Not Invented Here" psychology is irrational, but in fact, the loss of control over possibly crucial technology is an important cost, which makes all of us stop and re-consider whether we really want to use some software developed by an external team.

View on HN

And it's not only that - Time spent not doing our own designs (and instead spent memorizing how to use magical frameworks) is time not spent advancing our technical understanding.

It's a sad state when otherwise very intelligent people think it's bad practice to use plain C and a clean OS API that has been stable for decades (because that was somehow "magic" und impossible to understand and error-prone), and advise you to use the Boost filesystem module just to concatenate two paths.

> Time spent not doing our own designs (and instead spent memorizing how to use magical frameworks) is time not spent advancing our technical understanding.

I can't upvote this enough. I used to do mathematics, and there was the story of a professor would would hold up a book and say, "You should know everything in this book. But don't read it!" Which is to say, you have to go through the process of discovering mathematics to really understand it (maybe with a bit of guidance when you get really stuck). The skill of building complex software systems is no different.

Sure and those of us experienced in writing software tend to avoid reinventing the wheel (in a buggy, untested way).

I see the value in learning by building yourself, but from a software engineering point of view, using a tried and tested framework is likely to give you higher quality product in less time.

Well, the framework I was to use last (Qt) had bugs that simply can't be fixed by users (memory leaks, double free leading to segfault when exiting after reloading QML engine) and immature modules (for example translation) and forces complicated types to the user and forces bad architectural decisions to the user and significantly increases compile times...

> using a tried and tested framework is likely to give you higher quality product in less time.

This is a common sentiment, but note that a framework has huge handicaps

- not knowing your business requirements and concepts

- must be suitable for many software projects that need features you'll never need.

- there is a clear maintenance boundary (framework vs your own code), which requires a complex interface with maintenance overhead and typically forces you to use concepts that don't really match your requirements

If you're experienced in the relevant domain it's almost always simpler to do it yourself / reuse your own work.

Writing your own QT sounds like a long road however that would likely not be as fast as making work arounds or providing bug reports or patches.

I would never write my own Qt. Why would I? In this case I was creating a simple dashboard type application, and that can be easily done using fewer dependencies.

I don't think reinventing the wheel is the right approach. I think careful analysis of bugs and design deficiencies the likes of which you experienced before jumping in and using the framework is the way to do it.

This is why I absolutely love seeing hate filled developer rants about technology with deep dive analyses and links to bug trackers. That's how I was saved from learning Ruby on Rails back in 2008 when most developers gushed about how awesome it after reacting to its slick marketing and building a tiny website in 5 minutes.

What did you prefer instead?

I learned Django instead. I'm cognizant of its flaws but I'm still relatively happy with it.

I think the designers originally intended it to just be "rails for python" but they recognized the pain caused by rails magic and subsequently worked on making django less magical.

Concatenating two paths is difficult if you care about any of the following: security, multiple OS's, Unicode (multiple code units, validity, combining characters), file system restrictions, etc.

Your "two decades" maybe holds for Linux, but what about Windows or MacOS???!

I have seen too many people use string concatenation.

I think an intelligent person would recommend to use the normal library (appropriate for your language, assuming it is well written) since usually your program will be doing many other filename/path manipulations too.

https://msdn.microsoft.com/de-de/library/windows/desktop/bb7...

Exactly my point - that function doesn't deal with modern windows. MAX_PATH is 260 characters. Yet Windows now supports longer paths (\\?\ prefix). So I presume there is another Windows function to combine long path names (and maybe canonicalise Unicode better).

> Yet Windows now supports longer paths (\\?\ prefix).

I know. And this is the attitude that leads to complex software. It's a self-fulfilling prophecy: "I can't do it on my own since the problem is so complex". In which case the problem does get complex.

I recommend not wasting time supporting this crazy feature (unless you are writing infrastructure code for tools and you're required to - in which case I'm sorry). 260 character paths are more than enough for any project. And I certainly recommend against using the crazy abstractions from Boost::filesystem.

(Personally I think it's an unfortunate example. I'd prefer to avoid paths from the start, since nobody understands the semantics of hierarchical filesystems. Alas, typically you need to deal with them to some degree).

It is not a crazy feature.

It is a necessary feature for many development environments originally written for Unix (e.g. nodejs), because Unix file systems usually don't have such a low character limit.

Or maybe Windows has mounted or network sharing, a Unix filesystem... In which case your program better deal with long paths (or just fail?)

Really... You are showing exactly why one uses a library so one doesn't need to care about the "complex" details because one hopes the library does a good job of managing it as well as possible for you (although you still need to know the details to use a library function safely).

> Or maybe Windows has mounted or network sharing, a Unix filesystem... In which case your program better deal with long paths (or just fail?)

260 bytes is long. I've never seen a \\?\ path in the wild, and I don't want to. It's a misdesign (well, I'm sure the designer didn't want to design it...). But I'm repeating myself... I can't practically deal with paths longer than, say, 64 characters (can't read them, let alone type them). Fix your paths.

And again, this is off-topic. This thread was about not using a crazy third party library, when the authoritative semantics (however insane) are contained in the OS interface.

That is such an awful toxic viewpoint to take.

"Fix my paths"!? Fix your software! I work from a network share that is about 70 chars long to get to my one project's folder. The beauty of hierarchical file systems is that I can cognitively ignore all the previous paths and just work from there. However when your software doesn't work because "the solution is a misdesign", it's not my path that is wrong, it's your software that is broken and you that are stubbornly refusing to use solutions to well known problems.

> Your "two decades" maybe holds for Linux, but what about Windows or MacOS???!

Is my target application going to run on Linux? If so, then I do not care about Windows or MacOS, just like I do not care about CBM64 or Amiga.

My goal is not to write an awesome system that wins design awards in handling of obscure edge cases thrown at it in a coffee shop by hipster developers. My goal is to build a system that solves my business problem.

...and since you wrote straightforward code not relying on obscure platform-specific technicalities, even if you wanted to port it later, the port of the filesystem code was just a few #ifdef's away.

Time spent not doing our own designs (and instead spent memorizing how to use magical frameworks) is time not spent advancing our technical understanding.

That's a false dichotomy. The time it takes to "memorize a magical framework" is far less than the time it'd take to learn how to write code to do what the framework does. Consequently you can learn a framework and some other technical understanding in the same time it'd take you to only learn enough to implement your own version of the framework. In most circumstances that's actually more beneficial to do that. You'll be further forward in your understanding of the technical stuff.

You're also assuming that frameworks are written by individuals. They're not. In the case of some large frameworks it'd be practically impossible to implement what they cover on your own. You simply can't learn the underlying principles and then implement them all in code yourself.

It's definitely worthwhile learning the basics of the languages you use, and you should be working on things that improve your code and understanding as much as possible, but it's very likely in most cases that will mean building on top of someone else's existing code rather than implementing everything yourself.

These are still claims without any context. It mostly depends on how experienced the programmer is. And most importantly, note that one never needs all the functionality from a framework. Typically it's only a very small part, and often the existing functionality in the framework does not match the requirements 100%.

Very often the match is good enough that it pays off to slightly align the requirements instead of patching the framework. The amount by of man-hours poured into the very core parts of rails for example, just processing and dispatching requests, safely decoding the input from a webserver to a useful set of parameters, routing the request to the proper handler and rendering and returning the response is huge. Certainly, you could take something slightly more modular, such as padrino, but that’s still a mind-melting amount of code if you look at all the libraries and dependencies.

You could reimplement most of the basics, but that would be month or years of work and probably still buggy as hell. I’ve seen my share of “oh, we’ll just build our own framework” and they all turned out to be much more complex than the initiatiator expected.

I would think this claim is self-evident

> In the case of some large frameworks it'd be practically impossible to implement what they cover on your own. You simply can't learn the underlying principles and then implement them all in code yourself.

Not even DHH would claim to have been able to build the whole of Rails by himself.

I would disagree about that.

Learning Django taught me a lot about the proper way to things. When I first started using it I implemented a lot of my own stuff myself (as I wasn't aware that the framework had certain features). I basically wrote my own equivalent of class based views before I understood Django's own. (http://ccbv.co.uk/ helps understand them a lot)

Reinventing Django's class based views is something that I have seen in a few inherited apps (like where I currently work).

If your application is going to be something long lived and gets a number of developers while in maintenance mode, then memorizing a standard framework is a good thing - new developers should be familiar with how a framework works - as opposed to needing to spend time going through someone else home grown code.

Then there is the issue that writing your own code will be untested relative to a framework that has some level of popularity.

Documentation is likely to be better with a framework as well.

(This is my perspective coming from a Python / Django background - I have noticed that JavaScript frameworks and libraries often have a lot more problems).

The most valuable skill is knowing when to externalize your tools. You don't always want to reinvent the wheel every time you need something, when you have deadlines to consider.

Exactly this. There's a ton of gray here. Have to pick your battles as best as you can.

Another example are game engines. Hard deny the value engines like Unreal and Unity provide. They are hard to ignore and have thousands of expert hours put into them.

A good way to pick battles is not to fight most of them. There are so many solutions that don't have any real problems...

Could you provide an example of solutions without real problems, game engines or otherwise?

Basically anything overengineered? Concrete example: C++ "manager" objects. Why not finally learn how to structure applications and keep allocations and resource use in check? The program will be so much simpler, compile quickly, be easy to understand (execution threads stop jumping around like crazy), and typically have less memory leaks / use-after-free etc.

RAII, garbage collectors and other fancy inventions for freeing resources from the call stack automatically? Not needed. Use global resource managers. Globals are needed and semantically the right thing. Not a problem.

Any crazy programming language with all the features they could conceive? The one you need is guaranteed to be missing. Better, express your problem yourself and write a simple generator script that translates your concepts (mostly as plain old data) to the minimal language.

Another: XML (and even JSON), possibly for anything except true document markup. It's slow and doesn't buy us anything. Yes, there are ready-to-use parsers, but after parsing you still hold a mess in your hands (and only strings / floats). Finally learn database basics and manage data in tuples. Make a text file parser that takes simple table (= list of column types) descriptions and then parses tuples from single lines and puts them as typed data in arrays of structs, done!

How does a "global resource manager" work and in what way is it better than RAII and garbage collectors? Googling wasn't helpful.

Just global state. Simple example, a global variable that holds an allocated memory buffer. You initialize it at program startup and tear it down at the end (you can be sloppy and leave out the latter). While the program is running, you re-allocate as needed.

This is better in that it really is a very simple thing to do. There are no possible memory leaks. But what happens is much more explicit - you have precise control, and no bumpy control flow (much nicer for debugging).

But it doesn't have to be memory. If you want to look at my project https://github.com/jstimpfle/learn-opengl (careful - it's not tidied up. But I think it demonstrates my point), most resource state there is global module-wide state. I simply have init_module() and exit_module() pairs that I call from the main function. Problem solved. Not a headache at all.

Is that in support of using the Boost filesystem API?

IMO, only if you're using g++ or older versions of the standard. MSVC, clang and ICC have all supported the experimental::filesystem module for years at this point.

I don't use C++ outside of games programming, I'm not familiar with the Boost filesystem API.

It depends on what you're doing with it, I'd guess.

The amount of security exploits due to memory corruption in software written in C proves that during the last 50 years, it isn't that well understood.

Recently, I spent some time trying to run several machine learning jobs simultaneously across AWS machines. This was a fairly simple use case: all the jobs were totally independent of each other, you could run them by calling a Python function with different parameters.

I'm mostly a stats guy and not much of a programmer. I got a hacky, do-it-myself version using Python scripts up and running with about two hours of work, and learned about Python threads for the first time in the process, a tool that I can reuse in many different areas.

I then tried to do this the "right" way, which according to Amazon is to use Docker and the AWS tools ECR, Batch, and SQS. It took me about 10 times as long to get that working. Yes, this offers much, much more functionality - but most of it is stuff I didn't need. The only real gain I got was my models running about 20% faster, and the knowledge I learned is ephemeral.

I also like to ask myself, "Do I want to become an expert at working with external software X, or do I want to become the kind of person who can build software like X?"

That's great. However, how does the time to solve the business problems fit in?

That's a great heuristic to keep in mind, thanks for that.

Worth dropping a link to this Joel Spolsky article, where he discusses this concept and talks about the fact that the Excel team (in the 1990s) had their own c compiler:

https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...

I am glad I read this just now, I was jumping from hoop to hoop. and not only hoops but craft paradigms, reminds me of the Fred Brooks idea "There is no silver bullet".

Brooks argues that "there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity." He also states that "we cannot expect ever to see two-fold gains every two years" in software development, as there is in hardware development (Moore's law)

from wikipedia

Glad it helped. Brooks should be required reading for every developer: software is hard.

Sure, NIH syndrome can be rational for technology that is crucial/central to your system.

The problem is it is often used to justify re-inventing even mundane stuff. I once worked with a client who wanted to implement their own bug tracking system. The client’s main product was something totally unrelated.

For 99% of companies, including the ones employees post here container and container orchestration is a resume driven development of devops/engops/sre/seniordevs caused by improper hiring, improper vetting of ideas and personnel and fixation on cargo cults.

It is no different from cargo culting in sales where a perfectly functioning company slowly but surely building up brings in a CRO (can you imagine this showed up as a title?! ) who says "We will sell differently! Give me account managers! Give me sales development representatives! Give me customer happiness coordinators and we will sell to enterprise accounts at 50x contract value!" So the company hires a hundred people in those roles and it does look like the contracts are creeping up. So the CRO says "I know! The issue is that we are spedning too much time on paperwork. Hire me salesops! And give me all these Sales Force integrations! And special IT people reporting to me operating these new tools" So the company hires more people for those roles, burning through millions of dollars in salaries but at the end... the new customers are still just a trickle.

Eventually CRO gets fired and most of the people who got hired on that push are gone as well as millions of dollars are spent. If they spent those millions on Google ads or Facebook ads they would have definitely gotten more revenue but plain Google ads and plain Facebook ads are not sexy.

If even mundane stuff keeps breaking your workflow and demands all of your time to keep up with, then all the power for reinventing it.

All the better if it is simple.

Bug trackers are one of those areas where a lot of companies should build there own. Everyone has there own workflows and information to capture and end up either conforming their process to the bug tracker or spending more time configuring the bug tracker than they would to build a new system from scratch. Those uber configurable systems always suck to use.

Ones like JIRA can takes weeks to setup for your org and include their own query language. All of this complexity just to do something so simple is not rational.

As long as you don't over engineer it then it's only a days work to get something up and running and it's a great project for interns.

> Bug trackers are one of those areas where a lot of companies should build there own.

This sounds insane. I have not worked at a place where the workflow was so holy and important that it couldn't be captured in a near default JIRA install.

Most of the customization asks I've seen with JIRA come from dysfunctional organizations that demand new swimlanes like "QA" and "product approval" and "spec design".

I have not worked at a place where the workflow was so holy and important that it couldn't be captured in a near default JIRA install

JIRA out-of-the-box is surprisingly sane - most people who hate JIRA really hate the custom workflows their own organisation has inflicted on them.

Making a serious bug tracker is hundreds, more like thousands of man hours. You’re saying that instead of investing 50 hours into configuring Jira properly, it’s better to do that? Even with intern work, it doesn’t make sense, especially since interns will be gone tomorrow ;)

I think your either over complicating the requirements or under estimating how long it takes to throw together a system with half a dozen tables and a a dozen views/forms. Or underestimating just how far you can get with such a simple system. Then you've got something tailored to your workflow.

> Even with intern work, it doesn’t make sense, especially since interns will be gone tomorrow ;)

But there will be a new batch sooner or later that can extend and update it.

Most places only need a fraction of the functionality of Jira.

Most developers underestimate the complexity of any project :)

Especially one that becomes a crucial component of a company's workflow (as is the case for bug trackers in any properly run software company).

I would say its more of the case that developers unnecessarily make projects more complex than needed. Certainly on the last couple of projects I have inherited.

This is why BPM systems exist. To take all of those disparate “Good enough” tools that don’t fit your workflow and make them fit your workflow.

Really a tech stack that doesn’t get nearly enough attention IMO.

I don't think that's the whole picture. You could copy an interface (and even implementation) and fork it. But often with NIH we see a reinvention of a technology without even looking at alternatives; often this ends badly (in Linux land much more often than not it seems).

The problem is that reading (and learning from) code is hard. The other problem is that there is so much bad code out there (most of my own is certainly not an exception) that it gets even harder.

And it's not about the code. Writing code is easy. It's about finding the right problems first. And then it's about finding the right abstractions. The easiest way to build a clean conceptual world is to start with a clean slate and ask yourself before introducing new code, "does this code solve a concrete problem? Do I really need it?"

Unix had simple and clear ideas, and it has many reimplementations (NIH!), and most are not that bad - are they?

I gave him mine on your behalf :)

Reply on news.ycombinator.com