1. Greater appreciation for the dependency
2. Better chance of modifying the dependency when it breaks or doesn't work the way I want, because I'm at least somewhat familiar with the types of tradeoffs you have to make to implement such a thing.
3. A chance that the problem turns out to be simple enough to not require a dependency and the prototype works just fine forever.
4. Get to learn something new.
Obviously there are exceptions. I wouldn't try to implement a relational database just because I need one. But in these situations I try to take a step back and ask if I even need a full DB. Would flat files work fine for the task at hand?
My general feeling towards dependencies (in programs or life) is "do I really want to depend on this thing?" Almost always the answer should be "no". But the follow-up is "do I really need to depend on this thing?" and sometimes the answer is yes because I don't want to or can't re-implement the thing due to time or because I don't have the expertise.
Things like passing protobufs/gRPC when C structs, JSON, or CSV files will do, job scheduling systems that could be replaced by a postgres table, most Kafka applications.
I once worked with a guy who would pattern match whatever problem you had to the trendy Apache or Google library and then spend weeks/months setting it all up. He was easily the least productive person I've ever worked with.
These will always get the job done, but at great cost to future maintainers and callers. Explicitly declared interfaces that give some thought to future evolution aren't that hard, and can save a lot of pain down the road.
I vividly remember the outage I caused by changing a struct field name, only to find out it was implicitly a critical part of the API "contract" when serialized to JSON. Another where I relied on a certain field from a dependency's response, only to find it out it was populated in some situations (the examples I used to learn the interface) and not others (revealed by production). Our Thrift services do not have these problems. Anyone can look at a diff and tell a) that the IDL is changing, b) whether the change is breaking, and c) whether a field is required or optional.
C structs are so brittle they should not be used for any kind of serialisation. You're just setting yourself up for upgrade nightmares. I wish BSON or the ASN.1 toolset were easier to use for this use case.
Meanwhile, their encoding/decoding is infinitely faster than most other serialization formats, allowing you to serialize data back and forth to disk or as IPC faster than anything else.
#pragma pack(1) and byteswapping solve all this and neither are more work than specifying some intermediate format and calling some external tool in your build system. In C++ you can make types that swap automatically if required.
> And you have to keep the data in the structure itself, so as soon as you hit a pointer you need to write explicit serialisation.
I have yet to see a serialization format that does this well and quickly. Just make a new message type for it that follows the first.
You could add completely new revision of the API. But having to maintain it - typically additionally to the old revision since both sides might not updateable in atomic fashion - is a lot of extra work.
Serialization which allows to add and remove fields in a backward compatible fashion is a mandatory requirement for any bigger sized project for me. If you need super high performance maybe use Flatbuffers or Cap'N'Proto instead of Protobuf/JSON/CBOR - but I really wouldn't recommend to go for plain structs.
GRPC is designed to enable peer-to-peer communication in a microservices mesh, so it has hooks for a number of things like service discovery, authentication, health checking, retry policy, connection pooling, observability, etc. If you're just looking for a serialization format you might prefer Protobuf itself.
Lack of unit tests for serialization can be deadly for projects where the serialization format is defined implicitly via the application model. I've seen enough projects bitten by this to be convinced it's worth writing ser/des unit tests for every API object. For some reason it's a form of testing that even gung-ho unit testers tend to leave out. Arguably it should be covered by integration tests, but integration tests usually only test round-trip serialization, which rarely breaks when you're testing a codebase against itself.
But I think the issue you faced was a documentation issue as well. It should have been evident which objects defined the API. In some projects the objects have to be heavily annotated to make ser/des work at all, which makes it obvious, but when it's possible to separate concerns and define ser/des separately, there should be documentation or naming to make it just as easy to see.
Person tries to find a library that does what one line of code would do (cough left pad cough and others) and then guess what, the library fails at that thing that maybe not 90% of people need but 50% need
And that even applies to some corners of famous apps or libraries big sigh
Invented here syndrome: https://dev.to/mortoray/invented-here-syndrome-4mg8
I agree with your comment. I would add that there are cases where your problem looks really easy right now, but will get increasingly complicated over time in ways you don't expect. In this case, "overkill" 3rd party solutions are actually better. For example, you could write an HTTP client using sockets pretty easily, and it will work in simple cases. But you will, over time, run into edge cases and problems that a mature library has already solved for you.
Also, this problem reminds me of having to decide whether to solve the integral, or look it up, back in the day. Solving from scratch was always preferable if you could do it, but sometimes it was just too hard.
It’s entirely possible that the new problems you encounter will be different than the ones the mature library was designed to address, and then you end up hacking around solutions to problems you don’t have in order to fix the ones you do.
We became the third company I know about to ever solve this problem, but I had built what my colleague called “the hardest to use HTTP library ever”. I was happy to move back to AFNetworking once we could do so without a regression in performance or hitting that bug.
> job scheduling systems that could be replaced by a postgres table
I've never used Postgres in my life ever.
Django-Q (the multiprocessing task queue framework, annoyingly named the same as Django Q objects), allows this as one of the configurable brokers. https://django-q.readthedocs.io/en/latest/brokers.html#djang...
I have used nowait before to batch scheduled jobs together for a slow API call.
Moving everything to rq, and so far so good.
I've noticed that a lot of "dependency" problems on "complex systems" are usually programmer implementation problems. Changing the implementation forces you to review the original implementation (or throw it out) and fix the bug that caused you to change the implementation in the first place.
I wish you the best, but I cannot help but think you will regret that decision.
Either Mongo, MySQL or List in Redis.
Is there anything sepcific about Postgres when it comes to this? (except SKIP TABLES, as mentioned in other comments)
API is pretty small and comprehensible.
Job scheduling is trivial. Doing it reliably, tracably and in a way that handled failure predictably is really a pain in the arse.
There are zillions of ETL frameworks out there, just use those. some are unspeakably complex (airflow I'm looking at you) Some are simple, but require glue (AWS batch) some are trendy (argo on kubernetes)
However, most CI tools are effectively job scheduling systems. Jenkins works surprisingly well as a cron replacement for >100 machines
What about null?
JSON Schema – a whole topic unto itself – gives you further control over locking down how this can be represented (e.g. field MUST be present but can be string matching this regex or null).
Shameless plug: https://github.com/RantyDave/cppbor
Other things I take into account are how important the X is to the business. My company's product parses a very strange industry-specific file format. There are some commercial parsers and some open-source ones which do most of what I need but I opted to roll my own as working with this file format is very important and I can't risk license concerns.
Also, it might not be very nice but I don't want to help my competitors.
By expending resources to build "X", your company is now in the business of building "X". Is building "X" a business the company wants to be in?
The story behind Amazon building AWS for internal needs, and then selling it externally, is an exemplar of this thought process. They were already in the AWS business... They just decided to package and sell the product to others.
He wrote an article on the topic: https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...
Let’s imagine you’re a software shop that sells a SaaS app. Something like Slack as an example.
Should you roll your own dedicated compute for your SaaS app or use AWS?
Should you roll your own web framework, or use Rails/etc?
Should you use AWS for file storage of Slack media files that are uploaded, or roll your own dedicated storage solution?
Edit:
I have the upmost respect for Joel. His company created their own Programming Language because they felt it was so core to their business to do so [1].
As did Facebook did with Hack.
For example, maybe the Slack like app wants to be able to upload, distribute, and archive huge computer files. Everyone has tried to upload a file to an email, slack, or discord message and was told the file was too large. Our platform is going to be the platform for sharing 1TB sized files for collaboration purposes, with the goal of supporting 10TB in summer 2020 and 1PB before 2022.
The company should 100% make their own network protocol for transferring files.
* Is your business developing and maintaing a general-purpose web framework? No, so you pick one off the shelf.
Once you have your product, have proven your market and have a healthy customer base you can start thinking about optimizing your verticals.
* Maybe your cloud hosting bill is enormous because of your workload and it turns out that colocating at a local data center will be a lot cheaper so you migrate.
* Maybe you realize that none of the current web frameworks make your devs feel productive and they keep having to fight the tooling to accomplish what they want. If you find that you're carrying a lot of patches for your framework maybe it's worth your time to have someone working full-time to upstream them. If you think it's completely lost then it might be worth it to hire a team to work on building a new one.
So should Dropbox roll its own storage solution, or use cloud hosting - based on your statement?
- Do you need the get results now, or long-term maintenance is more important?
- Are you going to roll your own internal tool or publish it as open source?
- How responsive are the developers of the original library? Did you attempt to ask them for help/consulting/etc?
- Can you wrap the library so that you get 80-90% of the work out of the way? Can you fork it so that you get 50-70% of the work already done? As everyone mentions here, licensing is important for this question.
- Do you have people who have created and maintained libraries before? Do you know what it takes to publish your own library?
For example, I've written hundreds of small libraries and projects in Javascript, so I know the tools, common issues and details of creating a new library. So for me, creating a new one is fairly easy. For someone who has written the same amount of code in their career, but mainly focused on a single codebase, the cost (both short and long-term) of creating a new library is a lot higher.
- How stable are your needs? Will your ideal setup be the same 6 months from now?
- Is this a core piece of the business or a peripheral concern?
- Which is more valuable: developers that understand the underlying theory, or developers that are familiar with the preexisting library?
- Will the act of rolling your own library make your team more valuable? Will it make them too expensive?
- How will this choice affect developer retention? Your ability to hire new developers?
You're not getting that with django, or ror, or ... And no, caching will not get you there, especially not if you take every framework's approach to caching: always set cache headers to never ever cache because it might be dynamic.
The thing is, if you expect to really grow, you're going to have to roll your own, as that's the only way you'll ever get it really fast, really doing what you want.
The frameworks you mention are great prototyping tools but as a community, we’re missing the knowledge of how to take a proven prototype and continue improving the quality rather than bolting on questionable new features.
No, we aren't. We know how to do that, and can do it when we want to. In software dev we rarely want to, because (loaded use of the qualifier “questionable” aside) product teams tend to perceive (not entirely inaccurately, though sometimes the particulars are in error) market demand for additional features. (Also, what physical products are very often optimizing is unit production cost for equivalent products, not quality. But software already has an essentially zero unit cost, so there's essentially no gains to be had optimizing that.)
Are you saying that developers should be kept ignorant and incompetent so that they'll have no choice but to stay at your company writing poor-quality software?
On the other hand, withholding opportunities for growth is also a way to frustrate employees and cause them to leave. Know your people, and take their changing needs into consideration when planning business strategy. There’s lots of strategies that can work here, but not all of them are right for every company. A few off the top of my head:
- Build a training pipeline so you can regularly hire promising young developers for cheap and help them grow into a more senior position elsewhere.
- As your developers improve, be prepared to give raises and promotions to reflect their increasing value.
- Hire experienced developers with the skills you need at a fair price and offer stable employment instead of fast-track growth.
E.g., you need a web-framework to program your e-commerce site, The web-framework is not a core business need (you are just selling shit online), but incidental. Therefore, don't roll your own.
You need a web-framework to implement a SaaS app, and this SaaS app is your main business (think Canva, or lucid charts etc). Therefore, you should roll your own to suite your SaaS app, and make it fit intimately with your business needs.
in that case, their business need is to have fun! And i would say writing it all from scratch is most fun of all.
For example, I lead a project where we wrote a driver instead of using an open source one. Writing our own manged risk, because the open source ones were unstable or had data integrity issues. Overall, the "cost" of "writing a driver" was about 5-10% of the total effort, once we consider integration, QE, installers, signing...
The thing with 3rd party frameworks is that they aren't built for your requirements. So, if you don't "roll your own X," you need to make sure that the X you choose will meet your business critical requirements early in a proof of concept.
Also, if you are going to "roll your own X," it's worth it to do a few POCs with other Xs. This way you can learn how to make a better design, and what to encapsulate in your X.
(In my case, the 3rd party drivers encapsulated poorly. Most of my effort spent writing our driver was things I'd need to figure out anyway, because the other drivers didn't encapsulate basic details that I didn't care about.)
---
Other times, it makes sense to "roll your own X" to keep things simple. ORMs are a great example. Their leaning curve can be higher than simple queries and boring code. If you only have a few queries, why bother adding something that's just going to get in the way?
Which gets to: Do not use frameworks in place of design patterns.
Say you are working on a FOSS project to replace something and the one of the dependencies doesn't work out. If you have the project capacity, it might add more value to roll a replacement.
Example: say you are writing a metrics collector, and you need something that can run inside a container because you want to deploy it as a sidecar. A dependency you use needs root network access for some reason, and you can't really bypass it using CAP_'s because it's hardcoded to look for UID 0. You can patch it if the code allows for it, but if it doesn't, you still need a fix so you might as well roll your own.
On the other hand, say you are in a commercial setting, then value is probably not added by rolling your own until it's either freeing up resources or addressing a business need.
Example: you are running a bunch of APIs behind a gateway, and the gateway exposes those APIs on a subdomain based on their name. But you kinda want them on the path of a single domain instead. You could do an ugly patch with a reverse proxy in front of it and rewrite all requests, but at that point you might as well implement a replacement gateway that does the same thing. While it might be a slightly bigger effort, it both frees up resources and gets a business requirement resolved.
There really is no one-size-fits-all rule, even if you take licenses, quality, maintenance, sourcing, cost etc. into account it still won't cover all cases. Say you are a small company but your software is used on billions of devices. Rewriting a part of a kernel would give you a few cycles less CPU time required for a task that multiplied by billions saves a lot of computational power, but you, as a small company, don't have the resources to do so. Even if the license, quality and maintenance is right, you still wouldn't rewrite something like that.
Unless you pay the cost to keep up with the industry (or switch away from your custom solution once others meet your needs), you will fall behind. But keep in mind that others are most likely not paying this cost, so they're investing in features, continuing to outpace you in areas that matter.
In short, only implement something yourself when absolutely necessary—for example, when it's a core competency, or when no current solution exists. Even a de facto solution that's suboptimal tends to be better than your own, because it will be documented, new contributors won't have to train on it, and it may improve over time. If you have to switch away from it at some point, there may be tools to automate or simplify that transition.
The smartest decision for things outside your core competency is "do what everyone else is doing." Focus on innovating on the things that matter instead.
Outside of licensing. It is usually based upon the needs of the project, if I can find a library that does 80% of what I need and I can add the remaining 20% and I am happy to live with that code then I'll definitely use a 3rd party. In general I favor those libraries, but just so many times the licensing trips the flag that it isn't worth the risk and adding a few extra days or even weeks to a project is well worth the time.
One other point, I also am really big on logging/metric collection, if I need to go in and instrument an entire library then it adds another level of work so I start thinking about just creating our own. But not all libraries do need detailed logging and metrics (although IMO most do).
"This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file." from https://www.apache.org/licenses/LICENSE-2.0
I'm not a lawyer and not your lawyer.
Over the years I have sought council and been advised by multiple different attorney's that for a closed source commercial product to avoid certain licenses, like GPL and others. Apache in fairness is not one of them, my bad. MIT and Apache are the two we have used and included in projects in recent memory.
In the end I am not a lawyer either and my core point still is the same, a major question for whether to use a 3rd party library still is around what license the library is released under. And it is usually best to get advice from an attorney for any of the more complex license models.
The entire point of the GPL is to spread the adoption of free as in freedom software. If you aren't willing to share your project with the community you are not allowed to use the community's work.
An example I dealt with a few years ago, I was working on an embedded project and the associated desktop software that the original developer utilized some GPL licensed open source libraries, compiled into the binaries (both embedded and desktop). When I did a license review and discussed it with the attorney it was deemed that GPL licensed components could not be included due to the licensing terms and the way that product was distributed along with the closed nature.
In general I follow the rule, and have been told multiple times, that GPL (any version) with rare exception is not to be used in commercial software (including API's and server based products) unless you are strictly dynamically linking to the library. But if you are compiling it into your commercial product and especially if you are making alterations to it and compiling it in you are in violation of the license terms (and definitely the spirit of the license). Of course, if your product is open source, then that is a totally different story.
Even more open licenses like Apache and MIT still have copyright notices you need to comply with at minimum, so all that has to be taken into consideration as it needs to be added to product manuals, about pages etc otherwise you are in technical violation of the terms. Hence, the license terms of any library should be a key consideration on whether you include it into your project.
People tend to understand open source software as software that is free to use however you want. This is not what GPL software is though. It's entended to be an ecosystem of software that is open and free to use as long as you are also willing to share the software you create using the ecosystem.
This attitude has basically eliminated unhappy surprises from external code for me. This is probably a disposition earned through developing mostly in the JavaScript community (I am now the polar opposite of the NPM duct tape developer, and I prefer flat dependency trees).
I view lines of code that other people have to maintain as less costly than lines of code I have to maintain.
Initially... but they can break your lines of code in unforseen ways in the future, and when that happens it will be you maintaining those external lines of code, because no one else cares.
To make things worse external dependencies are often highly generalised and complex because they are trying to cater to more than your project, as a result you may incur the cost of those lines of code even if your project doesn't need them as you have to wade through them and accommodate them when things go wrong with the parts you care about.
I'm talking worst case, but the point is, you want to only use dependencies you trust, which usually means minimizing your total number of dependencies also.
Code reuse is initially harder, as you have to figure out the existing system, how to integrate with it, how to frame your problem in its terms, etc. This is usually slower than just writing the thing. It's over time, when other people/teams are contributing to and fixing bugs in the shared component, where you and the company are saving work vs. each team with a similar-shaped problem learning the same lessons independently.
My gut says always: most of the times turn-key stuff has weaknesses and good libs with perfect apis, well maintained and with a great community are rare but they are there of course.
Hard question since what the gut says is more fun while the alternative is just about gluing libs together. But using libs and once they don't fulfill your needs build yourself is the right but more boring way to go.
I'm saying it's dangerous to have this line of thought that the "correct" way to build a system is with a mish mash of third party libraries and that doing any non-glue coding yourself is only for fun. You're insinuating that the decision OP is making is between efficiency and fun, whereas in reality it's an optimisation problem for efficiency that a lot of people mess up because they don't understand one side of the equation. (and if OP wants to have fun and roll his own that's great but it's another case entirely, I'm assuming he's not asking HN for permission to do that).
I use NPM as an example because for JS devs in the places I visit it's becoming a cultural thing, which IME is not the case in other communities. I don't see as many python devs entertain the thought of maintaining something for years with "crappy libraries" in it for the sake of saving "days" of work. But in JS land that attitude is all over the place and the phrasing of the first bit of your post reminded me of it, so...
Libraries also tend to be general purpose so up to half of the code in a library may not be pertinent to your usage.
I wrote an app that used a lot of Oauth and ended up writing my own instead of using the excellent Passport library. It would have been easier to use that, but I gained an understanding of Oauth that I would not otherwise have, and my code is small, understandable and easy for me to maintain and modify. Just my 3 cents worth.
For work projects, I like to look for the best available FOSS option and then try to help out on that project so I can feel a bit invested in it. This helps calm down my hacker urges to build my own thing from scratch :)
I've had a lot of situations like that and been accused of NIH syndrome, but what is the alternative…not doing something at all because nobody else has built or released something that can be used? Every library or tool in existence started out because somebody wasn't content with the status quo enough to bother inventing a new thing. This is the only way the state of the art can progress.
As an example, if I'm building some new piece of SaaS that has an API, I could write my own JSON parser, my own HTTP server, my own network stack, my own OS. Heck, I could eventually learn how to make my own servers, my own processors. But I'd be comfortably dead before I learned how to do all those things as well as off-the-shelf components deliver for a SaaS API.
Instead I'd rather spend my time on activities that are more valuable to my users. If one of those things turns out to writing a custom JSON processor, sure, I'll do it. But if the only purpose in me doing that is to learn something, then I shouldn't be jamming my educational experiments into a production system that doesn't need them. I've had to clean up too much of other people's flavor-of-the-month experiments to want to inflict that on whoever maintains my code next.
Writing something from scratch just because you want to understand it is fine. But you should think long and hard before actually using the new thing you just wrote in production code.
1) If it doesn't work or works poorly.
2) If it is trivial to implement.
3) Licensing problems.
4) If the implementation is overly complicated.
5) The (framework) vendor have a habit of updating and breaking your code. This happens a lot with the Angular team. There is nothing like the annoyance of an Angular update and they have broken some feature you are using and you have to update code that was working absolutely fine. Since then I have avoided Angular entirely.
Things I am less strict on:
1) The project is EOL. It may work perfectly fine. This happens to open-source software as well. Typically this means there is a better alternative out there.
2) Awkward syntax / api. You can normally just wrap this.
3) Maintainers are difficult to work with or don't fix bugs.
Some libraries have hundreds of megabytes of their own dependencies hard to remove.
Some libraries have very low code quality, makes it expensive to modify.
Some libraries are very hard to build, e.g. I've never managed to build Skia. When you can't build it you can't adopt it.
You should also pay attention to license, there're GPL licensed libraries out there, if you'll try to use or adapt them you'll have to open source your complete software under the same license.
We needed a sensor data capture platform at work. There were a lot of odd requirements, like consuming data from embedded hardware, severe bandwidth restrictions, and high-precision (GPS) timekeeping. We hacked something together in python + flask in 2 weeks, and it was easy to experiment on the analysis side by just writing another flask route and generating some html with plot.ly and leaflet.
We're also evaluating thingsboard and the elastic stack, but setting those up, experimenting, and extending them was slower, less fun, and harder to debug. At some point we'll do a shootout between all the platforms and pick the winner if this is something that gains significant users, in which case the community and active support of the platforms will be a big advantage.
Then if it's something that doesn't really matter and the library code is pedestrian (a logging library), I'll use the library.
If it's something that matters a lot and it's expert knowledge and is a large body of work (a complex math/compression/crypto library), I'll use the library.
The core product works fine and has little maintenance which is nice but its cost scales badly and takes every bit of my budget now and the fancy features never work or can actually be used in production. If I had that budget instead we would have something far more suited for the company but seems like few managers will take that risk unless its the core function of the company.
Also the mostly relevant xkcd: https://m.xkcd.com/1205/
If there is no Z, don't try to find the needle in the npm haystack of poorly maintained libraries with tons of needless sub-dependencies. Unless the problem domain is very beyond your capabilities, just roll your own.
- you are on your own and don't care about the future maintenance cost of your code
- you want to learn
- the available solutions are objectively crap
Invalid reasons:
- job security
- you're a control freak
- you think all code you write is immediately better than other people's battle tested solutions, aka inflated ego.
- you want to be seen as the maintainer of some prestigious project rather than as the contributor to someone else's prestigious project.
An example might be analytics. It's good to know that you can drop in a third-party library and just go, but you need to make sure to look at how you can move your data off of that platform and use it yourself. You also need to consider how much of a time investment switching off of Y would be. How much effort will rolling your own be? How much effort will rolling your own after using Y? How hard will it be to swap Y out for Z?
Another classic case here is web hosting. I use GCP. I've used AWS. I really don't want to, but if Google or Amazon kicked me off/out/raised the prices too much, I'll host from my own hardware, and go to Fry's if I need to get more hardware than I have. I really don't want to, but I will.
1. You don’t want to take the time to learn an existing product
2. You’re not going to properly code and test your own solution
3. You think your product is too unique to use an existing solution
4. You’re going to leave features out that you would have gotten for free
Existing libraries can be very complex. Spending time on them is not guaranteed to pay off. When a library is marketed as a framework, it’s a red flag, such libraries often come with a strong opinion how the software should be designed.
> You’re not going to properly code and test your own solution
This article is old, but still good: https://www.joelonsoftware.com/2002/05/06/five-worlds/ Different software needs different tradeoffs for quality, budget, and the rest of them. If you’re working on a tool you’ll run only once, grab the output data, and forget about the tool, you don’t need neither proper coding, not much testing.
> You’re going to leave features out that you would have gotten for free
Are you going to need these left features? For many real-life libraries, like https://www.boost.org/, you only going to use small subset of these libraries. The features you’re not using aren’t exactly free. Inflated binary size, slow compilation, tons of dependencies, complicated build setup, complicated deployment are quite common issues.
A programmer who used to work here, wrote his own PHP micro-framework, instead of using one of the popular, standard ones.
Instead of just using one that already existed, and taking the time to learn it, he used his own. He may have done it partially for the experience or his own desires (he was probably actually bored because we have a bad manager, but that's another discussion)
We regularly find and fix bugs with this PHP framework. Had he chosen a mature, community-supported product, this wouldn't happen.
There are incomplete features you would have gotten for free with other frameworks. It has maybe, 20% of the features other frameworks have, even though it has much of what we 'need'. We had to write our own password-reset email code, for example.
1. When they started, pre-existing frameworks weren’t such popular nor standard.
2. When they started, the functional requirements they had were very different from what the product grew eventually. Many software projects start small and grow.
3. Couple times in my career I worked in projects where management didn’t believe in functional specs, the requirements changed dramatically in the middle of development. With large libraries, some changes in architecture are handled by replacing the library with another one. Quite expensive to do. An in-house library can be more flexible, just because it’s less code.
I’m not saying making in-house libraries was justified in your case. Maybe it wasn’t and that developer did it mostly for fun. Just the trade off is quite complicated, and your reasons are not necessarily that bad.
In many cases, I think it’s technical management’s job to make the decision, i.e. evaluate alternatives, maybe allocate resources to make a prototype.
Also, from the management standpoint, if a developer can write their own frameworks, there’s very high probability they can use existing ones. Maybe it makes them slightly less happy i.e. need to pay them more to compensate, but they’ll do the job. The opposite is not true, there’re people skilled at combining libraries and copy-pasting third-party code who can’t write their own code.
For example I would much rather write 200 lines of my own code to deal with user authentication (leaning on my framework or a lower level library to deal with the gory bits of setting and deleting sessions), vs. pulling in some higher level authentication library that has 3,000 lines of code but tries to be so generic that it's a jumbled mess which is difficult to customize.
Basically the more important the feature is for my application, the more inclined I am to write my own code. Although I may at times draw inspiration from other libraries that don't quite solve my problem but helps me get closer to what I want to do.
From scratch only if I'm doing something truly unique. New algorithm/protocol. Unpopular language I'm forced to use is missing something I need. Existing solutions too slow or buggy. Maybe a couple times a year I get to make wheels.
NIH is a huge problem at most (all?) software companies. At my current place I spend about 1/2 of my time dicking around with crap internal tools and libraries.
It's rare I write anything that's not glue code. Makes everything boring but I don't get paged often and we have relatively few bugs.
Please don't make another build system. Please.
https://en.wikipedia.org/wiki/Polyfill_(programming)
In web programming, it might sometimes be useful to write your own polyfill when you need to add a single feature for backward-compatibility reasons. Well, a third-party polyfill library that provides that missing feature perfectly might be available. However, the library is too big (with respect to file size) or adds too much complexity during the build process (by pulling in lots of other dependencies).
Which do you think will take more time to develop and maintain: the wrapper layer that adapts the existing library to your needs, or the ground-up homegrown implementation? Your estimate of the answer here (along with the risk calculus of the external dependency and the reliability of it's maintainer) are the key factors in answering the question
If it's a feature that can be cleanly added into the architecture - i.e. it's just filling in a gap.
But if you have to modify the whole thing in a way that doesn't quite fit... it's a nightmare. Even with your own code, it's simpler to start from scratch with a clean slate.
Also Alan Kay's extreme position: "those who can create their own [libraries], should"
The moment you fork something, you've created work for yourself when you want to upgrade it.
That may be fine for a prototype. But I've seen it done on software that lasts 5-10 years and those are always the things that turn into maintenance headaches and blockers to unrelated upgrades.
But in general, if X is not part of your core business, only as a last resort.
99% of the time, even if there is a suboptimal library, as long as its actively maintainted, you're better off extending the functionality, than re-implementing.
There are lots of very good reasons to prefer building something. First, the design of the code will fit your use case more or less perfectly. Unless you get it wrong, in which case you can change it relatively easily. If you use an off the shelf library/framework, you may need to jump through hoops to use it. Those hoops may result in other hoops, which result in other hoops, etc, etc. In the end, you may introduce nearly as much code complexity adapting your code to someone else's design than you save by using their library/framework. (NB: You could just use "convention over configuration" ala Rails, which is another way of saying, "Do everything my way and you'll never have any conflicts" ;-) ).
Second you have control over the code. If you use a library/framework, it may change over time in ways that don't fit your project. You are stuck making the choice of maintaining all of that change or eventually suffering from bitrot. A good example is if your library has another dependency. It replaces that dependency with a new one which isn't supported on your system.
Related to that, you may decide that you need new functionality that the library/framework does not give you. If you were working with your own code, it's easy to add that functionality. If you don't, you need to either maintain a series of patches, or try to get your changes merged upstream. Some projects are easier than others to work with.
When I've decided that I'd rather not roll my own, if possible, I look at the alternatives. The first thing I do is read the code. Can I understand it? Will I be able to fix bugs if I need to? How much of the code is related to what I'm doing and how much is unrelated? How will the structure of the code affect what I'm doing? Are there any controversial dependencies?
Then I look at the community. I look at open issues. I look at how discussions progress. I look at closed issues. Do people get yelled at for asking questions? Are suggestions for improvement valued? I look at open and closed PRs. How easy is it for an outsider to make contributions?
After that, I make an estimate in my head of the maintenance cost of code I wrote myself vs the maintenance code of using one of the alternatives that I've researched. Usually if we are talking about 1 week of work or less rolling your own wins out (there are exceptions, though). If I only need a handful of lines of code, I'll frequently make my own derived library with those lines of code (For example, I do a fair amount of Ruby code and there is useful code in Rails... But I'm not going to grab ActiveSupport just because I want stringify_keys or something similar).
Finally, if a library is important to my code, I'll usually make an adaptor for it. Instead of calling the functions of the library directly, I'll make another library that wraps it. That way if I run into problems with the dependency I can swap it out without much difficulty. Of course, for frameworks that doesn't make sense because part of what you are buying with the framework is the design.
On some teams it is easier to say "the wise developers of leftPad think these are the hoops we need to jump through" than it is to present your own design that has strengths and weaknesses. On others they will walk miles through flaming coals just to avoid the downsides of someone else's designs.
I honestly have no idea which group is right but devs seem to fall into one or the other. I myself started in the first camp but am wandering toward the second. Sounds like you went the other direction.
If you decide to open source it then kudos to you. If you don’t, I won’t think less of you.
1) Encryption 2) Date/time math 3) Distributed locking/concurrency primitives
All of these are of sufficient complexity to virtually guarantee that you’re going to miss a score of subtle edge cases, and the consequences thereof are likely simultaneously pervasive, silent and catastrophic.
I mean, obviously, someone has to code crypto, or else it wouldn't exist. But, whenever you try talking about the topic online -- whether you're doing it professionally or for self-education -- a probably well-meaning chorus of spam comments warn against doing anything.
I suspect that some crypto-folks have given up rebutting the don't-roll-your-own stuff and just down-vote reflexively.
I still think that ties into the original question, when does it make sense to roll your own? If it's not part of your core business and doesn't give you an edge, then it never makes sense, unless you're doing it for the fun of it.
You have millions in the project and a dozen PhDs working with you, and you've already authored public papers demonstrating the weaknesses of things previously believed to be strong.
If you are not at least one of those things, it's malpractice for you to even consider letting your employer pay for your crypto "invention"s. That's like a nurse building her own OR and insisting the hospital let her cut people open in it. No. Don't. Stop.
Generally X existing things are going to be way too complex and heavy. Implementing your own allows you to avoid all that. But of course this only applies for individuals. Anyone stuck as part of a company or institution won't have the choice.