The art of over-engineering your side projects
elsyms.com
elsyms.com
For my side projects, I actually want to go down rabbit holes. I will try the same thing 7 different ways and benchmark the hell out of them. I'll read all the documentation and every blog post I can find. Learn a new language just to see if its features make things ever so slightly easier. Learn a new framework, write a new framework, throw it away and write a new one.
The point isn't to ship code, it's to learn. Honestly I don't care if it ever ships. And even when I do release one, I don't care if nobody uses it except me.
#1. One that I am very clear about how to implement. One that I want build, ship and grab users as fast as I can. The real goal of the project is to get it out their in the market and make a tiny dent in the universe ;)
#2. One that I am not very clear about what it's gonna become. This one I want to take it slow. Experiment. Play with tech that I haven't had a chance to flirt with. The goal is to enhance my toolbelt and learn new concepts/tech.
Clearly keeping these projects separate helps me keep things clear. I quickly take decisions and move fast with one. I play and experiment a lot, no matter what the outcome is, with the other one.
I have no doubt that this effects the side projects developers choose, or how they present them. Places like HN will always to a certain extent push people to "launch" and monetize their projects.
Whenever a post from one group (such as the article) comes out, a post from the other (such as OP) will quickly appear. And everyone loves it :)
For what it's worth, I think that both perspectives are valid. It depends only on your goals.
I've noticed that people who can do this are disproportionately better at achieving their goals than those who can't, especially in engineering organizations. Exceptions of course for zealots and revolutionaries.
This usually only happens when one begins to make personal insults, i.e. "only a poor programmer would think that!"..
I'm not sure if persons with the skill are more disposed towards studying philosophy or the study of philosophy prompts the development and demonstration of that skill more, but hang out with the philosophy kids and you might come to refine this skill by social osmosis.
People rarely switch from one to the other. It's part of our core ideals.
However, there is a large, vocal, percentage from both these groups that will always voice their opinion against an opposition opinion. It's not a competition and there's no correct answer. It's purely subjective.
Or else.
War begins!
A writer can have lots of poetry, personal expository essays, journals, unfinished drafts lying around OR s/he can "ship" a best-selling novel, publish a high-traffic blog or something.
Similarly, a painter would experiment with lots of techniques, colors, textures "on the side," again having lots of personal work lying around, yet would probably treat something s/he would put in an exhibition differently.
In the first case, you are experimenting, discovering, breathing; while in the second case you have to think hard about your audience, your readers, or your customers.
It's built-in.
There are also writers who can do all of these: my personal favourite is Robert Graves (possibly because once many years ago I got him to autograph his Selected Poems and his memoir Goodbye to all That).
Yet it would not exist without the second group because it was started by an investment fund/startup incubator.
I agree with you, I love both perspectives and the passions they ignite. I am squarely in the first group, and I love reading about the real life realities of product design/management, pitching, etc.
I'm somewhere in the middle.
> HN is made up of two groups in ever constant warfare, the yin and yang, as it were-
> the group who thinks that programming should be fun, experimental and enjoyable, and
> the group that thinks that programming is a means to the end of getting a product to market and nothing more.
> For what it's worth, I think that both perspectives are valid. It depends only on your goals.
Turns out I was wrong in some earlier maths, and the curve I wanted to find the distance to wasn't an ellipse. I was able to generalise the method for arbitrary curves, and now I've solved my problem!
Relevant blog post: http://wet-robots.ghost.io/simple-method-for-distance-to-ell...
OT: I had the problem as part of my 1st day job in 1990 on Parasolid.
IIRC, there was more than one way of converting the problem into solving a quartic polynomial, which is solvable by radicals and so doesn't need Newton's method.
If the point is inside the evolute of the ellipse then the quartic has four real solutions, which gives two local minima and two local maxima of distance.
If the point is outside or on the evolute of the ellipse then the quartic has two real solutions, which gives one local minima and one local maxima of distance.
Having said that, all methods using quartic polynomials could be unstable if the ellipse was pathological, for example very long and thin. Maybe that would be a time to revert to Newton's method.
I try to avoid using quadratic solutions in code because of numeric instability issues. Granted I never implemented the direct solution, but the number of opportunities for catastrophic cancellation gave me shivers.
One approach is to use the rational quadratic parametrization of the ellipse and substitute that into the squared distance function to get a quartic.
>I try to avoid using quadratic solutions in code because of numeric instability issues.
Sensible!
Back then the quartic solver had been written already and extensively tested by someone else - probably for line X torus intersection.
Even so there were numerical problems introduced because the quartic was in power basis and so the coefficients didn't have geometric meaning. I guess phenomena like Wilkinson's polynomial could occur. [0]
If I was doing it today, I would probably proceed as follows:
1. special case for the point exactly on an axis.
2. otherwise, choose the quadrant containing the point and represent that as a rational quadratic Bezier curve. This parametrizes the quadrant from 0 to 1 and is guaranteed to contain the global minimum.
3. substitute the quadratic Bezier curve into the square distance function to get a Bernstein basis quartic polynomial.
4. differentiate to get a Bernstein basic cubic.
5. solve the roots of the cubic numerically in the range 0 to 1: either Newton's or clipping or some other hybrid numerical method. A good reference is [1].
[0] https://en.wikipedia.org/wiki/Wilkinson%27s_polynomial
[1] Shape Interrogation for Computer Aided Design and Manufacture by Nichola M. Patrikalakis and Takashi Maekawa
How the heck did "" get in there?
> 5. solve the roots of the cubic numerically in the range 0 to 1: either Newton's or clipping or some other hybrid numerical method.
Is there a reason not to use the cubic formula in this situation?
Something like
(E(t)-P).(E(t)-P)
>Is there a reason not to use the cubic formula in this situation?
Numerical stability in edge cases, which will occur for code exercised as extensively as Parasolid's.
I guess you could a Bernstein version of the cubic formula, though I've not seen one derived in the wild. It would be nice to have as well as Bernstein quartic formula. The Bernstein quadratic is easy.
The objectives for a side project are subjective, and so they should be. However I doubt most people truly have no intention to ever ship a side project or "don't care if nobody uses it", as admirable as that would be.
I look back at my naive, recently graduated self and remember how I truly wanted to ship my side projects but loved to play around with new toys so much that I rarely shipped anything. I wish I had read a few more articles like this that would have helped me strike a balance between trying new things but still having enough experience in the stack I'm using to actually ship.
In short, if you have a great idea you want to ship then deprioritise learning to stand the best chance of getting it out there!
But some projects are more innovative or productive than others. Maybe you should abstract those projects away from eachother? You could put them in seperate classes. You could have the productive project use the innovative project.
Because it was a hobby project, I went completely mad on it. I built my own SPDIF interface, from scratch, using an FPGA. I let the hardware broadcast the sound frames over Ethernet. I also re-implemented Sony's proprietary consumer electronics control protocol, in hardware as well (a microcontroller could have easily done that instead). I greatly overengineered a circuit board, despite having a functional "test setup" already that did the job more than well enough. I wrote largely unnecessary client software, using language extensions I've never used before.
The task at hand could have been solved with buying a cheap USB sound adapter from eBay and some manual work instead. Had I gone down that route, I probably would have completed the task in a fraction of the time that building this project took.
But I had so much fun along the way and learned so much about electronics, FPGAs, SoCs, static typing for python etc., all because it was a project for me and nobody else.
One big lesson I've learned, though, is that having an actual task you want to solve greatly helps you in finishing your project. I ran out of steam a number of times and the project fell to the wayside, but because the underlying task was not fulfilled yet, I always went back to it and quickly got completely focused on it again. Projects that I only did for playing around and learning most often got abandoned.
The main reason I've wanted other people to use the things I've made is to learn more about what I've built and how it could be better.
Now, if you personal project is intended as a gateway to your bootstrapping of commercial venture you _must_ make completely different choices in what/how you build (lean, and minimum useful/viable project) tec.
But there are also some side projects where you just want to get your hands dirty with some different tech, if you do deliver something then you can showcase it on your portfolio. I know when the app store gold rush first started a lot of people did this as experience to get iOs dev jobs.
More often than not the skills I learn doing this kind of thing end up being very useful to me in the day job.
I'd like it if the side project was successful enough to justify all of that yak shaving, but I don't really expect it to be and I'm not that fussed when it isn't.
But if your primary goal is that your side project should become your day job then the article is pretty good advice. Leave your yak unshaven :)
99% of the time, if you want to go pragmatic, you should buy a ready-made solution (and then possibly adapt it to your use case). It's efficient (especially if you need the problem solved to make money) - but it's no fun.
Considering this is only a side project, you have a high likelihood of abandoning it out of boredom, lack of traction or simply burn out. However, if your side project is "interesting" to you because you are experimenting with various tech, you come back to it because it is fun and it will keep you motivated.
It is totally cool to build something to learn and throwaway, but why not build something to learn that might also generate some income in the future?
That's not a problem, that's the goal!
I can't imagine not over-architecting this kind of stuff. I know I'd be in a world of pain several times if I hadn't.
This article feels like it's written from the perspective of using a side-project to bootstrap a business rather than as a learning experience, hobby or toy.
My side projects are absurdly over engineered and I get nothing done. But I learn what not to do when it matters.
I had a lot of fun doing it, and indeed learnt to not do it in prod, the compiler errors are a nightmare to read and good luck maintaining that in the long run.
There are two types of side-projects though: one such as yours, where you try and understand a concept or engineer something just for the fun of it.
And there is second one: a game that will mature into something; a hobby project that will actually develop into a company. I think it's good to disambiguate these two as the OP was more about the latter where you may never get to a point to see it run or build let alone flourish into something meaningful - as overengineering kills the fail fast fail often mentality and the project itself.
Still, I completely agree with the point that "experiments" should be kept off the critical path as much as possible. It all depends where your risk/reward is, though. The people that took the risk on a stack and got rewarded with success are highly visible. Just as many took the risk and failed. (Probably more?)
Exploring new technologies that could be beneficial (or harmful) to the company shouldn't be a side project. Perhaps we can shift the thinking from "this person doesn't have time for side projects" to "how can we make exploring new technologies a part of the job?"
Start the day with a "not really a standup", where people can quickly explain what they want to accomplish in the next four hours.
Once a month, you spend a couple of hours of the time demoing the work you have done for each other, explaining the technology and what you have learned, and then discussing and evaluating our individual technical weaknesses and deciding what to do next. This rest of this monthly meeting could also be spent improving social coding skills - code and commit hygiene, team processes etc.
Not only would this keep people technically sharp but it keeps them from playing with toys in production and breaks up the work week, while still fulfilling company goals.
We know about lots of best practices but if no one is advocating for them at their companies those practices we'll just be shining cities on a hill that we look to from a deep, dark valley.
Last week I wrote a utility in Python to parse a configuration file that we had so far been editing by hand, or with bad, slow and cumbersome tools. Now our Linux guys can make the changes is seconds across multiple machines, and the parser is much safer than the bash hacks we had been tempted to use before.
Point being, development time can be great! More companies should embrace the idea of letting their employees improve their own workflow. Some of our best ideas were just that: a little spark that an employee had the opportunity to actually explore.
Part of that is support (though about half of the jobs I've had involved little to no on call duty), part of it is training one self, in addition to everything I have to do during the day.
Let's call it discretionary overtime baked in pay since we're usually salaried.
Lately I'm pulling back from technical deep diving at work. A lot. My role as an enterprise architect (bleh, don't judge too much plz) means I necessarily have to shift to big picture guff and stop spending time on the details pieces. In short, I don't really get to engineer as much as before.
Of course, I'm an engineer at heart so the idea of ignoring the details is anathema and it frustrates me. When I'm deep in whiteboards, board meetings and shitty powerpoint at work, the side projects and home lab become my release valve. It's a coping mechanism, I think.
Likewise, when I'm sucked into low-level grind at work which happens from time to time, side projects stagnate. I don't need them for a while.
Most of the stuff I've seen done at typical development companies are intellectually interesting maybe to people who really started learning to code on the job, and generally don't have much experience. They're discovering new things every day, so it's fun! However, if someone has already few languages and various types of projects behind them, then the job becomes much less technically challenging and much more repetitive.
One thing I do not do, is just rebuild something someone else has already done in a new "stack", just to see if I can. Any project I undertake is a new idea I've not seen done, or with features I want.
Actually starting to livestream them now as I figure if others can see what I'm trying to do I might be able to get instant feedback on how to do things better - crowd sourced learning :)
For a side project, pick one on that you're interested in on each side. For example, maybe try out CircleCI and deploy to Google App Engine. Then you'll learn two new things, maybe one is great but the other is a pain in the ass. You can take the one that works well to a "real" project and talk knowledgeably to the architecture.
Should've been titled something like "Don't Over-Engineer Your Side Projects" or "How to Make your Side Project Resemble Your Day Job (If You Work for Sensible People)"
Any kind of project has a "risk budget". If you have few other risks, then you can spend your risk budget on new tools. If you are facing high risks in other directions, it makes sense to stick with what you know.
The good news is, once you've done that, you can get a pretty decent job.
The majority of software engineers seem to be under the illusion that potential customers care about the stack they are running.
This is not entirely true. The technology do surface in terms of user experience. It does in the details: app start, ui responsiveness, etc. I'm writing a game using go right now, and the end user can tell it's different because the user's cpu seems more powerful.
All in all, I think those are good mistakes, not bad ones. I wish more engineers could spot over-engineering, and they don't until they did the mistake a couple times. Side projects are, I believe, how one really learn to write programs. I think the right advice is: do whatever you want, do all the mistakes. How can you write a successful side project anyway if you never learn how to write a program? Also, the worse type of programmers are the ones telling you about over-engineering because they read about it in a blog. Code more, read less, I guess should be a good motto.
Where games are concerned I'd tend to agree (though Minecraft was built in Java and was TERRIBLE for the first couple of years, now look at it).
Well, it still takes absurd amounts of memory for what it's doing, which I guess is on par for Java. (On my notebook with 4GB RAM, modpacks fall into one of three categories: "works well", "have to shut off as many OS services as possible before starting" and "forget it".)
the reason why this makes memory so heavy is because even a simple int list, needs boxing. So you end up with a big chunk of garbage which you don't need. On the server, this is not a problem because most of the time the list is a young gen object and die really fast, so G1GC solves most problems. The problem of course is really big, because arrays are a pain to work with and not many people do that.
However as soon as java 10 hits, we might get value types and maybe this changes the game (but we will see, since it would've taken way to long for that thing). Also there are more and more types inside the standard library who would be great value types, i.e. LocalDate/LocalDateTime, but at the moment they take way more memory than needed, especially since they only contain 2 immutable shorts and 1 immutable int.. in the perfect case it would be something like 8 byte, however it takes way way way way more. (basically a simple class at least takes 16 bytes, which is already double the field size, so you end up with 24 bytes [probably more due to various other stuff])
btw. for a game like minecraft you probably have a lot of types that follow the same stuff like LocalDate, small class with immutable int's/short's for position, etc. these have the same problem and will fill the memory more quickly than needed
Does it though? There's a non-trivial amount of data being loaded (and generated) all the time.
You have mentioned modpacks. Those can be HEAVY. And a single not well programmed block can bring down and entire server.
smem | grep mine
5728 jcelerier minetest 0 433136 439287 466348
7044 jcelerier java -jar /usr/share/minecr 0 691476 708501 753948All of these are stack-related.
Proper engineering and design are required no matter what you choose.
No silver bullets ;-)
Is there some way I know of to package Flash games so that Firefox users can play them without downloading Flash?
I get that done of the examples I mentioned are arguable, but some of these are pretty unambiguously stack intrinsic. Some usability concerns cannot be addressed in some stacks.
In other words, when confronted about your high bug rate, you're not going to get anywhere going "heh, you see, Python was just more convenient than a statically-typed language and this lead to..."
You are right that nobody will care if you use them as excuses. But that is not what the GP said.
And you'll likely have to.
I have always found it very easy to talk all the energy out of a good idea before I've even got any real work done.
http://www.psych.nyu.edu/gollwitzer/09_Gollwitzer_Sheeran_Se...
I don't think people start projects with the conscious intention of building identity, but with a genuine intention of delivering. I do think, though, that there's a small (or big!) part of your unconscious brain that binds these efforts to an overarching persona-building (both inwards and outwards) project. Although unconscious, it contributes to this net energy.
I guess that this communicating your intentions satisfies this portion of your brain, and reduces the amount of energy involved in the project.
If someone else could get hurt, I push through.
Looking back on my many projects, the feeling of delivering on an external commitment seems much weaker than the urgency of "wait 'til my colleagues see this!"
Publicly declaring your intention to quit a habit or lose weight has also been shown to increase success in some instances.
I think it depends on whether people reward you with positive regard for stating an intention to do something, and whether you look foolish if you don't do the thing you stated. When you can get the reward just for the statement and there is no consequences for failure, pre-stating your intention seems harmful. On the other hand, if you are not rewarded for pre-statement, and failure is embarrassing, pre-stating your intention seems helpful.
I somehow concluded that I don't care about approvals on my 'integrity' yet I highly value my 'creativity' viewed by others.
But the majority of mine stall for other reasons; I don't usually announce/discuss them.
To me, that would suggest it wasn't a good idea in the first place.
Either way I learn a lot more by building a bad idea than I do by talking about it.
> The king of Israel answered, "Tell him: 'One who puts on his armor should not boast like one who takes it off.'"
I found that worrying about deployment automation early on is immensely valuable, especially for side projects. I don't always find time or interest to keep working on side projects after a regular day of work. Quite often there are weeks or even months where I abandon a side project only to come back at a later point when motivation is back or things have settled down and leave more time for my side projects. If I didn't take the time to automate testing and deployments (which makes a lot of what continuous delivery is about) I find myself struggling with getting things up and running and become frustrated before I even get started.
Then there's only one way to make sure that my test and deployment automation keeps working: running it regularly, e.g. in a pipeline.
So, yes, don't over optimise your deployment pipeline early on. But having one in place early can pay off really soon -- and maybe even keep your side project alive.
I'm a big fan of doing the hard work early on. When you push that step off it makes dev later slow and frustrating.
I have a back-burner project that touch very infrequently, but I can jump in, make a quick change, and have it deployed to "production" in just a few seconds.
Why? Well, because almost all my projects are designed for businesses and enterprises, and with that audience - things better work well and they better work first time during the demo or you don't stand a chance of getting a customer.
Perhaps if I was writing a social media app for sharing cat pictures, then yeah, I would slap together an MVP and show it to cat lovers. After all - what is the worst that can happen if your cute cat picture doesn't upload? Just try it again or delete the app and walk away, and there are no consequences.
But what if my HR app doesn't send that crucial reminder email to a department manager that one of your electrical tradesmen's license expires tomorrow, and you end up sending him out to a job next week and he makes a mistake that ends up getting your company sued and eventually shut down, putting hundreds of others out of work?
I am sure as sh*t going to ensure that my reminder email system is absolutely over engineered to ensure reliable email delivery and tracking. You don't get ANY chances to blow that kind of stuff with a business of enterprise audience.
Of course, by the very nature of enterprise customers, sometimes there isn't a whole lot of scope that you can cut.
I wrote about it here; https://ma.ttias.be/release-notes-driven-development-rndd/
It reminds me of many release notes describing a ton of work without any woah.
E.g., I love to program in assembler. Preferably 8 bit SoCs like AVR. The point is not to finish the project fast or even at all by using a more efficient language that may have libraries for everything, but to enjoy programming in 8 bit assembler. Reading the absurdly detailed manuals. Comparing chips at register level. Obviously, I need to write my own libraries/frameworks, because, I want to do that myself, in assembler.
Even if not using assembly, I may still want to write the library myself in order to, well, do that.
Wood workers often seem to have similar ideas about side projects, like when they build a project without power tools, without glue, from old scrap wood, etc., just because. But for some reason, I seldom meet hackers who understand the fun in doing stuff from scratch.
I guess the author and I have a different purpose for side projects. It sounds like the author is trying to actually start a business while still employed, while I am just trying to mess around and have fun.
Go visit your local hackerspace :).
That said, it's probably because in tech, there's much more vertical space for one to pick their definition of "from scratch" from. You go low, to assembly level, but you don't fab your own chips :).
For my current side project, which I've been working on for a year (and is probably a month or two away from being "beta-able"), I've learned Clojure, Elasticsearch, ES6, mithril.js, Ansible, and have expanded/refined my skills with RabbitMQ, Postgres (or SQL in general), and distributed architecture (not to mention getting to figure out how systemd service and timer configuration files work).
I've really been working hard on it because my current day job doesn't involve any programming, and by the time I get home in the evening I'm really jonesing for some code.
Have my side projects made me a better developer? Without a doubt. Am I any more employable? No, not really - I still haven't figured out how to improve my 'Cultural Fit' via side projects.
If you change the interface a lot or how features work, you'll annoy users that don't like the change and you'll have to be extra careful not to introduce any bugs. If users can save data, the new version will have to work with the old data format. If you want to move features from a free version to the paid version you'll annoy users. So to me, releasing a MVP you charge for that might change a lot later might lead to more headaches than developing the idea further.
Are there ways around this?
Side projects you release for free are one thing but when you want to charge for them the amount of polish and attention to detail you have to pay to marketing jumps by an order of magnitude. People will tolerate bugs and rough edges for free projects but not for paid ones. I agree with the advice about not wasting time with CI, switching frameworks etc. though.
I've often fallen in the trap of "oh that would be cool", but this time I'm going to try to only build the necessary features that actually bring in customers, instead of wasting time on a bunch of useless stuff.
> Over-architecting infrastructure
Yep, I think it's a good idea to just put everything on Heroku until you're spending at least $100 per month. It's just so easy to get up and running.
But sometimes I need that knowledge again for the next side project. And this over-engineering is often a way to simplify a reference architecture and be much faster at the end.
So for me over-engineering equals learning but is also a big part of the motivation.
Thinking about using Unity now...
It gives me pause to think about features and to show people to get feedback. You can also use it as a sales tool to sell an app you are thinking of building without investing weeks and months.
Nowadays its so simple to do nice mockups its so easy to find stencils for SemanticUI, icons etc. Makes it a breeze to get an interface going and I have never been this productive in the 'idea' stage before.
Users will absolutely notice if your product crashes or breaks, which is more likely with some languages than others. The single most likely reason for me to stop using a nominally useful product is that it doesn't work reliably. I hate the fact that so much technology today is broken, partly because people have this cavalier attitude towards choosing technologies that help to keep things working.
Working this way is much nicer for me. It's more organic. I end up with code that evolves, and takes shape of it's own accord. It makes me feel more creative. The best thing is, I have code that does things. It's not out in the wild, but when I come home from work I have a couple of projects that are coming along nicely to work on and play with.
In the context of hobby projects, over-engineering can even serve purpose of learning new things you otherwise could not.
For side projects, or startup candidates, however, you just ship it
Eventually backed up and used Make to run Babel and then cat the files together.
Probably the best thing I have come to understand in building side projects is that you need to get it out in front of people as soon as possible. After that, just get feedback, use analytics, and iterate quickly to fix issues.
Even more important, go look for existing problems, whether it be in online forums or talking to people in person. If you making a unicorn detector and you spend several years with your nose down, its a rough lesson to learn that no one wants a unicorn detector.
My biggest hurdle was friction between using instagram and my app. But may that's not what you're targeting.
Either way, I was just curious...
There is not much traction, the 1% internet rule is real in terms of the number of people that contribute verse those that consume content.
The author's use of "side project" is referring to "I'm building something outside of my day job that I hope will turn into a business." IF your goal is to make money, then the author's advice is spot on. Get to MVP. Get users. Listen to them. Iterate. All the advice we hear that applies to startups, applies here.
For a lot of software, making the best decision today without worrying about tomorrow, while still following tried and true engineering practices (i.e. test driving, etc), leads to a project that continues to progress.
All I did I did for fun and maybe learning something. So I think over-engineering side projects is fine.
In fact, I have a side project currently where I just needed to fabricate a tiny fixture for a physical device. I caught myself over-engineering the prototype, went to sleep, woke up, saw this article and came to my senses. Consider this article bookmarked!
For those who are interested: Video: https://streamacon.com/video/laracon-us-2017/day-2-mathias-a... Slides: https://speakerdeck.com/minicodemonkey/launching-and-scaling...
That's a pretty ignorant and potentially short sighted statement. Look at what happened to Instagram. They almost lost it all because they hadn't planned for the possibility of handling scale quickly
Sometimes you simply don't have the luxury of monitoring and modifying to account for load and redundancy. Sometimes you're monitoring says "hey traffic" shortly followed by "OMG YOU'RE SCREWED" because you can't possibly think up and implement an appropriate scalable architecture, and modify the codebase to handle it and and and while your servers are melting and everyone is yelling at you.
Yes, this is very unlikely to happen, but when it does it can cost you your business if you're not prepared. Obviously apps with no networking and no viral component are far less likely to have this problem. Got a macApp that creates icon sets for XCode? Yeah, you can ignore this scaling issue. Writing the next Instagram? You're a fool if you don't have plans in place, in advance of launch, for . handling the onslaught of people that will make your business successful.
I had never heard this version before. The story I heard was that they pivoted from burbn, which they felt was to cluttered and unfocused. [1]
If they followed your advice, they would have built a scaled up version of burbn, which nobody would have used.
[1] http://www.makeuseof.com/tag/1-billion-in-2-years-the-story-...
It may be harder to come up with examples of people who failed, for the obvious reason - they failed and are therefore not well known, but I really don't find the statement that Instagram "nearly lost it all" very compelling. Did they? Really? If they had had even more problems scaling, would they really have lost users? Look at Twitter, they took years to resolve issues.
I actually think the importance of stability and uptime are overblown in the early stages. If I think about the times downtime has significantly impacted my view of a company or product, it's when large established services I'd come to rely on professionally went down. You need to establish a reputation for reliability before most people expect it of you. That's not to say it's not important, but you need to look at context. Noone relies on Instagram minute by minute for their professional work, so the priorities are different. UX, marketing, network effects, etc. are going to trump stability.
That's a problem that essentially nobody has. You're talking such a very small percentage of start-ups or projects, as to be meaningless as a reference point.
Simply put, that isn't a problem you're going to have.
If by some small miracle you have that kind of scaling problem, you can very likely fix it thanks to the fact that you've struck gold.
The number of homeruns that failed because they couldn't scale, is few and far between. There are so few of them, that you're naming one that became a huge success instead of listing some huge prominent failure. First, start with the fact that there are so few of these Instagram rapid hyper-scaling scenarios to begin with, then narrow it down to the percentage that failed due to poor scaling, then narrow it down to the number of those who failed due to poor scaling in the modern cloud era (ie not Friendster from 2004-05). You're very likely down to close to zero examples.
A number of commenters point out that many side-projects to them are to explore new concepts/libraries, or as a means of learning. If it's for the sake of learning for your job, then yes, it'd be a side project.
But if you're looking to take a project to completion (the author talks about products; my projects aren't), I find that using it as a learning environment extremely detrimental. Yes, you do learn things over the course of development---through research, struggle, and growth. But if too much of your time is spent on things you don't have a good foundation on, you may burn out too early, wind up frustrated, and wind up with a mess that leads to refactoring. Use a foundation that you have experience with and know well, and learn the parts that are necessary.
(I do distinguish, though, a research project with a project for playful learning. The former is much more formal and disciplined. But you can still burn out early.)
https://blog.skywelch.io/2017/06/over-engineering-bunnies-io...
He so aggressively avoids overengineering that he goes the other way - if you make ANY effort to properly engineer something he gets really annoyed and says time is being wasted.
It's almost like he feels software MUST be engineered to NOT scale in order to be an "MVP".
It's strange because for me, thinking about how something is built means that it can be built right the first time, without extra effort, just by thinking it through.
Same with my AWS "framework". And my CSS "framework" and my React component superclass and lots of other stuff I carry from one side project to another.
Some would (and have) accused me of over engineering; but I can discover my project is pointless very quickly.
That's one of my side project vices!
Every time I write a new login database... I wonder if I should make the ID a MEDIUMINT or INT.
> 5. Build your project first – then worry about continuous delivery.
These contradict each other. Building your own deployment system is just as much of a distraction as building your own CSS framework.
Triggering that pipeline from a commit trigger instead of manually just means copying the build command into a config file. It pays for itself in the first hour.
You can get quite far without any kind of "pipeline system" nor does building imply a pipeline.
I find it incredibly frustrating that much of this isn't a solved problem that I can then just insert my side project into.
hacking is fun until I have to maintain it and then I start wishing for all of those features instead of having to spend days or weeks implementing each of them from scratch
Ideally, you'd want to understand both worlds.
As such, I feel that the projects I work on in my own time and the crazy things I do in them help me to better understand what kind of things really do need to happen when working under tight deadlines in the workplace.
I.e. It's easier to produce an effective product quickly if you know from personal experience exactly which approaches or things really are essential.
Also, note I haven't read the actual article. I'm replying mostly in response to the comments :)
Well, maybe not a 32 node k8s cluster. However, spinning up K8s clusters is so easy now, that it costs almost no time to deploy stuff there. And so many things are taken care for you: configuration, secret storage, networking, versioning, keeping services up, etc.
If on AWS, there's Kops. If on GCP, that's even easier, a couple of clicks and you have a cluster.
Or just test it in a VM and stop wasting money on side-projects.
"Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away"
I've always taken that quote to apply only when you have been taking things away; not when you've ended up with a colossal complex monstrosity where removal of any single strand would cause total collapse.
If you want something to generate a nice supplementary income stream, or practice for starting a company when you get "that great idea" then this article is very good advice.
On the other hand, if you are trying to learn a new technology stack, develop experience in building scalable architectures or even beefing up your project management skills then this is terrible advice.
So, the real art is figuring out what you want out of your side project.
A few of them turned into great side businesses but most of the times it help me explore technologies and projects while still solving real problems.
I'm always looking to make the code as easy to change as possible -- the idea being that change is a precondition for any form of improvement ... Reading this article makes me wonder if I've turned that pursuit into an anti-goal
You own it, nobody it paying you to work on it, over-engineer the shit out of it if you feel like it.
If the side project is for learning purposes then taking time to explore different approaches is OK.
If your side project goal is to build a product, then you should approach it as a more traditional project, budgeting your time, and focusing on time-sensitive tradeoffs, etc.
I can't even count how many times I seen projects with awesome infrastructure and basically zero functionality after depleting whole budget (and usually some more).
In case I want to actually deliver a side-project, I usually write some basic tests. As such I can focus more on adding new (fun) features instead of hunting down regressions.
Then you can send the link to anyone and it always just works.
sure it would be great to have scaling to 10,000 users already sorted by the time you get there but odds are your project will never get there, when you get to 5k users you need to start worrying about that
if you're just doing it to learn or for fun then obviously this article or my comment don't apply
it does have a nice side benefit however of actually learning a new framework :)
My side projects have all been over engineered, an unsuccessful. They have also helped my career enormously :)
This whole agile sdlc idea is a distraction from writing good code.
"those software engineers often have the tendency to over-engineer their side projects"
When your first sentence is false, so do is the rest of the article !
Not necessarily.