The Start-up from Hell
jacquesmattheij.com
jacquesmattheij.com
There's no better way to learn the true character of a man than to ask him to write a very large check based on an oral contract.
Part of that is giving credit where credit is due. The vision was his, the implementation, and dealing with the fall-out of all the bad decisions in every way except monetary (he never missed a payment of the agreed upon monthly amount) was mine. All of us, also those that were brought in later worked quite hard.
There would not be much point in writing this in any other way, the only thing you'd get out of that is that I am either vindictive or that I still haven't come to terms with all this, 20 years after the fact.
I think I have come to terms with it, I see it as a very expensive but ultimately very useful lesson that I apparently needed to learn in order to do better.
That is probably true, at the same time he knew full well to sell when I announced my departure so he must have had some idea. The interesting thing is though, in retrospect, the 'ill advised platform' brought out the best in me in many ways, and I wonder if I had learned as much as I did had the platform been more powerful. There's nothing more satisfying than taking something that any sane person would say can not be done and then doing it anyway simply because nobody told you it couldn't be done.
I think of the whole thing as a character building exercise ;)
And I have learned to have great respect for the people that make industry work, the guys behind the lathes and the mills, the 'operators' that have more knowledge in their pinky about metal and metal working than most people will acquire in a life time of book knowledge.
They taught me more about feeds and speeds and materials than I ever wanted to know. And they also taught me that I'll never be a good machinist, even though I'm safe and can make what I want I just can't spend the lifetime on gaining all that knowledge.
One nice anecdote about that. One guy was complaining about the precision, this was in a factory of gears called 'Mak aandrijvingen', the guy told me while listening to the machine (in a factory full of noise at that) that it was running about 1/100th wide close to the chuck.
I thought he was pulling my leg, brought out the micrometer, sure enough 1/100th wide. We adjusted the table a tiny bit with to reduce the offset but overcompensated, during the next trial run he told me it had gotten worse, probably around 0.05 mm. Sure enough, the exact amount was very close to that. After that I started to slowly believe that this guy was so attuned to his machine that he could hear how much material it was taking off and as a consequence knew how much was left. When he finally said it was good it really was. By then I was a firm believer :)
Incredible.
My dad and I are kind of like you and that guy, by the way. He's the shop guy with no schooling but almost half a century of experience building stuff, and I'm the one who knows all the math and computer stuff... ;-)
Another option would have been to take time out to study the underlying theory better. The initial spec was trivial, if there had been any indication of how immense the requirements were going to be I would have either refused the work or I would have taken out a year or more to educate myself on the subject matter.
Funny thing though, even now, knowing as much as I do I would probably refuse that job. This is not a one person thing, there are too many specialties involved and the codebase is far too large for a single person. The demands placed on such software have only gone up over the years.
One thing is funny though, even today in terms of ease of use the UI I came up with still is better than anything else on the market, as is the toolpath optimizer that is built in.
Eventually, during the creation of a parallel development (the milling software, which I've left out of the story, it would not have added too much I though, but now I realize it would have added one more thing), I got so stuck on a piece of very hard math that I managed to convince Fred that we needed outside expertise and by great exception I was allowed to spend some money on a consult with a real mathematician.
Another thing where we brought in an outsider was when the display requirements went up way beyond the initial spec (real time zoom during the runs), I was allowed to hire a friend of mine (Maurice), a guy that I'd worked together with on programming games to write a chunk of assembly given a datastructure. This was doable because it was a fairly isolated part of the code, but Fred was totally paranoid about bringing on board other programmers having access to large chunks of the source.
[1] A nice example of such a situation was when we had to do the thread cutting that was so easily discarded in the beginning after all. This required a synchronization with a third axis spinning at anything between 50 and 1500 RPM or so, yielding a pulse train of up to 25Khz, something that none of the interrupt inputs of the chosen computer could handle with any reliability, especially not while driving a stepper at the same time with very tight constraints on latency. Normally encoder output is fed in to specialized counter boards but those were way out of the budget, and besides would only fit the IBM architecture. So I designed our own counterboard, and fed the output in to the midi port with a bunch of trickery to be able to re-construct the position of the toolbit relative to the index on the main axis.
This was a very nasty bit of code, because it depended on a lot of things that could change in real time (such as the density of the material being hit, the supply of coolant and so on) and there was not a whole lot of margin for error here. Once a thread has been 'started' you can't back out, you have to complete the sweep. So what the program would do is look in a table to figure out how much lag we could reasonably expect after hitting the material with the toolbit, then measure the sitation as it actually occured and compensate. Eventually I got it to work reliably enough for production but of course that was a total hack.
On the other hand, maybe you actually learned it faster this way. I guess it depends on how good you are at book learning, but there's no way to grok a process like jumping in and coding it.
So my sources of information where I was located were fairly limited. If I could have taken the time off I would have spent that time first on a literature search and after that on tracking down copies and reading those books.
I agree with you that I learned faster this way but not as thorough and as deeply integrated as I would have liked, it usually was just enough to get the job done before moving on to the next crisis.
> every time I managed to fix something[1] I proved him right in a way deepening the misery and the complexity of the next challenge
I have a similar situation with the startup that I work for now. I've done some decent work, even though I don't nearly have the toolset (or background experience, tbh) that would usually be required or available. This has been interpreted as "you've validated working with what we've got" as opposed to "you can do impressive work, in spite of what we've got." Now I'm asked to do the usual improvements (better performance, more features, and lower cost), and expected to figure a way out with this "validated" workflow.
I guess this is just a wordy way of asking if you have any advice on that matter. It would be appreciated.
(edit: the more I think about it, the more I want to claim that it happens to me a little more, because I end up taking on the challenges and producing results, without the support I think I should have)
But the fact that you think that it is company wide suggests the issue may be with your management.
In that case I would confront them with this. Maybe suggest that that is not a long term workable solution and if they refuse to change tack start looking for other options or put down your foot once to see how they respond to that.
good luck, that's a tough situation to be in.
Don't fall like I did for the 'I've already invested so much in to this, one more thing I can do' trick, that carrot can be moved ahead of you for ever.
I wonder if having a catastrophic start-up experience is practically a rite of passage for the serious entrepreneur.
Obviously some people hit it big first time, but your very first business is obviously the one that's most likely to fail catastrophically, and if you can take that kind of blow (and it is a very big blow) and keep going, it's probably a predictor of success. Thoughts?
I'm sure there are many serious entrepreneurs out there who did not need to lose their spouses to see that they were working too much!
It's painful enough as it is to dig through this old stuff and to try to abstract out the lessons learned in such a way that someone might get some mileage out of it.
As for the private aspect, what happened is that we completely lost focus on what was important in life and what was not (or at least, I did) and that I allowed myself to be used in a way that was not healthy. The fights about money, the talks they would differ so much from one person to the next that I don't think that it would contribute much to the story.
I'm still in contact with the woman I was with back then, she has since found someone else to live with and has a really nice kid so at least I know that she landed on her feet. She was/is a good person that stood by me until she couldn't take it any more and I really don't blame her. After I quit the company we tried to get back together again but it never felt like it was before and it fizzled out.
The carrot was being held about a week to a month ahead of where we were all the time so we really believed that we'd be out of trouble shortly. And I think that 'Fred' believed that too, he was no longer objective about any of this.
The road to hell is paved with good intentions.
Yes, but one in which bodily harm is also on the table? I've read a lot of entrepreneurial tales, but this one puts "sacrifice" and "earning your stripes" at a whole new level.
that was some of the nicest work I ever did -- it's interesting how destructive processes can also produce great work. (This is not to say that the results are worth the destruction.)
Take the feedback you receive and use it to improve the product but unless something is really trivial it is better to sell what you have than to make some promise you can not keep -- another general lesson I would take away is, "be very wary when the salesperson is allowed to write the spec". I bet this is true across all business domains.
Fred never did hire a replacement, and hoped that I would stay on because I would not let my buddies down. -- I think this dynamic is built into the business models of some companies, especially consultancies. You feel guilty if you go home to sleep while your friends are coding 'til their eyes bleed, even when the overwork is brought about by extrinsic factors (see above).
why do programmers allow this tactic to work?
This is the problem; if you give me better work when I pressure you and treat you like shit than when I treat you respectfully and let you set your own hours, what do you think the result will be?
If you tried such a trick on me today I'd catch on in 5 seconds flat.
Part of it lies in that programmers are fascinated by technology, which means that if you give us shiny toys to play with and the occasional interesting puzzle to solve we'll take a good bit of abuse before throwing in the towel. We're also much too eager to show that we can do stuff.
Another part is that you have - as an inexperienced froggie - too little feeling to sense that the temperature in the pot is rising and that if you don't jump out now you may lose your ability to jump at all.
Finally, and perhaps crucially, the more you have invested in a project like this the harder it becomes to walk away.
We aren't nice to people because it makes them work better, instead we "aren't nasty" to people because it's nasty.
I think that relying on feelings of "don't be nasty" to keep businesspeople in check in the face of nastiness that confers sustainable advantages is... not so sustainable. Some people /enjoy/ being nasty. So yeah, we have to build the world we want, and social mores are strong things.
One thought, from a market efficiency perspective, is that when I see an apparent sustainable advantage to being nasty, when there are successful people not being nasty in that way, is that maybe it's not as much of a sustainable advantage as it seems?
In this case, I think that jacquesm nailed it when he pointed out that this can happen, at most, one time for each programmer, and relying on only fresh programmers is difficult, just because without past experience, it's very difficult to judge the quality of a programmer.
Been through that at the early stage of my career. It was a startup too. Right now if anybody trying to pull that out with me, its not going to happen again. If they should hire more people but not hiring, then its their problem. I'm not staying more than I should, and if I put on extra time (funny that I still do this a lot, I guess I just love what I do), its on my terms. Not happy, then let me know when to quit.
I also found out that when you go through the extra mile a lot, without proper compensation, they will treat you cheap. In a way, its sort of true, since if we put more effort that we are paid for, then we are cheap in that sense. Its easier to get respect when you just stick to your guns and politely declined those request for extra effort unless its paid up for. You may not get what you demanded, but its easier for them to treat you with respect (oddly!).
The lesson that I learned from my version of failed startup is that you should have the funder/entrepreneur to know the trade. Warren Buffet says never invest on anything that you do not understand. If the funder/investor/entrepreneur does not know the trade, then there is no way they could tell that you are telling the truth, other than pushing your buttons. And they will push your buttons, as much as they can get away with.
Its a good lesson learned, and I'm glad I'm through. But there is no way I am going through that again.
Yep. He even said that he'd gone back while the fire was still being fought to look.
> Is he identifiable from your story?
Not by you. I figure four or five people could put 2 and 2 together, none of them read this blog and they're already in the know.
It was reported to me after the statue of limitations had expired (nobody got hurt, even though that was pure chance) so as far as I know I'm in the clear.
Arsonists always do. That's why they take pictures of the crowds at fires. The person who started it is almost certainly there watching. In fact, it will probably be a man standing there with a bulge in his pants (seriously).
http://www.google.com/search?q=arsonists+sexually+aroused
Seems to confirm it.
http://www.newyorker.com/archive/1936/01/18/1936_01_18_018_T...
Requires a subscription.
[1] http://www.sciencedirect.com/science?_ob=ArticleURL&_udi...
Not that you were 'just starting' with hardware, but still.
Oh man... this is the hardest and most important lesson ever, I think. The allure of "sell it first, then build it" is very strong, but in the long term, that way lies madness.
You also need to ensure you are converging on a product and not just doing services work around a common problem.
the problem is that it's /really/ easy to get yourself into the kind of trouble you don't recover from that way.
I understand that different markets are different, but in every business I've run, the limiting factor was my commitment or my reputation (sales) or some combination therof.
If I over promise a little, I can deliver for a limited time (as the OP did) at the risk of burnout. It sounds like burnout was a huge factor in the failure of the business this story is about. Burnout of key people can very easily kill a small company.
If I over promise a lot and fail, then we start taking hits to the reputation, and if I've accepted pre-payment that I can't pay back, well, at that point it's pretty much over.
so yeah, while it can result in dramatically positive outcomes, it's a /very/ high risk move.
Having worked metal with these things and mills you can easily understand why cnc machines are so damn expensive. There is so much to deal with and I think if you would have chosen a mill instead this would have ended in more headaches more quickly.
One question that has more to do with the tech than the hell part I have is: did you have to deal with the excess strips of metal (which I've seen build up to dangerous length of spinning razor metal when people were being stupid) in the computer or was it under manual inspection? (There's plenty more that I thought about but that was my first)
If you don't do that in some materials it will build up to the point where it will form a giant mess which can cause the whole thing to stall out, cause toolbreakage or even machine damage.
Swarf control is a very tricky sub problem of the whole machining thing.
I left out the milling program that was developed in parallel, when I compare the two I'd say the mill was (in spite of being one dimension more) the easier one of the two.
I must say I would have given up much more early, although you probably would have too knowing what you know.
Another question I had was if these machines required a large amount of manual interaction (though lubrication probably was at least provided by the machines at this time).
Because they were 'retrofits' rather than machines designed from the ground up things like tool exchange, zeroing, tool breakage and so on caused quite a bit of operator interaction.
If you look closely on the photo there is a little switch above the big red STOP button that we called the 'softstop', it was polled every few milliseconds in all situations except for thread cutting, and when engaged the machine will pull the tool away from the work (in all operations, including drilling, inside work, outside work and cut-off) so that you could stop the spindle.
Later versions had a tool changer driven by a third stepper motor, this was mechanically a very weak device and caused tons of trouble.
Lubrication on most manual lathes is either done by a hand pump that you operate ever so often, on more advanced machines it is timer driven, or coupled to the amount of movement on the carriage.
There also is the coolant to take care of, we had a relay box hooked up so that the various functions of the machine could be switched on or off under programmatic control.
All these mods together meant that installing the kit on an existing lathe was quite a bit of work, it wasn't unusual that more than one day went in to mating the x-y table to the machine (this is a very precise job), running tests, hooking up all the auxiliary stuff and so on.
But it is rather interesting how you handled all of it especially for the time when cncs had only been out for like less than a decade and you were modifying existing machines.
In my experience, it's really easy to fall into the smooth-talking salesman trap, especially if you aren't fully aware of the scope of work. When you're already committed to the project, things like "let's just make a big push to get out feature X" don't sound so unreasonable until you realize that all you have been doing is making big pushes.
As a side note, in a previous lifetime, we built some software for the Atari ST and we had quite a bit of experience with it. You are spot on about the robustness of the hardware. If the machine were to ever stop working, standard practice was to lift it off the desk about two inches, and let it drop. The theory was that it would reseat whatever chips had worked loose. It was a little bit more than a game computer, but as you say was pretty fragile.
After that from 1', if that worked the customer was charged and if it didn't there was no point in trying to repair the TV any more.
You can imagine the kind of trouble we had with the computers sitting in an enclosure right next to a vibrating CNC rig.
More than once I had to drive halfway across the country to re-seat a chip that had decided to go for a walk after jumping out of its socket.
Starting out in business we all make mistakes. Learning from them should be sufficient reward.
*As easy as a lawsuit ever is, I mean... it could have taken a couple years and about 100k investment, but likely they would have realized the problem and just settled.
But frankly I didn't care enough about it to put that much negative energy in to it, court cases are extremely hard on the system.
I simply chalked it up to experience and enjoyed my new project, which was building a very large scale message switch using QnX, it was my first experience in networked computers using message passing and a completely new world.
The contrast with the other company was remarkable in every which way and Jan (the guy that owned the new place) and I are still friends today. The funny thing is that a couple of years later I employed him for a bit!
http://www.cncci.com/resources/parametric%20programming.htm
The work you are referring to requires 5 axis control, something that is fairly common nowadays but back then was extremely rare. The software I wrote was not capable of such complex movements because the machine itself was only a 3-Axis machine, a 5-Axis retrofit would be extremely hard to construct. Basically you'd have to rip apart the main structure of the machine and rebuild it. (see http://www.youtube.com/watch?v=lWCaQ4XvG6Y&feature=relat... from about 3:48) You could conceivably do that if you only worked on one kind of machine but retro-fits tend to be bolted on to whatever is available that still has a rotating spindle and good bearings.
The software did have some parametric tricks but not a full suite, mostly related to making different sized versions of the same part.
Most CNC machines are programmed in something called 'G-CODE', with a translator step between the CAD and the CAM part. The software that I wrote was much more tightly coupled (cad-cam all in one) so we had the full power of a regular programming language at our disposal to do complex part generation.
I agree with you though that they're not practicing proper safety and I've seen numerous examples of millwrights and machinists missing digits, eyes or worse.
A toolbit spinning fast enough will throw swarf around over considerable distances and it doesn't take much to take out an eye.
In production use normally there would be an enclosure and mandatory goggles. No 'walking around in the machine'.
Even the doors normally have interlocks, if they open it counts as an e-stop (emergency stop).
I'm also wondering why he stuck around so long. To me the writing was on the wall that it wasn't going to work!
This is not something you should give me 'kudos' for, it's just something I was born with, though I feel as I'm getting older that I'm definitely slowing down.
The problem with quitting was that there seemed to be light at the end of the tunnel all the time, only then when you almost got there the tunnel got longer.
And in the end, it did work. The software ended up working as advertised (ok, you can keep adding features to something like that forever, just like any other cad program) and it ended up as the basis of a reasonably successful company that is still in business over two decades later.
It's just that to do this sort of thing properly the first time out of the gate you'd need at a minimum 3 people, a domain expert (which 'Fred' definitely wasn't) driving the specification, a UI guy and a low level person.
If it had been like that I could have saved literally months of rework based on the assumptions that were fed in to the process early on as 'immutable', only then to be changed at a later date anyway.
I mean, there are a lot of important lessons you learn when you have a "project from hell", so my reasoning is that the project better be small, so at least the price will be low.
I had done a large number of smaller projects and several mid-sized projects, as well as one very large one by the time I started working on this though, so it's not as though I was green behind the ears or something.
Maybe the morale should be 'know when to walk'. And I should have walked long before I did. The violence is what made the coin drop for me, it was a realization clear as day, 'I'm being abused here'.
Suddenly the veil came off. After that quitting was relatively easy.
They'd always said that. Then at midway, suddenly "we need the money, and there is this client willing to pay if we make it such and such. It shouldn't be that hard eh? I know you can do it". On and on.
Till this day, if anyone says "I know you can do it", my alert level would increase.
To me, what made it easy for me to quit, was the fact that my boss didn't not want to spend any money on sales to sell the finished product, when all this while he had been spending money on the development of the product. For all the talk of "I am being most loyal to the company" and "I sacrificed a lot for the company", he hesitated on doing the final push.
As naive as I was, I suddenly discovered it would be stupid for me to stay. That made it easy.