We’re Fucked, It’s Over: Coming Back from the Brink
scott.a16z.com
scott.a16z.com
http://www.fastcodesign.com/1671172/how-a-story-from-world-w...
"In WWII, Allied bombers were key to strategic attacks, yet these lumbering giants were constantly shot down over enemy territory. The planes needed more armor, but armor is heavy. So extra plating could only go where the planes were being shot the most."...
"[A guy in charge] said the military didn’t need to reinforce the spots that had bullet holes. They needed to reinforce the spots that didn’t have bullet holes."
Having experienced 2 "WFIO" things in the last 6 months, I agree that entrepreneurs need to be resilient, but I think it's more valuable to read post mortems than these "we almost died" posts because they prove more instructional.
For instance, Aaron's post about Tutorspree is a really useful post for me, although it definitely has less of a feel good vibe to it.
http://www.aaronkharris.com/when-seo-fails-single-channel-de...
The sentiment seems very popular in the start-up scene :)
(I don't do regular credit.)
Agree.
Also noting that hotmail in particular (1/2 of the n=2) was a free service.
There wasn't much that the customers who lost their mail could do legally (most likely) because they weren't paying for the service. So I am guessing it wouldn't be something that you could hire a lawyer to pursue [1] or even band together for a class action.
[1] Which is a really important concept actually. If you are providing a service to a customer and charge either nothing, or a very cheap price, and have a good contract, the chance of someone hiring a lawyer to pursue an action is pretty small.
Apocalypse Now:
> Why are you sitting on your helmet?
Furthermore, have you considered that a general partner at Andreessen Horowitz has also seen plenty of what causes companies to fail? Maybe he just didn't outline it here?
I found out later when stopped by the (UK) police once, that their "Have you ever been in trouble with the police before?" is a similar one. After i answered it pretty dismissively, he said "It's amazing what some people will tell you".
Upvote to the top replier for this anecdote, which i'll keep for special occasions: > "[A guy in charge] said the military didn’t need to reinforce the spots that had bullet holes. They needed to reinforce the spots that didn’t have bullet holes."
That should read, "One of the most influential mathematical statisticians in history..."
He also invented the "Wald test".
But, really, it's a clever observation that was made by a historically great statistician. And the anecdote leaves out the fact that he had probably been thinking about this problem before --- he was working as part of the war effort. So it was probably harder than it looks.
The greatest probability of being destroyed is .534, and occurs
when a plane is hit by a 20-mm cannon shell in the engine area.
The next most vulnerable event is a hit by a 7.9-mm machine gun
bullet on the cockpit.
Another thing, I'm not sure that the method is unobvious. This is not the fault of the author but of the fairly breathless retellings that sound like an episode of 'Numb3rs'
An important addition to this: get everyone involved, but also listen to their ideas.
Having been a part of a sinking ship, the most frustrating thing about having everyone pour their blood, sweat, and tears into reviving the company is when the top brass decides to ignore all of the hard work and carry on with their own ideas that were never cleared with anyone else. They were their own iceberg.
http://edu.mkrecny.com/thoughts/how-i-fired-myself
"Um those [backups] got really expensive, so we stopped doing them about a month ago" sounds painfully familiar.
It should probably also contain the word "documentation of system configurations".
The issue was that my manager never caught it in my migration file and it went live.
I was lucky that we backed up every day..and didn't really lose a ton of data.
Works great. Especially for anything that needs history, e.g. medical data.
One thing I haven't figured out is how to archive old records, I think it's not a problem for modern DBMSs, but we have a problem at work with 15 years' worth of historical data.
I've personally f'd up pretty bad restoring a database to the wrong the site and blowing away 3 months of content. We didn't ever do any offsite backups and I just lucky to have an in with our hosting company and they managed to get everything back but those 3 months.
Once bitten, you're always way more cautious but it's easy to get lazy again from time to time.
Even a major "DELETE FROM Foo" should just immediately get the DBA switching to read-only mode, and then restoring using the transaction logs. It's annoying, it causes a bit of downtime, but it's not a "Big Deal".
But we do have daily backups and binlogs in multiple places, so it wouldn't actually lead to data loss. It really sounds like you were lucky NOT to be there!
Although all those extra parameters in the URL are a little odd.
Of course, this particular link doesn't refer to HN in its campaign code, so their statistics are going to get a little confused.
I am a complete neophyte at this, but tried pressing F12 to access various debugging tools. I could not find any reference to the comments.
Also interested how you discovered this in the first place?
By the way, when the "what can we do now that we couldn't do in the 90s" debate comes up it always occurs to me that the majority of the applications I use are far more malleable due to being web applications built on its fundamentally open (if admittedly complex and fragile) tool set. So instead of thinking "well, this UI sucks", I can actually fix it myself if I care enough to.
The css for #comments in the inspector pane doesn't specify dimensions for the div, so it should be the size of its child elements. None of the child elements overlap the body text or sidebar, so something else is causing the parent div to cover those elements.
The div directly before #comments is the sidebar, and clicking on it in the inspector pane shows the css floating it.
I then just added a clear: both to #comments using the inspector and it fixed the problem in the main window. You can add css to an element in the inspector in chrome by hitting the "+" in the right-hand pane (there are other ways, but that's probably the easiest).
stop with these annoying content protection people!
EDIT: Interesting that I've seen problems like this pop up more than a couple times in recent months. For instance: https://news.ycombinator.com/item?id=7382206
Easiest thing for me was "view source code" and copy from there.
I've worked for people that believe this, and others that don't. Nothing builds morale and loyalty like rolling up your sleeves and doing the hard stuff with your employees.
I once had a project manager who couldn't write a lick of code, but whenever a deadline approached or something blew up, he'd be right there at 10pm with everyone else bringing in fresh pizza, testing whatever he could, and giving pep talks. Contrast this with someone that says, "get this done by tomorrow, OK?" and then walks out the door. Who do you think is going to have a higher performing team in the long run?
The project manager who manages the team, schedule and risks such that things don't routinely blow and run down to the wire would be preferable to either.
The solution was days away from roll-out with our first big corporate client. Late on a Friday afternoon we were busy with final field testing - paying students a few $ to use their phones to test the app.
As the test data came in we realised we had a major problem: a small % of the phones weren't returning the correct test vectors for hashing algo.
After checking for obvious user error we came to the conclusion that something big was broken. Specific firmware sets didn't execute the crypto part of the code correctly. The entire value-proposition was that it works on every phone that can run an app, so it was a pretty big deal. I thought we were totally fucked.
We didn't sleep for two days and finally found the bug in the way the phones implement a bit-shift operation (doesn't carry a bit about 1/10000000 times). Then had to figure out a workaround that was still fast enough.
We shipped a fixed version before the end of the weekend, but I wouldn't wish that kind of stress on anyone.
Tracking that bug down was a fun three days.
I also followed the link to the other article where that phrase is used[1], and found something a bit concerning: "Nawaf moved the entire engineering team over to work on it. He called them all in to work nights and weekends until it was fixed, [...] Nawaf saved our bacon."
Umm, what about those engineers? I'll give the benefit of the doubt that they were rewarded appropriately, but the wording here seems to almost deliberately stoke the developers vs. management flames, especially coming from a VC...
?utm_content=bufferc2fec&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
I'm glad that ycombinator isn't listed as the source, at least. I think it would be fairly interesting to see what kind of data they have about various social medias and link propagation.
And also, a slight sickening sensation that marketers are doing this. Not that I'm surprised, it's just that I'm sure they have papers written saying that "Twitter has a 64% link click rate if you write your post like X, whereas Facebook can achieve as high as 70% if you do Y."
I know it has been going on for years on the web, and decades for other marketing, but here it's plain to see. It fills me with unease, like I'm no longer in control and what I like in life is already planned out.
Also, not sure if this will make you feel better, but most places that are doing this type of in-depth analytics, etc. are much more often trying to bend themselves to your will ("what feature, benefit or product that people will get value out of") than to trick you into liking something you otherwise wouldn't.
One way of interpreting that query string is as the answer to an implicit survey question of "How would you like to get the stories we write?" the answer in this case being: "Twitter" [2].
1 - https://support.google.com/analytics/answer/1033867?hl=en 2 - Obviously in this case it's pretty well statistically messed up by that link being posted to HN, but you get the point
(Looks like HN doesn't set it.)
Very true. Many times when something happens the automatic response is to focus on the worst case scenario instead of the immediate problem at hand.
Also, I couldn't select any text to copy and paste.
A very good point for our personal lives too.
1. Things like validation from peers / popular kids
2. Screwing up on that major test that sometimes "defines" where you end up. A lot of suicides result from academic pressure and failure (plus the environment/parents hinting that you can't succeed in life if you can't do X, Y or Z properly).
3. The academic thing is worse if you are artistically inclined and have science and math thrown down your throat telling you that humanities are for those that can't cut it in science (I know friends that have had a variation of this conversation with their kin/parents, etc) quite true in countries like India / Pakistan, Korea and perhaps even China.
4. Humiliation/shame from something that seems drastic in the moment (Getting rejected by the cute girl in front of all your friends and classmates. Yea that sucks). Getting labeled sucks too, a lot of girls can be mean to their own gender at that age which is absolutely disgusting.
Unfortunately for most people it is something that they only figure out in hindsight.
Heavy debts are a significant contributor to suicide rates, fear and shame play a large part. The potential collapse of companies would heighten this risk, so here's some statistics on the Great Depression (bottom of the page):
I was wondering if this was actually true or not, turns out that there is some evidence suggesting it.
http://apt.rcpsych.org/content/8/6/418.short
http://www.tandfonline.com/doi/abs/10.1080/13811110590904016...
"This midterm is one third of your grade. This class is one of four you are taking this semester. This semester is one of eight you will take at $University. And while people will tell you otherwise, your time in college is a small part of your life. It will be okay."
Seems to be a Safari bug. I had the same thing, after loading the website I couldn't even select text on HN anymore. Restarting Safari fixed it.
In the case of hotmail, how did he ever let a situation arise where one bug in a nightly cron script could obliterate his users emails?! No backups because they cost too much?? "the sun, the moon and the stars lined up against us for it to happen" No, you were just reckless and too busy chasing growth.
With Ironport, he says his whole business was built on the back of anti-spam yet they were totally unprepared when their partner who provided the anti-spam tech they used pulled the rug from under them (despite knowing full well this day would come). The real lesson here should be: try not to outsource the key component upon which your entire business depends.
I had to laugh at the end when he says the only time he's had a failure was when he was using his own limited resources to fund a business and not investor money. Well yes... mistakes are easier to hide/absorb when you're swimming in cash. Not so much when things are tight and you can't afford to learn after the fuck up.
I would guess a lot of companies in a WFIO situation that take a CYA approach rather than customer-first face a worse fate.
Also, survivorship bias much? Two examples --> conclusion? Was nice hearing about those company's histories though, didn't know about the Hotmail thing.
His response: "The Brightmail agreement was 80% went to them, 20% to us. IronPort A/S was 100% to us. Customers liked BM and we were unproven."
I initially misread that too. He's actually saying they faxed a notice to their customers that they were canceling the contract with Symantec. It's worded poorly.
In this situation it was quickly apparent that my role was to focus on a solution, not dwell on the problem or the cause. This should be true for everyone on the team, unless you are the specific individual who caused a global financial meltdown, deleted all the email or caused a critical failure.
It might be one of the rare situations where it was so bad, that after stabilizing the situation, many of the responsible parties needed to be put down.
I know from my experience working with Wachovia, the gross incompetence, system wide narrow minded greed and the fuck-everyone-but-me attitude was so bad it was mind boggling.
It was not a mistake, this was a concequence of the culture and goals of those companies.
I'm not sure this example is relevant, since this realization would not have been possible without a massive leg-up and pat on the head from your national government.
That is: "too big to fail" negates "near-implosion."
Though this probably shouldn't be exactly a surprise to me given how much shit people give me any time I try to talk about getting well after doctor's basically wrote me off for dead in some sense. No one wants to learn from that either and I honest to god don't get it.
I particularly agree with the point on leadership. If you can motivate those around you, people perform so much better and can sometimes come up with better solutions. Doesn't always guarantee you coming back from the brink, but stands you in good stead.
regardless, that's precisely why you need a cofounder who balances you. whether it's a different perspective, questioning some of your choices, or holding you accountable (and holding you up): you need it. you'll make an order of magnitude more poor choices on your own.
I like this, because it says two things at once, both true.
First, if you're in a position of ownership/authority but you're not willing or able to do the grunt work, you won't have the credibility for long. People will follow orders out of fear of getting fired, but you'll never get more than the bare minimum. There's a point where leaders are replaced or outnumbered by true executives (lazy, bikeshedding, rent-seeking parasites) and after that point, the organization can't even motivate shit to stink.
Second, it might be that it's just not worth it to lead from the front. That means that you're not really in a position of leadership. You might be a middle manager who realizes that the people above you will never buy in to what you're doing. Then you'll probably lose that desire to make those sacrifices. That's fine. You shouldn't tie yourself to the mast, at that point, because you're not really in a leadership role anyway.