Hacker News Reader

>nobody on staff has ever had to fend off Russian hackers attempting to take down your code at 3 am, nobody on staff has had to deal with thousands of users who wake up one morning, can't log in and no money is coming into the company. These are the battles that most programmers learn in the trenches of big companies, fighting tough battles, fixing complex problems under fire. It's not ideal, but I'm not sure if you can really become battle-hardened without going through real battles with real jobs on the line.

You make an interesting point, though I'm not sure that learning these lessons has to happen at a big company. I'm currently an engineer at a startup, but I came here from a small company—a ~30 person digital agency—where we were dealing with a lot of what you mention quite regularly. Because we had so many applications in production for a diverse range of clients (some at fairly large scale), and a small team of developers, the sorts of challenges you mention came at us much faster than they appear to arrive for us at my current company, which is focused around one product.

The fact that the dev team at my agency was small meant that, when facing issues like this, we were flying by the seat of our pants, learning as we went along and jumping between roles, and learning was happening very quickly. I've come to realize, after having left, that the range of experience I had in the small-team agency environment was, I think, extremely valuable.

Personal anecdote:

I've worked 5 years in 3 startups, 5 years in 2 large development teams, and my experience has shown that there are, percentage wise, better developers working in startups. This is in Austin, so it may not reflect the valley, but the absolute best programmers I've worked with are trying their hands at startups. There's a sweet spot of people aged 28-35 that are really skilled developers that still have an appetite for risk that are working in startups.

Do you think this is more a reflection of the limited quality of the larger players in the Austin scene?

I came to Austin from the Valley as part of an acquisition and have been looking for other employers in Austin, but the options for larger firms seem rather limited. I have yet to start exploring the small start-ups, which is why I am curious.

In Austin, it used to be the case that IBM (and possibly Motorola or a few other places) had a serious lock on all of the seriously good people. The projects were cool, the environment was excellent (if you didn't mind managers selected by vocal volume). I'm talking about the guys who did the cell processor, the AIX kernel folks, a bunch of people who did good UI work, and so on.

When the internet boom rolled around in the late '90s and the startup scene hotted up, most of the people in it were straight out of school. When I interviewed around in the mid 2000s, experienced people were kind of rare.

On the other hand, about the same time IBM became a consulting service company. The cool, visible projects evaporated. All of the good people I knew at the time either got buried deep somewhere (and are presumably happy in their niche) or left.

It's been years since I moved away from Austin, though, so I've got no idea what it's like now.

"This is what happened at Twitter - when the government came after them because they were getting hacked too often, Twitter went out and hired 'real' programmers and got things working. That's something that gets missed a lot in the 'startup story'."

That is interesting. Did you work at twitter or was this in the news?

I learned this at an OWASP conference, where several Twitter programmers revealed this and what they did to resolve it.

The guys who were presenting never said 'they brought us in to fix things', but that's exactly what they were. None of the guys who were building the 'hardened' Twitter were originally with the company - they were seasoned pros brought in from other companies.

And yes, they did some bad-ass things to fix it. But the presentation was more about OWASP level coders wanting to see what issues Twitter had to face. Nothing was Earth-shattering at that level, but it was excellent to see the amount of data they were working with and where their breaches were happening.

I don't want to speak for them, but it was obvious that one of their biggest issues was people writing insecure code and checking it in without any review. They have automated code review tools in place now.

Thanks for sharing this. I'm trying to convince my own company to switch to mandatory code reviews. Do you have a link to the talk or an article describing:

>it was obvious that one of their biggest issues was people writing insecure code and checking it in without any review. They have automated code review tools in place now.

http://sproutsocial.com/insights/twitter-security-breaches/

He might be referring to when Twitter moved form Ruby on Rails and did a complete rebuild in Java.

http://readwrite.com/2011/07/06/twitter-java-scala

http://www.zdnet.com/twitter-rebuilds-on-java-7000021241/

Maybe true, but my experience in big companies has been complicated.

1 - Less than stellar people, aiming for the bare minimum

2 - Crappy tools and procedures (like not having administrator priviledges on your machine), or using an old Unix (not Linux) with VI. Hard to get motivated with that

On the other hand they have money for shiny toys (like very good test equipment), so there's that as well.

I've worked for both startups and big companies. Both suck. Big companies have more money and process. Small companies have more flexibility but almost no process.

So, for example, if I work for massive XYZ bank, it's most likely that no code is going to get promoted without being thoroughly tested, discussed, multiple meetings and approvals, and then implemented by an actual team of people in charge of making sure the servers don't break. Of course, this means that it takes a LONG time to promote even simple changes and there is a world of politics involved with a bunch of agendas.

If I work for startup XYZ, I can promote right to prod without anyone asking what I'm doing or why I'm doing it at 3 pm in the afternoon. Of course, that also means several other developers might be doing the same thing. Your manager might be pushing code that breaks your code, and then telling the founders it was you.

For me, neither is good, but at least in a big company I get paid more, there is a structure in place that people have to follow (in general) and I can spend my free time working on my own projects. In a small company, you ARE the company but just like with the big company, you don't actually own anything...you are just on call all of the time and are paid less.

I have friends who prefer startups, it really boils down to which things you value more.

There are some startups out there that are more in between. I've had the good fortune to work at two such places.

One nice thing as mentioned in the article is that you have more responsibilities at startups. If you have the intelligence and drive, this can help you with your career immensely. Personally, I have learned a huge amount at the startups I have worked in, many things that larger companies probably would have taken a lot longer before putting trust in me to do.

It is all individual dependent of course.

If your experience was in Financial Tech, you likely had the very worst experience in "Big Company" you can imagine.

I worked at Wells Fargo, where they had laughable procedures and couldn't even properly staff teams because they refused to value their tech workers at even a bare minimum.

Now I'm at Google and its completely the opposite. A lot of healthy code management behavior, flexibility to design and develop new projects, lower maintenance management etc.

In between I ran a startup where I'd like to think I created the most process that was needed and nothing more, while personally managing my engineers by deeply valuing their opinion and only providing them with direction based on product strategy & design.

"Big company" and "startup" are not homogenous categories. I've worked for Big Companies that are slow and plodding and I've worked for startups that are as nimble and quick as lightning. But I've also worked for extremely nimble Big Companies and startups that couldn't make anything happen and were mired in a giant pile of technical debt and process.

Like many things in the startup world, the investors are the men behind the curtain pulling the strings on these myths. They perpetuate this image that startups are staffed only by super tech geniuses and big companies are staffed only by incompetent, washed-up slackers who've given up on their dreams because they want to exploit the cheap, inexperienced labor pool of naive new college graduates who will work for something "cool" for almost no money. Once you grow up and get a wife, kids, and mortgage, it isn't so important to work for the "cool" kids anymore.

There is a great wealth of expertise and mentorship that is missing in the startup world because investors throw a bunch of college kids together, tell them they're the coolest guys around, and pay them a pittance. If investors were a little more willing to offer a real salary to a person with the self-respect to demand at least market rates and reasonable working hours, I think we'd all be in a better place.

I disagree that it's usually the case that "really, really bad-ass programmers work for big corporations for good salaries." The number of "big corporations" that pay "good salaries" is actually quite small and, in this field, limited mainly to the usual suspects (Google, Facebook, etc.). This is one of those assertions that is basically not provable and usually involves a hedged tautology of some sort ("if you don't work at X, Y or Z, you're probably not bad-ass <--> (almost) all bad-ass programmers work for X, Y or Z").

That said, I've yet to have any discussions, interviews, etc. with startups that lead me to believe they're truly interested in "rock star" talent. Like most big companies, they want talent that is just sufficient enough to shove a marginal but profitable product out the door at a low enough price-point.

There is a lot, lot more to computing than the web. The guys working at HFT firms are serious badasses in programming terms, for example, and their pay dwarfs anything Google or Facebook could offer.

Yeah, the Valley definitely has "Web Goggles" on at the moment. That's why the "usual suspects" are the usual suspects--it's just what people out here seem to think of when they hear "programmer." It's a bit sad, actually. I moved here expecting the opportunities to be a bit more broad than they seem to be. As a result, I'm considering an attempt to move into the trading/finance space (I've a very analytical/quantitative background related to my prior life in scientific research, but little desire to return to that).

My friend got an offer with two sigma. He said no because Google was paying him a lot more. That's my anecdote. :)

I work for a (rather generic) government contractor and the pay around here is not out of line with what Glassdoor claims for Google. On one hand, I might be making more at the Goog' given my experience, but on the other I'm an idiot and spent far too many years working for lower pay at places that were "interesting" or "cool". On the other, other hand, I know many of the people around are making significantly more than I am (and we're not doing rocket science, by the way), and if I have any more hands, they're definitely not interested in "rock star" talent around here. This isn't HFT, either.

It's not unusual for developers working for commercial banks to make $150-200K (or more). That's in low cost of living areas such as Delaware and North Carolina at that. Government contracting is another non silicon valley gold mine. The work may be boring with red tape, but the problems usually are some what sophisticated and always mission critical.

If you want to dedicate 100% of your energy towards solving tough technical problems then you're actually not a good fit for a startup. Team sizes are small and resources are constrained, so you have to be well-rounded enough to solve any problem that comes your way, technical or not.

This is a bit of a tangent, but does "bringing in 'real' programmers" to clean up the mess really work? In my 1-2 years of experience (which isn't much), it seems like time onboarding into a big system (like Twitter) can take months, even for the all-stars. Is "we'll bring in the cavalry when shit hits the fan" a workable mindset for a startup?

(And this isn't trying to be glib, I'm genuinely curious -- I only have experience at a huge place that has tons of process for transfers and such).

*Rockstar web application developer with strong mobile development experience. Must be expert in SEO and Social Media.

> This is what happened at Twitter - when the government came after them because they were getting hacked too often, Twitter went out and hired 'real' programmers and got things working. That's something that gets missed a lot in the 'startup story'.

I'm sorry, but this is just silly. Dealing with hackers does not make you a "real" programmer. It just makes you a programmer with special expertise in a very particular and narrow aspect of programming - dealing with hackers. It's a niche skill and doesn't necessarily translate into prowess in other areas. Startups need eager, polyglot programmers who are willing to get their hands dirty, learn on the job, and deal with the uncertainty you wouldn't normally have to worry about in a large, stable company.

Plus, security of the software is more about company priorities then about some real vs unreal programmer differences. It requires repeated code reviews with focus on security, it requires occasional penetration testing and it is often in conflict with speed of development.

Startups often simply do not have time and money for it. (Big companies are often unwilling to pay for it, so their software ends up equally vulnerable.)

Kind of getting off topic, but big companies who do not do serious security end up on the news. Startups generally do not until they big companies.

Companies who take security seriously may even do daily scans. I have a major client who does penetration testing EVERY DAY. And all source code is security scanned by multiple third parties before it is released....AND you have to make sure that your code is bullet proof beyond that, because you not only have to pass the security scanners but also you have to be aware that if your company gets hacked, and they can point to code YOU wrote, you are in deep trouble.

It's more than just dealing with hackers. It's the overall scope of having to be responsible for critical work under heavy scrutiny. Most companies have a myriad of technologies...that isn't just the domain of startups. In fact, normally, startups have much more specialization. If you work as a coder at a big company, you might have to know Java, C++, .NET, PHP, Oracle, SQL Server, plus a million other things like old mainframe, COBOL, lisp, assembly, Ruby, Python...it depends on the job.

In a startup, you can say 'we only support Chrome and FireFox' and often no one will question it. But in a big corporation, you don't often get to pick which things you 'won't learn'. Some companies still use freaking IE6 (some very VERY big companies) so if you want your jQuery to work with their app, you'd better know why using anything higher than 1.11.x is a bad idea in general.

This is a niche skill industry. You can't be everywhere, if you intend to be good, and if you're good at something, you are probably not going to be good at completely different things. And someone very good at a small niche is almost always more valuable than someone who is kind-of-good at a number of things.

Startups may need "eager, polyglot programmers who are willing to get their hands dirty, learn on the job", etc., but by virtue of that they're eliminated from the market for many of the higher-skilled, specialized people.