luu

I have no idea how to answer this question. There have been some problems where someone was stuck for weeks, and I came in and coded up a solution in a day. That seems, in some sense, good evidence of being hard, but those problems never seem hard to me. The reverse happens to me too, where someone else solves a problem that was hard for me, and it's easy to them. Is that a hard problem? It was hard to me, but the problem was easy if I saw it the right way, which I didn't.

Last year, I came up with a solution to a problem that we'd been solving sub-optimally for years. My solution is arguably optimal (given a certain set of assumptions) and requires multiple orders of magnitude less code than the previous solution. The solution is the core part of a paper that was recently accepted to a top conference in its field. That sounds like it might be good evidence the problem was a hard problem, but in fact the solution just involved writing down a formula that anyone who was exposed to probability in high school could have written down, if it had occurred to them that the problem could be phrased as a probability problem (that is, the solution involved multiplying a few probabilities and then putting that in a loop). When I described the idea to a co-worker (that you could calculate the exact probability of something, given some mildly unrealistic but not completely bogus assumptions), he immediately worked out the exact same solution. It wasn't an objectively hard problem because it's a problem many people could solve if you posed the problem to them. The hardest part was probably having the time to look at it instead of looking at some other part of the system, which isn't a hard "technical problem".

Another kind of problem I've solved is doing months of boring work in order to produce something that's (IMO) pretty useful. No individual problem is hard. It's arguably hard to do tedious work day in and day out for months at a time, but I don't think people would call that "technically hard".

I'm with you. Maybe I never work on "REAL hard things"? How would I tell?

> I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable.

> Then again when you're Google and have the resources to build a VM runtime from the ground up it's easier to convince management that "This is the right decision!".

Is it possible you're either underestimating the effort it takes to make QEMU solid or overestimating the effort it takes to write an emulator?

I worked at a company where I hacked up QEMU as a stopgap before we switched to an in-house solution (this wasn't Google, although I've also worked at Google). I made literally hundreds of bug fixes to get QEMU into what was, for us, a barely usable state and then someone else wrote a solution from scratch in maybe a month and a half or two months. I doubt I could have gotten QEMU into the state we needed in a couple months. And to be clear, when I say bug fixes, I don't mean features or things that could possibly arguably be "working as intended", I mean bugs like "instruction X does the wrong thing instead of doing what an actual CPU does".

BTW, I don't mean to knock QEMU. It's great for what it is, but it's often the case that a special purpose piece of software tailored for a specific usecase is less effort than making a very general framework suitable for the same usecase. Even for our usecase, where QEMU was a very bad fit, the existence of QEMU let us get an MVP up in a week; I applied critical fixes to our hacked up QEMU while someone worked on the real solution, which gave us a two month head start over just writing something from scratch. But the effort it would have taken to make QEMU production worthy for us didn't seem worth it.

> today's new developers are die-hard black boxers

How is anything else even possible? I spent eight years doing CPU development and even at the time I wouldn't say that I understood how CPUs worked. No one understands how an entire desktop or server CPU works today. They're too complicated. Back in grad school, I had a fairly good solid state & device physics background, stronger than most EEs. Ten years later, if I now want to understand how a modern transistor works, I expect it would take me 1-2 months to get a good high-level picture. And that's just the digital stuff -- I never had a good picture of how analog circuits work, and understanding that would take years.

In terms of complexity, the CPU is much less complex than the compiler or OS I use. For that matter, it's less complex than the browser I'm using to type this comment. If you don't believe that, look at the number of person years that goes into developing a CPU vs. the amount that's gone into any of those pieces of software. Relatively speaking, CPUs are simple. And they're still so complex that no human can understand them.

There's no way to understand the entire stack below you unless, maybe, you're doing materials science work and don't have anything else below you in the stack. And even then, "understand" is sort of an iffy term to use. When I was up to date in the area, there were a lot of open questions on phenomenon that worked but had no proven "why". While I haven't looked into it, it's a pretty safe bet that the total number of such mysteries has increased.

This isn't to say that people shouldn't develop some understanding of the building blocks they're using. Rather, understanding is a difference in degree, not a difference in kind. There's no such thing as understanding everything you use. There's just, at the margin, spending a bit more time on understanding components vs. spending a bit more time understanding how to use components.

> If you have tens of thousands of variables, you definitely don't want to risk dealing with the simplex method's worst-case behavior.

The risk is extremely low. In fact, the risk is so low that, for deacdes, it was an open question why the simplex algorithm was so good in practice on a wide variety of workloads despite having poor worst-case complexity (unlike, for example, pure quicksort, which is often problematic in practice).

Spielman solved that problem in 2001 and showed that even very small perturbations from a contrived worst-case input have good performance: http://arxiv.org/pdf/cs/0111050.pdf

I'm happy to hear that this is finally public so I can actually talk about the work I did when I was at Google :-).

I'm a bit surprised they announced this, though. When I was there, there was this pervasive attitude that if "we" had some kind of advantage over the outside world, we shouldn't talk about it lest other people get the same idea. To be clear, I think that's pretty bad for the world and I really wished that they'd change, but it was the prevailing attitude. Currently, if you look at what's being hyped up at a couple of large companies that could conceivably build a competing chip, it's all FPGAs all the time, so announcing that we built an ASIC could change what other companies do, which is exactly what Google was trying to avoid back when I was there.

If this signals that Google is going to be less secretive about infrastructure, that's great news.

When I joined Microsoft, I tried to gently bring up the possibility of doing either GPUs or ASICs and was told, very confidentially by multiple people, that it's impossible to deploy GPUs at scale, let alone ASICs. Since I couldn't point to actual work I'd done elsewhere, it seemed impossible to convince folks, and my job was in another area, I gave up on it, but I imagine someone is having that discussion again right now.

Just as an aside, I'm being fast and loose with language when I use the word impossible. It's more than my feeling is that you have a limited number of influence points and I was spending mine on things like convincing my team to use version control instead of mailing zip files around.

The numbers don't seem obviously way off when compared to the numbers of folks I know. This is with the caveat that, for example, I don't know the compensation of that many people who are T6 at Google or at Amazon at any level. The only thing that looks off is that the numbers for Microsoft look low.

Why would that be? I suspect that's more because the pool of people I know is biased than because the pool that posted is biased -- some orgs in Microsoft will match offers from other companies even though average compensation is relatively low, so if you're new and you know a lot other people who are new, the sample of people you know is probably biased high.

To be clear, I think that's bad both for Microsoft and for Microsoft employees. I'm just trying to figure why the Microsoft data is anomalous relative to the numbers people have personally told me. The reason it's bad for employees is obvious. The reason it's bad for the company is that it's pretty common for people who have been here a long time to shop around, find out that they can make much more elsewhere, and then get a matching offer from MS. Once word gets around that the best way to get a fair raise is to interview elsewhere, people start interviewing elsewhere, and some of the people who interview are going to leave, even if that wasn't their original intent.

Right now? I'm pretty unhappy. I don't think I want to talk about exactly why publicly while I'm still in this job, but to give you an idea of how bad the situation is, literally all of my friends have been trying to get me to quit for months, with the exception of people who have given up because they think I must be insane.

On the other hand, I have a decade of full-time experience and I've been happy for about seven out of ten years. All things considered, that's not too bad. The other way to look at it is that I've had maybe five roles at one company, two another another, and one at a third, and I'd say four of those have been good. That's only 4/8, but it's possible to bail on bad roles and stay in good ones, which is how it's worked out to being good 70% of the time. Considering how other folks I know feel about their job, I can't complain about being happy 70% of the time.

In retrospect, some of my decisions have been really bad. If I could do it over again, I'd bail more quickly on bad roles and stay in good ones for longer.

My dumbest mistake was the time I was in an amazing position (great manager & team, really interesting & impactful work), except for two problems: an incredibly arrogant and disruptive person whose net productivity was close to zero who would derail all meetings and weird political shenanigans way above my pay grade. When I transferred, management offered to transfer the guy the guy to another team so I'd stay and I declined because I felt bad about the idea of kicking someone off the team.

From what I've heard, the problematic dude ended up leaving the team later anyway, so not having him kicked off didn't make any difference, and the political stuff resolved itself around the same time. The next role I ended up in was the worst job I've ever had. And the one after that is my current job, which is, well, at least it's no the worst job I've ever had. Prior to leaving the amazing job, I thought that it was really easy to find great jobs, so it wasn't a big deal to just go find another one. Turns out it's not always so easy :-). If I hadn't bailed on that and just fixed it, I'd be 4/6 and I could say I was happy with my job 80% of the time. Oh well, lesson learned. Looking back, I was incredibly lucky to get the roles that I did, but that same luck blinded me to the fact that it was luck and that there are some really bad jobs out there.

As someone who's spent the majority of their working life designing CPUs (and the rest designing hardware accelerators for applications where CPUs and GPUs aren't fast enough), I find that when people say something like "If you could make your processor 5% faster with a simple change, why wouldn't you do it?", what's really meant is "if, on certain 90%-ile or 99%-ile best case real-world workloads, you could get a 5% performance improvement for a significant expenditure of effort and your choice of a legacy penalty in the ISA for eternity or a fragmented ISA, why wouldn't you do it?"

And the answer is that it there's a tradeoff. All of the no-brainer tradeoffs were picked clean decades ago, so all we're left with are the ones that aren't obvious wins. In general, if you look at a field an wonder why almost no one has done this super obvious thing for decades, maybe consider that it might be not so obvious after all. At zurn mentioned, there are actually a lot of places where you could get 5% and it doesn't seem worth it. I've worked at two software companies that are large enough to politely ask Intel for new features and instructions; checked overflow isn't even in the top 10 list of priorities, and possibly not even in the top 100.

In the thread you linked to, the penalty is observed to be between 1% and 5%, and even on integer heavy workloads, the penalty can be less than 1%, as demonstrated by the benchmark linked to above. Somehow, this has resulted in the question "If you could make your processor 5% faster ...". But you're not making your processor 5% faster across the board! That's a completely different question, even if you totally ignore the cost of adding the check, which you are.

To turn the question around, if people aren't willing to pay between 0% and 5% for the extra security provided, why should hardware manufacturers implement the feature? When I look at most code, there's not just a 5% penalty, but a one to two order of magnitude penalty over what could be done in the limit with proper optimization. People pay those penalties all the time because they think it's worth the tradeoff. And here, we're talking about a penalty that might be 1% or 2% on average (keep in mind that many workloads aren't integer heavy) that you don't think is worth paying. What makes you think that people would who don't care enough about security to pay that kind of performance penalty would pay extra for a microprocessor that gives has this fancy feature you want?

If you read the thread, you'll see that the person who actually benchmarked things agrees: someone implemented integer overflow checks and found that the performance penalty was low, except for microbenchmarks.

If you click through to the RISC-V mailing list linked to elsewhere in this discussion, you'll see that the C++17 standard library is planning on doing checked integer operations by default. If that's not a "performance focused language", I don't know what is.

There is no chicken and egg problem for most workloads. Processors are quite good at handling correctly predicted branches, and overflow checks will be correctly predicted for basically all reasonable code. In the case where the branch is incorrectly predicted (because of an overflow), you likely don't care about performance anyway.

See http://danluu.com/integer-overflow/ for a quick and dirty benchmark (which shows a penalty of less than 1% for a randomly selected integer heavy workload, when using proper compiler support -- unfortunately, most people implement this incorrectly), or try it yourself.

People often overestimate the cost of overflow checking by running a microbenchmark that consists of a loop over some additions. You'll see a noticeable slowdown in that case, but it turns out there aren't many real workloads that closely resemble doing nothing but looping over addition, and the workloads with similar characteristics are mostly in code where people don't care about security anyway.

This is timely, as I’ve just gone through the worst onboarding I've ever experienced. It took two very long days to install an OS on a machine and get it to boot (I had to assembly my computer and then use the corporate installer, which failed multiple times with errors like “an unknown error occurred”). It then took me about a week to run the local project’s “hello world” (I was initially pointed to documentation that had been deprecated for months, and then when I found the right documentation it was incomplete and out of date). The actual process to get there was hilariously baroque; for example (if my email records are correct and I didn’t get duplicate emails from the system), I must have clicked through 18 EULAs to get a subset of the permissions I need. I promise, that’s worse than it sounds since the form to do so took more time to click through than you can imagine. It was a month before I could effectively do any work at all.

I originally thought that I was just unlucky, but when I asked around I found that I had an above average experience, at least among people near me who started recently. Unlike many other folks, I was actually able to get a username/alias, a computer, and an office. I don’t know what folks without a username do since they can’t even start clicking through the necessary EULAs to get permissions to do stuff, let alone do any actual work.

I’m not sure how it is in other parts of the company, but if you imagine that it’s similar and do some back of the envelope math on how many people get onboarded and how much losing a month (or more) of time costs, it comes out to N million dollars for a non-trivial N. And that’s just the direct cost of the really trivial stuff. It’s hard to calculate the cost of the higher level stuff that’s mentioned in the article, but I suspect that’s even more expensive.

Who handles support for this? Is it Google, or one of the carriers that Google's using? Who do you talk to if the handoff between carriers doesn't quite work? I've skimmed the FAQ and I didn't see that listed in there.

For context, I use Google Voice, and while it's really nice in a lot of ways, it has some unfortunate bugs that have persisted for years that it seems impossible to get support for. I no longer unconditionally recommend it to people because while being able to take calls from my computer and get transcripts of voicemails is great, it also pseudo-randomly decides to only direct some calls to devices where I'm not present with no notification on devices that I'm using and also drops a small percentage of texts (my guess is around 1%). Most people I talk to are horrified by the idea of texts getting dropped, and I don't recommend GV to those folks, although it's fine if that's a tradeoff you're ok with.

I understand that it's a beta and that bugs are expected, but it's a problem when you run into a bug that interferes with your ability to use your phone and there's no way to get support for it.

Phone carriers aren't exactly known for their stellar customer service, but at least with them, if you call back enough times or escalate to higher level support, you'll eventually get a resolution. As far as I can tell, that's not true of Google's phone related products.

This is really neat and I love playing with stuff like this. But, fundamentally, it shows that stars aren't a good measure of anything besides stars, at least outside of the top scorers for commonly used languages. For those, it's not a bad proxy for fame.

If you like at the top javascript or ruby developers, yep, those are all pretty famous javascript and ruby developers. But if you look at the #6 matlab developer, well, turns out that's me. I've probably used matlab for less than 40 hours total, lifetime. And most of that was in grad school, a decade ago. Most of my stars come from a tutorial. Not a tutorial I created -- a tutorial I worked through, that thousands of people have probably done. Ok, so, not many people put matlab code on github, so that data is messy. What about popular languages?

Turns out I'm also the 240th most starred scala developer worldwide. I once used scala for two months and created some projects to help me learn that aren't even close to being polished enough to be useful to anyone. Like most code written by someone who's learning a language, it's not any good. But that somehow puts me at 240? Even in a pretty popular language, by the time you get into the hundreds worldwide (or the top few in most cities), it's people who just threw up some toy projects.

I wonder if this explains why I've been getting recruiters contacting me "because they saw my scala code on github". I doubt anyone who's actually seen my scala code on github would contact me for a scala position, but someone who uses a tool that counts stars might think that I actually know scala and contact me for a scala position. This particular tool is too new to be the source of that, but the page the source data comes from (github archive) shows how easy it is to make BigQuery queries to return results like this.

For Julia, I'm also presently ranked above all of the co-creators of Julia, despite having spent a total of perhaps 20 hours ever using the language (I'm 72, compared to the co-creators, who are 113, 143, and unranked).

BTW, in languages I've actually worked in professionally, I'm 98,582/244,375 in a language I used for years before it became trendy, 1,100/1,835 in a language I've used a lot recently, and 75,998/161,465 in a language I've used some recently. In the language I'm most proficient in, the language I'm mostly likely to reach for if I just want to get things done, I'm 14,800/25,094.

P.S. If the developer is reading this and wants bugreports, your service returns a "503 Service Unavailable" if you click the "top foo github developers in your city" for developers that don't have an associated city.

Am I the only one who thinks that this kind of design question is the new "how many gas stations are there in Manhattan?"

In both cases, the point isn't to get the right answer, but (allegedly) to see how the person thinks. With the estimation question, the trick is to make up some plausible-ish numbers and then multiply and/or add them to get a plausible-ish result. If someone's seen one of those before, they'll nail it for sure. If not, it's a crapshoot. In theory, you're measuring whether or not someone's able to reason well enough to spontaneously estimate something off the top of their head, but comparing the number of people who've seen those question before vs. the number of people who can answer that type of question without having ever seen it before, what you're really filtering for is people who have heard of or seen Fermi questions.

A while back, someone's interviewing experience got posted to HN and they mentioned that they got asked to design a URL shortening service maybe five times. They failed it the first couple of times and got progressively better each time. I doubt the interviewers meant to measure whether or not this person had done enough interviews to catch on to the style of questions that are currently trendy, but that's what they actually measured.

I think the most common objection to this is that the point of these questions is to drill down and figure out how the person really thinks, but in empirical studies on interviewing, they find the actual evaluation of chatty questions like this is heavily influenced by all kinds of biases. Techniques that have more clear-cut evaluation criteria, like work sample tests, and even completely non-technical interviews like behavioral interviews end up being better filters. Even then, the filtering isn't "good" (IIRC, the last time I read one of those studies, work sample tests score the best, and had a correlation of around .5 with an ideal filter), but it's better.

For these kinds of questions, even if you haven't seen the specific question before, there's all sorts of interview gamesmanship that helps tremendously. One thing in the blog post, not spending an hour on requirements gathering, is an example of that. In real life, if you're going to write an application from scratch, spending more than an hour on requirements gathering is perfectly reasonable for a lot of problem domains. But if you're playing the interview game, you know that you have, at most, an hour to sketch out the entire problem, so you have to cut the requirements gathering phase short (but not too short). The post mentions that this gets at real skills. That's true. It does. The problem is that everyone who's seen this kind of question five times is going to be good enough at the interview gamesmanship that they don't need to have good real skills to breeze through the "don't spend too long talking about requirements" sub-filter.

The claim:

“Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages.”

But how did they determine that?

The authors looked at the 50 most starred repos on github, for each of the 20 most popular languages plus typescript (minus CSS, shell, and vim). For each of these projects, they looked at the languages used (e.g., projects that aren’t primarily javascript often have some javascript).

They then looked at commit/PR logs to determine figure out how many bugs there were for each language used. As far as I can tell, open issues with no associated fix don’t count towards the bug count. Only commits that are detected by their keyword search technique were counted.

After determining the number of bugs, the authors ran a regression, controlling for project age, number of developers, number of commits, and lines of code.

That gives them a table (covered in RQ1) that correlates language to defect rate. There are a number of logical leaps here that I’m somewhat skeptical of. I might believe them if the result is plausible, but a number of the results in their table are odd.

The table “shows” that Perl and Ruby are as reliable as each other and significantly more reliable than Erlang and Java (which are also equally reliable), which are significantly more reliable than Python, PHP, and C (which are similarly reliable), and that typescript is the safest language surveyed.

They then aggregate all of that data to get to their conclusion.

I find the data pretty interesting. There are lots of curious questions here, like why are there more defects in Erlang and Java than Perl and Ruby? The interpretation they seem to come to from their abstract and conclusion is that this intermediate data says something about the languages themselves and their properties. It strikes me as more likely that this data says something about community norms (or that it's just noise), but they don’t really dig into that.

For example, if you applied this methodology to the hardware companies I’m familiar with , you’d find that Verilog is basically the worst language ever (perhaps true, but not for this reason). I remember hitting bug (and fix) #10k on a project. Was that because we have sloppy coders or a terrible language that caused a ton of bugs? No, we were just obsessive about finding bugs and documenting every fix. We had more verification people than designers (and unlike at a lot of software companies, test and verification folks are first class citizens), and the machines in our server farm spent the majority of their time generating and running tests (1000 machines at a 100 person company). You’ll find a lot of bugs if you run test software that’s more sophisticated than Quickcheck on 1000 machines for years on end.

If I had to guess, I would bet that Erlang is “more defect prone” than Perl and Ruby not because the language is defect prone, but because the culture is prone to finding defects. That’s something that would be super interesting to try to tease out of the data, but I don't think that can be done just from github data.

This is one of those things that's well known by people who spend a lot of time doing low-level nonsense that really ought to get wider press. You really don't want to use x86 hardware transcendental instructions.

I've seen people try to measure the effectiveness of their approximation by comparing against built-in hardware, but their approximation is probably better than the hardware! If you're not writing your own assembly, you'll be ok on most modern platforms if you just link with "-lm" and use whatever math library happens to be lying around as the default, but it's still possible to get caught by this. On obscure platforms, it's basically a coin flip.

I used to work for another CPU vendor, and when we implemented more accurate sin/cos functions, some benchmarks would fail us for getting the wrong result.

Turns out those benchmarks hardcode a check for a couple results, and that check is based on what Intel has been doing for ages. My recollection is that we had a switch to enable the extra accuracy, but that it wasn't enabled by default because it was too much of a risk to break compatibility with Intel.

If that sounds too risk averse, there's a lot of code out there that depends on your processor precisely matching Intel's behavior on undefined flags and other things that code has no business depending on. It's basically the same thing Raymond Chen is always talking about, but down one level of abstraction. I've seen what happens if you just implement something that matches the "spec" (the Intel manual) and do whatever you want for "undefined" cases. You get a chip that's basically useless because existing software will crash on it.

I love that this has a big fat "reproduce the results" button with detailed build instructions.

Why isn't reproducibility required to publish in CS? Unlike in fields like psychology or chemistry, reproducing results should be trivial if the authors provide instructions on how to do it.

Cutting costs isn't IBM's strategy to get out of the recession. Cutting costs is IBM's strategy. I interned there in 2003, and it was already apparent that they preferred to hire people overseas and let attrition reduce the ranks of folks locally in order to reduce costs. From all the complaints, it was clear at the time that was causing serious problems with R&D.

The joke around the office was that we'd replace one person locally with three people overseas, losing the output of two people, since the three overseas people were so clueless they'd need a local person holding their hands full-time. I was doing microprocessor design at the time, and the community is small enough that everyone know that Intel was getting good folks overseas. But even at the reduced wages outside the U.S., IBM was cutting corners and not hiring the best people.

Even locally, they try to reduce costs. A friend of mine who stuck around long enough to make it into management told me that they try to keep salaries at about the 40%-ile to save costs. On finding that out (as well as a few other gems), he left. Until I heard that, I couldn't figure why brilliant friends of mine often got raises that didn't even cover inflation. Don't they know that people are going to leave because of that? They know, and that's their strategy.

The thing about Bernanke comes out of left field. IBM didn't use low interest rates to invest therefore "the companies that were expected to spend us back to better economic health didn’t do so" therefore low interest rates didn't make the recession less severe than it otherwise would have been? No comment on whether or not those last two statements are actually true (I'm not an economist and haven't studied the issue), but Cringley certainly doesn't make a case for them.

I was just talking to a friend of mine at amazon, who said that a lot of people (including some pretty high-level people) thought prosecuting the one-click patent was a mistake because it actually made it harder to recruit.

That's nothing compared to this. Seems like a classic case of the left hand not knowing what the right hand is doing.

When I look at the best programmers I know, none have them have done any open source work whatsoever. This group of people ranges from IBM fellows who created some of the earliest minicomputers to IMO medalists who are unbelievably fast problem solvers. I'm not going to say that's a generalizeable statement. It's not, and that's the point -- it's a side effect of spending a career working in teams where people care more about playing with their kids or doing research or whatever than open source.

It's not even that they don't work on projects in their spare time. They do, but they're not part of open source culture, where people share things by default. They're satisfied with having made the thing itself, and they're not going to put their hardware or software project on github any sooner than they're going to make a website for the house they built from scratch.

If you work at an open source shop and hire with this attitude, all you're going to do is make sure that you have a monoculture in this dimension. That's not the worst thing in the world, but it filters out people who don't particularly care about open source, which is the majority of programmers. If that's a moral imperative for you, I don't have any objection to that. Otherwise, I don't see why you'd want to do this.

When I interviewed with Tesla, before the technical phone screen, there was an HR phone screen that consisted, mostly, of an HR person trying to sell me on the company with statements like "we don't pay as well as Google or Facebook, but you'll learn so much more because you'll be part of a small team".

I wasn't convinced that I should take a below market salary so that someone who was (at the time) worth almost $10B could make another billion.

Musk is a visionary, in a number of ways. He has an impressive ability to come up with great ideas, and an impressive ability extract rents from people much poorer than he is, whether that's the average tax payer or the average Tesla employee.

For a decade, I've gone after the most interesting thing I could find without regard for visibility. My current job is the most interesting thing I've done so far, but it's likely that my employer won't disclose my current project until after another company publicly discloses they're doing the same thing. Looking at how that's worked for other systems/infrastructure projects around here, I doubt I'll be able to talk about my job for N years, where N is a significant fraction of a decade.

Now that I'm a year out from my previous job, I can talk a bit about what I did back in the 2010 timeframe. By the time I'm able to talk about what I did for them in 2012, it will be ancient history.

And yet, my career has been full of interesting work and I regularly get inquiries for neat sound jobs based on (as far as I can tell) nothing more than my having a pulse. It's true that, on occasion, someone who's familiar with my work contacts me. But much more often, it's someone who has no idea what I've done who's desperate to hire because, nowadays, everyone is desperate to hire.

I expect that, one day, the job market will cool down and I'll have a bit of trouble finding something I really enjoy. But, from where I am now, the tradeoff seems worth it, even if that day is tomorrow.

Everyone has different reasons for their choices, but speaking only for myself, none of those things matter much to me. Here's why I'm currently at a big company instead of a startup (and I've done the startup thing in the past, so I have some idea of what I'm missing out on):

1. Research-y projects. Big companies have the resources to place long-term high-risk bets on research projects that will probably fail. Some startups have a single such high-risk bet, but you can't just decide to work on whatever crazy idea you come up with because the company can't really absorb your salary for years if the bet doesn't pay off.

2. Really broad scope. One cool thing about working at a startup was that I could do whatever I wanted as long it was relevant to the startup. In fact, that was the biggest reason I went with a startup just out of school; I would have been some tiny cog on a giant team at a big company (that's not inevitable, but that's what the offers I personally had entailed). But I now have enough credibility that I can go off and do whatever I want at a big company. Since the big company has a much larger scope than a startup, there are many more things I can work on.

3. Resources. At the startup, I could take 100 machines to run an experiment and it wasn't a big deal. If I wanted to use 1000 machines I'd have to get people to ok it because I'd be eat into resources that were necessary for the company's operation. Now, if there's something I'm curious about, I can run a map reduce across the entire internet.

4. Relaxed environment. The vast majority of my team is married with kids, so the office is deserted by 5pm. There are a couple of folks I've seen stay late for a week or two to hit some deadline, but that's like one or two weeks out of the year. The startup was one of the most relaxed startups I've ever seen, but there were still month long crunches of 50+ hours/wk as some emergency came up. In principle, a startup could be as relaxed as a big company, but that's extraordinarily rare. At the big company, one of the most respected people in the office casually remarked to me that he's been at negative available vacation for all but his first two weeks at the company (and he's been here for seven years). At the startup, the most respected people all lost vacation every year because pushed up against the vacation cap.

5. This should really be like 25 or something, since it's not in the same league as the others, but I'm making a lot more money than I was at the startup.

Don't get me wrong -- there are a lot of advantages to being at a startup, and I'm glad I spent a good chunk of my career at one. I'm just saying that there's a lot I'd have to give up to go back to a startup, and the things on that list don't even make the top 30.

I don't understand why this sort of thing happens. At Intel, I got the silent treatment after getting an offer. It was for an implementation (RTL -> circuit) position. We chatted about schedule and it was apparent that they wouldn't actually have a pressing need for that sort of thing for at least six months, and that they were hiring as they could and they expected to be hiring implementation folks for at least another six months to fill all of their reqs.

When we got around to discussing start date I asked if I could take a few months off between jobs and start in a few months. The hiring manager (who was an engineer, not a recruiter) stopped responding to my emails and returning my calls. Months later, I asked a friend of mine what happened and was told "X is busy". Too busy to spend five seconds responding to an email? Really?

Today, that hiring manager is at ARM and I declined an a chance to interview in his group when I found out who was in charge. I think I dodged a bullet; I can only imagine what working for the guy is like.

By contrast, I applied for a job I was completely unqualified for at another company around the time I interviewed at Intel. I didn't expect to even get an interview, and I didn't. But I still got a nice snail mail rejection notice. Many years later, I now work for that second company.

Like I said, I don't understand why this sort of thing happens. Putting aside kindness and basic human decency, even if you're completely selfish, it costs basically nothing to be polite, and the benefits are pretty large. Conversely, being rude gains you nothing and has noticeable costs.

Funny, my experience is the opposite on literally every concrete assertion about people and motivations.

Looking at my team, and the broader team under my director, only one was hired "straight out of college". Well, if you consider fresh PhDs to be "straight out of college", there are a few, but I doubt Aaron meant folks approaching 30 with kids on the horizon when he was talking about people just out of college.

Perhaps a couple are cynical, but I wouldn't call any childish or enthusiastically adolescent, which isn't too surprising, considering that the median employee within two levels of management of me is mid-thirties with two kids.

We have a couple of visiting scientists (professors at major research universities), and they were surprised by what we're doing, so perhaps the secrecy doesn't do such a bad job of keeping things inside the company after all.

I don't doubt Aaron's experience, but it's exactly what you'd expect due to selection bias. How old was Aaron when he wrote that, 20? He probably wasn't hanging out with the median person from my team. There's nothing wrong with that. But, all things considered, that essay contains a lot of awfully strong assertions.

Personally, I've found that I don't retain much of this sort of material without working through exercises. If you learn the same way, you might want to check out the series of progressive exercises from Andrew Ng here: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial

For reference, I have a copy of my solutions here: https://github.com/danluu/UFLDL-tutorial. Debugging broken learning algorithms can be tedious in a way that's not particularly educational, so I tried to find a reference I could compare against when I was doing the exercises, and every copy I found had bugs. Hope having this reference helps someone.

As usual for HN, all of the most upvoted replies disagree with the article. Let me play devil's advocate and not disagree. Sure, the analogy isn't perfect, but what happens if we take the idea seriously? [1]

Isn't it amazing, the lack of sacrifice necessary to make fat stacks of cash writing software? Even when law school was thought of as a golden ticket, it was a lottery, and to win, you had to sacrifice your personal life for a decade before you made partner. And those hours looked relaxing, compared to medicine and investment banking. Back when auto jobs were a sure thing, they were also sure to use up your body by the time you could retire, and that's if you were lucky enough to avoid a career ending (not to mention crippling) injury.

I just met a kid who graduated with a BA in philosophy who was offered 4x the median U.S. income [2] at a software company. A prop trading firm offered him 50% more, and the software company matched the offer. One reason he turned down the offer at the prop trading firm is because people there regularly worked 50+ hours a week.

The most upvoted article on HN from a couple weeks ago was full of comments debating whether 20% time is really 120% time, at a company where mid-level ("senior") engineers can have total compensation that's something like 8x the median income in the U.S. And was outrage! Outrage!

Last year, coursera ran a course on deep learning from one of the guys who's widely credited with inventing deep learning. The pre-requisites were some basic programming, and, either, google + wikiepdia, or a basics of machine learning course, like the one that's offered on coursera regularly. After taking the course, you'd have enough understanding of deep learning to reproduce papers published that year, on the state of the art in machine learning.

Never have we had a privileged class that's so easy to enter. It's genuinely surprising that this is the case. You don't have to be born into the aristocracy. There's no licensing body limiting the number of developers, and no hazing process that makes requires giving up the best years of your life. The knowledge is available to anyone with a computer and an internet connection, and those are cheaper than they've ever been in human history. You might say that there's just not enough "smart" people in software, but, that's part of what's surprising.

Why do so many people who want a career involving intellectual curiosity study philosophy or mechanical engineering, when CS also gives you interesting problems, and happens to pay much better? Why don't people switch? Unlike with ME, CE, etc., you don't have to take the PE and get all sorts of licensing to find work. You just need to be able to pass some interviews. I met folks at Hacker School [3] who switched from econ, ME, OR, and other quantitative fields to CS, because you have more freedom to pursue ideas, can do more without being part of a huge team that makes you a tiny cog in a giant machine. And, by the way, it pays twice as well. But, it's still not common to see people switch.

What's the barrier to entry that's keeping us from being flooded with supply? I'm told that CS enrollments are now at record highs, past even the numbers we saw during the dotcom era; perhaps the answer is that there is no barrier, and we're about to get flooded with supply.

[1] In his notes, Lectures on the History of Moral Philosophy, Rawls talks about how his students eagerly come up with clever refute the propositions of great thinkers. He takes the opposite approach. It's been a long time since I've read this, so I'm very loosely paraphrasing, but it's something like, if you disagree with someone who's clearly very smart, maybe it's worth taking the time to figure out why they hold their opinions rather than just dismissing them.

[2] median personal income in the U.S. is about $30k/yr.

[3] https://www.hackerschool.com/

Google - Madison, WI. Sorry, no remote work, but Google does sponsor visas. All levels of experience welcome. We've recently hired an ACM fellow, as well as a new college grad.

Of course Google is hiring. So, why I posting this? Every time I tell someone I'm working at Google in Madison, they're shocked that there's an office in Madison, and I often hear people complaining about the lack of interesting technical work in Madison. There's fun technical work in Madison, I promise.

I'm working on a hardware/software co-design project that's attacking a fundamentally hard problem, which started as a 20%-time project. There are a couple other hardware projects in the office; most hardware projects start as prototypes of crazy ideas, and go from there. The majority of people here are doing low-level systems programming, usually networking related, and a handful of people are doing data analysis (call it big data, if you like) to figure out how to optimize Google's next generation hardware and software platforms. I'm sorry I can't describe projects in much more detail -- Google is pretty secretive about what goes into datacenters.

The office is small (just under 30 people), and manages to avoid any bureaucracy you might expect from a big company. The work is interesting enough that in the five year history of the office, only one person has left (and he retired to a ranch in Nebraska). Feel free to email me (see profile) if you have any questions.

https://www.google.com/about/jobs/search/#!t=jo&jid=45087&

https://www.google.com/about/jobs/search/#!t=jo&jid=2069001&

Edit: Interesting to see this downvoted. If you're downvoting this, I'd be curious to know why. Because Madison is in the middle of nowhere and you don't care about Madison? Because you don't like big companies? Because you're cynically trying to push your job post above this one? Because you think job postings need bullet points?

This is indistinguishable from selection bias.

If you want to write the opposite article, pick any three wildly successful companies. They've all grown enough that they've had to move into a new building. Pick a project that went well around the time of the move; they're big enough that there must be at least one. Now, attribute causation and come up with some pithy bullet points. Ta-da! You've got yourself a wsj article.

There are a lot of comments saying this isn't feasible. I disagree. I don't mean this as an appeal to authority, but, since people are attacking Steve Blank's background, I used to design CPUs, and I've done pretty much everything except low-level stuff like layout (which I only did in classes).

I have no opinion on whether or not there is a backdoor, but, here are some possible mechanisms.

1. Periodic SMI on steroids. Intel used to have a debug mode based on an SMI++ like mode, where the chip would periodically dump the entire state of the machine out to memory. That's not nearly as useful for debugging now as it was 15 years ago, but it could dump out compromising information to some buffer that your network card DMAs out.

2. RNG weakness.

3. Put the machine into ring 0, with no other changes.

4. Put the machine into ring 0, while transferring control to some address.

5. Access to the microcode patch mechanism.

It took me about 15 seconds to come up with those ideas (I thought of 4 when I wrote down 3). Regardless of what you think of the NSA, they have the of the best security people in the world. They can probably figure something out.

Any of these things could easily be triggered by a sequence of obscure instructions. There are plenty of userland instructions that are never used today. An arbitrary sequence of, say, 20 of them is likely to never be discovered even by brute force attack. If you're really worried, you can load up a few registers with some specific values, and now you've got a 192-bit keysize (or more, if you want).

If you want to keep it secret from the companies themselves, you're probably better off using a secret register key in some microcode instruction, since that would be relatively easy to surreptitiously sneak in after the official tapeout. Looking for a sequence of instructions wouldn't be technically difficult (I doubt the decoder/translator is on the critical path, but, Intel's might be custom, in which case it would be a lot of work to find spare space), but, it would be more work to sneak it in seamlessly. Then again, they almost certainly have free gates lying around so that post-silicon bugs don't require a full-layer tapeout to fix. You could write a program that edits the right mask layers to access those and patch your change in. But, if those actually get used for debug purposes, you'd lose the ability to make the change seamlessly, and, it's much more work than the first approach.

I suspect '5' would require restarting the machine, although you could design a mechanism that lets you hot-swap microcode. Doesn't seem worth it, though, considering how easy it is to compromise a machine if you control the hardware.