tlipcon

If only that were true -- the shuffle is typically seek-bound when the intermediate data doesn't fit into cache (plenty of papers show this pretty conclusively).

Cloudera isn't primarily consulting. In fact we have about 8x as many engineers in product development compared to consulting, last I counted.

Early in your career, trading salary for experience is a smart move. In my case, I turned down a Google offer to work at a startup after college (taking a 15%-30% pay cut) and my equity ended up being worth less than I paid to exercise it. But, in the two years that I was there, I learned way more than I would have at Google, and got a much bigger network, both of which helped immensely when I interviewed for my next position.

I agree, working for peanuts is dumb. But 20%-40% less than what Google offers is hardly poverty -- it's still much more than what most non-engineers make out of college.

Both APIs are available and will continue to be available for the foreseeable future.

-Todd

This isn't quite true - data is streamed from the client through a pipeline made up of all of the replicas, as it's written. It's true you'll lose data if you crash in the middle of a block, _unless_ you call the sync() function which makes sure the data has been fully replicated to all of the nodes.

The 1.0.0 release is actually formerly known as the 0.20.205.1 release -- ie just bugfixes since 0.20.205.

Hadoop's been "production ready" for years - there are hundreds of companies running it in business critical applications. But some people want to see "1.0" before they move to production :) So we recently decided to call it 1.0 so that the version numbering matches the maturity Hadoop has already achieved.

-Todd (Hadoop PMC)

Several comments here indicate that this practice is outdated. I disagree strongly. Even in 2011 autoneg is flaky on many switches - in a recent shipment of ~100 machines, ~5 of them autonegotiated to 100mbit instead of gigabit. And these are name-brand machines with name-brand switches.

Forcing duplex and speed with ethtool is still relevant.

I was an intern in Leo's group at Google 6 years ago and learned a lot from him. So, if you're looking for LA jobs or internships, I would highly recommend checking out his team! Of course, if you're based in the bay area, also check out my posting above :)

INTERN or FULLTIME (REMOTE for some positions)

Offices in downtown San Francisco, Palo Alto, and Research Triangle Park

Cloudera offers enterprises a powerful new data platform built on the popular Apache Hadoop open-source software package. We enhance the storage and processing technologies originally developed by the world’s biggest Web companies, allowing our growing list of global customers to use Hadoop to solve problems and achieve their particular business goals.

We're hiring across a number of departments. A few that might be interesting to the HN crowd:

- Distributed Systems Engineer, Platform team: get paid to work on open source projects like Hadoop, HBase, Hive, Pig, ZooKeeper, Avro, etc. The work is challenging and impacts hundreds or thousands of companies worldwide, plus you get to collaborate with folks at a number of other companies on a daily basis

- Distributed Systems Engineer, Enterprise team: work on monitoring and deployment software that scales to 1000+ nodes. This software needs to collect metrics, generate alerts, search logs, and manage deployment at very high scales.

- Senior UI Engineer: build rich web interfaces to simplify the user experience for operators trying to manage 1000-node clusters. JavaScript, CSS, HTML, and a bit of Java.

- Software Quality Engineer: help test, validate, and debug the products mentioned above. You will get to drive 100-1000 node test clusters to their breaking point and then dig into what broke.

- Customer Operations Engineer: help Cloudera customers solve daily issues they run into when using software in the Hadoop ecosystem. Learn these distributed systems inside and out. Be the Dr House of Hadoop.

Our company is currently around 135 people, but growing fast. Our investors are top notch: Accel, Greylock, Meritech, Ignition. Our most recent round was $40m about a month ago, so we are well funded and will compensate you with more than ramen noodles (though we do offer free lunch!).

The full list is available here: http://www.cloudera.com/company/careers/

Feel free to contact me at todd@the obvious domain.com if you have any questions. Otherwise apply above.

I turned down a job at Google (not 300k but hey, it's Google!) to join a startup. The startup started to sink about 2 years later.

I learned a ton and didn't regret it for an instant. Moved on to a new startup 2 years ago when it became clear the first was a dead end. Google recruiters continued to ping me religiously every 6 months regardless.

Moral of the story: Google, MSFT, Facebook, etc will all still be there in 2 years. Especially if you're early in your career and don't need the cash today, go wherever you will learn the most.

Yes, they currently depend on kernel patches to provide bulk map/unmap APIs as well as some other vm_area trickery to allow parts of physical memory to be mapped to multiple places in virtual memory with different protection flags.

(assuming I understood their talk correctly)

Azul's concurrent GC is much more clever. They use some tricks with memory mapping in order to perform really efficient read barriers (which traditional VMs have considered too costly). This allows compaction to happen while mutator threads proceed.

This is a nice presentation about it: http://www.research.ibm.com/mre05/Click.pdf

And of course the paper: http://usenix.org/events/vee05/full_papers/p46-click.pdf

Cloudera is hiring engineers in Palo Alto and San Francisco, plus a few customer facing travelling people (eg support/consulting/services) based around the world. Also various sales/PM/other roles.

http://www.cloudera.com/company/careers/

Getting paid to work on an open source platform (Hadoop) is pretty sweet, plus we have a fantastic team: http://www.cloudera.com/company/team/

Neat visualization, but I wouldn't say this is the kernel trick. The kernel trick is a trick because it maps the higher dimension dot products into a kernel function in the lower dimension. The neat thing is specifically that a kernel function on low dimension points can actually correspond to a dot product in an infinite-dimension space. In fact this is the case for RBF kernels like Gaussian, which is probably the most popular SVM kernel choice.

If you're paying $50-$100/mbit at the 60mbps level, you're paying list price (aka getting ripped off). You should be paying $40 at most, and if you are any good at negotiating and making vendors compete, you can get them down to $15 or so. They're paying in the <$5 range, so they're still making some margins at $15, just a matter of how much they want your business.

Without knowing that the + centers on the circle, is this actually possible?

The issue with intelligent pooling is that you (a) might end up with a lot of bookkeeping, or (b) people will forget to return things to the pool. It's way easier to do that kind of design in C++ where you can use scope and copy constructors to automatically refcount and then return back.

NullWritable is pretty useful sometimes, but what's wrong with Counter objects for counters?

3-7 on Friday isn't that convenient for those of us who work down south... why not a Sunday afternoon/evening thing instead? I'm often working there on sundays