LisaG

Kinect is more powerful than video footage because video can show what someone does, but the Kinect can analyze what you do and get a deeper understanding.

San Francisco CA (remote okay)

Crawl Engineer and Big Data Enthusiast

We're looking for someone enthusiastic about open source, net neutrality, open data and keeping the web truly open. Common Crawl is dedicated to building and maintaining an open repository of web crawl data in order to enable a new wave of innovation, education and research. If you're looking to do work that matters, come join us!

We're set to do amazing things this year, and there is no better place to hone your big data skills than helping us manage and process our 50 TB corpus. Plus, you'll be working within a passionate community and have the chance to interface with plenty of talented researchers, educators, startup folks, and an incredible advisory board.

Responsibilities

  * Improve the stability, scaling, and visibility of our distributed web crawler
  * Use, improve, and extend our post-crawl, Hadoop-based data processing pipeline 
  * Design and build mechanism for specification and execution of custom crawls

Desired Skills & Experience

  * You can architect and code for a system with tens of billions of documents
  * Strong coding ability in Java 
  * Strong coding skills in at least one scripting language (Python, Ruby, Perl...)
  * In-depth knowledge of HTTP and are familiar with web crawlers
  * You have development and administrative experience with Hadoop and HDFS
  * Ops experience with Linux or other UNIX
  * Some familiarity with AWS, including one or more of EC2, S3, EBS, and EMR
  * Like to build useful, thorough documentation of code and systems
  * Self-starter wiling to take ownership of projects

http://www.commoncrawl.org

Love this sentence: Think about it, every event likes press coverage….and I was a quote on quote…“member of the press”!

Cheesy formatting, total misunderstanding of "quote unquote" - the use of quotation marks afterwards is priceless, and an attitude of having discovered and revealed some great secret which is in actuality a ploy so tired it has been in bad movies for decades. This guy is passing, right. Definitely a dancing monkey.

Is your reasoning because "The Government will not take down Google and most normal searches would be intact."? I am not sure I would put too much faith in the thinking that Google will get special dispensation from the government because it is so useful to so many people. Google is vulnerable to SOPA. In fact they are vulnerable on more than one front.

The kinds of people that go to Google.com are also the kinds of people who don't fully understand the repercussions of a law like SOPA. Google not working would drive home the message to them. They could learn about SOPA from an informational page like the one Wikipedia is going to put up during the blackout.

Google really should go dark, even if it is only for a few hours. Policymakers need to understand that if the internet gets broken, it will have repercussions well outside of silicon valley.

" good luck getting meat and milk on a consistent/reliable basis "

Maybe in 1950. Or on Niʻihau. But on the bigger islands it is no problem. There is a giant Safeway in the Manoa neighborhood of Oahu that is just as good as the one on Townsend in SF and there are many Foodland grocery stores. I lived there as a broke postdoc and had no trouble.

San Francisco, CA

Are you passionate about Open Source, Net Neutrality and Open Data? Do you want to do work that matters? Join us at Common Crawl and you will work with big data and help ensure the web is truly open.

Common Crawl is non-profit that produces and maintains an open repository of web crawl data to give everyone access and we need your help! We're looking for a truly talented data-minded Hadoop/Java engineer to help improve our crawler, API and do data analytics!

Contact lisa@commoncrawl.org for more info

New Common Crawl blog post addressing many of the questions raised here last week. http://www.commoncrawl.org/answers-to-recent-community-quest...

I work at Common Crawl.Thanks for all the interest and the good questions! Lisa

That is true at the graduate level. If you pursue a PhD in life science, physical science or computer science your tuition is waived and you (usually) get a stipend. If you want a PhD in liberal arts you have to pay tuition.

San Francisco, CA / Los Angeles, CA

Are you passionate about Open Source, Net Neutrality and Open Data? Do you want to do work that matters?

Join us at Common Crawl and you will work with big data and help ensure the web is truly open.

Common Crawl is non-profit that produces and maintains an open repository of web crawl data to give everyone access and we need your help! We're looking for a truly talented data-minded Hadoop/Java engineer to help improve our crawler, API and do data analytics!

Contact lisa@commoncrawl.org for more info

I thought it was about electrons too :) But I think post about elections was fantastic.

There are some great judges for this event http://datainsightsf.com/jury/

Agree that the 15% Uber discount makes up for it. That and the AT&T discount are about the only discounts I notice, what other FC discounts do people use?

We have a very interesting open source project that is actively looking for new contributors. It is an interactive art project. It started as a collaboration between Shawn Fanning (tech side) and Fernando Mastrangelo (art side). You can see Fernando talk about it here http://vidsf.com/2010/sep/27/fernando-mastrangelo/

We are having a brainstorming/work session at Shawn's (in San Francisco) on Thursday and we would love to have some more coders! Let me know if you are interested in attending and I will give you the details.