drew

Yeah, in my org we have a dedicated data engineering team that owns the pipeline and production data systems plus an analytics team that owns reporting and making data available to non-technical data consumers. That leaves complex research work for data scientists who are (in theory, anyway) building on top of stable data infrastructure and good data sources and free from ad hoc reporting work.

The problem is DS is really 2-3 different disciplines under one nebulous title. What you're describing is folks who are prototyping and productionizing models. That's definitely in short supply, but random STEM PhDs are in no way competitive for those roles unless they're coming from CS programs + have work experience in production engineering.

But that's by no means all of the DS field. There are lots of DS jobs where you're collecting and interpreting and communicating about complex data sets. An engineering mindset is occasionally helpful, but a bias towards building versus towards analyzing and writing can just as often be counter-productive. Not all problems are solved by systems; lots of problems are solved by better understanding the problem and then letting other specialists build the right solution.

The bootcamps have contributed to the problem by focusing so much on building things. The idea that you can go from an econ undergrad to being a self-sufficient member of a production ML team in 6-12 weeks is nuts. What's less nuts (and what I wish programs like Insight focused on) is taking people from having data skills in one domain and with one set of tools (e.g. logitudinal medical record data, stored in CSVs and handled in Stata) to another set of tools (billions of rows of event-based product data stored in a data warehouse, processed in R or Python). But instead the bootcamps behave like the missing skillset is the ability to make a predictive random forest model on some arbitrary data set and build an AWS web app around it. THAT job market definitely doesn't exist and is completely over-saturated.

But people who are smart communicators about data, can manipulate and make sense of massive data sets, can ask incisive questions about their data, and can use data to convince people of a complex argument are always going to have job opportunities, even if they're not production grade engineers. If that sounds like you, I'm hiring - hit me up on Twitter: @drewwww.

We track clicks on each of the four login methods, and compare it with successful sign in events with each of them. So we know completion rates for each type, plus which types are preferred by users. Nothing fancy, just google analytics events + checking the users table.

You can see how we communicate it on our site if you're curious. Basically it's a modal dialog with four options:

* Facebook * Google * Twitter * Email

We don't use the persona messaging, and I think people's expectation when they click the 'email' button is that it's going to just be a normal email flow. We don't call it Persona or Browser ID or use any of their assets or messaging, because we didn't think anyone would click on it if we did.

But yes, we see a small preference for a button labeled 'email' versus facebook, and a medium preference for either over twitter.

The paradox here is that people are more familiar with the appearance of home-grown-style login systems and are more willing to follow through on those than the novel Persona flow, even though the security characteristics of Persona are stronger. It's a chicken and egg problem, and until someone really big takes the plunge and gets everyone comfortable with this style, anyone implementing it is going to be somewhat of a cost to signups.

The gmail case is special, actually. For a few domains (gmail, yahoo, not sure which others) it will fall back to a flow that's more like OAuth. But for unknown domains it sends you an email with a link, and then requires you to create a new account (with a new password) that is persona-specific.

In principle, Persona is great. Not storing passwords is awesome, a non-FB/Google/Twitter identity option is important.

I would encourage you, though, to look carefully at your login completion metrics. I implemented Persona on my site (http://www.sixquestions.co) to have a pure email option and although users clearly prefer it, about 35% complete the Persona login flow successfully. That's 10 points lower than our next-worst performer (Twitter), and half the rate of our best performer (Facebook). For all the concerns people have with authorizing Facebook/Twitter access, that is (in my view) offset by the alien-ness of Persona's login flow. We've heard from lots of users that logging in with Persona is unusual and they thought they were doing something wrong because they'd never seen anything like that.

So, as much as I believe in Persona, I'm about to deploy a change that removes it entirely. It adds a lot of surface area to our testing and future development, but if it means we lose fewer users in their signup flow, it will be worth it.

Very cool! Excited to see what comes next.

Is there anything people looking forward to these features should know to plan? Like, I was about to start doing some load balancing stuff soon. Is that a waste of my time and I should just work on the core logic and figure load balancing is just going to be made easier in the near future and I shouldn't waste my time with it? Or should I roll my own option for now and then just figure at some point I should be able to transition to something that performs better in the future?

I'd love to know too, but I haven't had much luck getting profiling to work super well with node. I think I mostly just don't know what I'm doing and someone who did could answer this pretty quickly, but I haven't been particularly successful and figuring out exactly how that cpu time is being spent.

This is obviously not super precise, but the CPU pegs (load average 1.0, 100% usage) once we get above ~9k messages/second. I don't know how much overhead there is on socket in terms of bandwidth, but some back of the envelope math makes it seem like we're well below network bandwidth on a 100mbit interface. My instinct is that (per rauchg's comment above) that if those kinds of improvements can be made, then we're not yet butting up against a bandwidth issue yet.

As for comparing to other systems, I think we're certainly at the same order of magnitude, but some rough tests a friend of mine ran made it seem like a java server/client pair could send about 4 times as many packets per second. But that's a somewhat unfair comparison - socket.io offers a lot more than simply raw packet transmission and that additional abstraction is going to come at a performance cost. But I think being on the same order of magnitude as a really low level approach like that is a good place to be at this level of maturity of the technology.

Unfortunately, this isn't just a question of interpretation. Legally, there's no differentiation between those two categories. One is more ethically problematic than the other for sure, but you're just as screwed having either on your computer. You can see this most clearly in sexting cases against high school students. You can still get on the sex offender registry for sending a picture of yourself to your boyfriend or girlfriend and there's tons of stuff like that on /b/.

(disclosure: one of the authors)

We're definitely not interested in business success in any sense. In some ways, anti-success is more interesting to us as researchers because academics tend to study communities that are successful, popular, and self-sustaining (like Wikipedia, Twitter, Facebook, etc). That introduces a bunch of funny biases in our understanding of how identity and persistence work in online communities: we've only really studied sites that have pseudonyms or real names and have searchable archives.

To the extent that we do care about "success", it is useful to be able to say "4chan has TONS OF POSTS" because it's a way of saying that the site does, in some sense, work for quite a few people. If it was just a quirky design but wasn't tremendously high traffic (I think the comparison we make in the paper is that /b/ alone has 16 times more daily posts than the 8 biggest news groups on Usenet), it would be a lot harder to get something like this published. People would be more inclined to brush it off and say "well, it's the internet - you can get a few thousand people to use anything." But that kind of traffic is hard to deny. Plus 4chan's role in the web's collective unconscious makes for a great story, too. So bottom line, what researchers choose to study is very strategic, but the metrics that matter definitely aren't about money, but there are lots of other factors.

The thing is, cp includes self shots of people under 18. There is TONS of that on /b/, of both genders. It's awfully hard to tell 19 from 17 when you're taking a topless picture of yourself in the mirror.