The cold start problem: how to build your machine learning portfolio
towardsdatascience.com
towardsdatascience.com
There's a major issue when this is applied to a certain class of problems in ML and Data Science that people tend to ignore.
If you could get a job as a civil or mechanical engineer (building bridges or whatnot) by showing that you built a small bridge in your backyard... We'd have some unstable bridges.
If hospitals just let residents run the hospital... We'd have a lot of mistreated illnesses.
If you could show a realty company that you can build a recommendation engine and they hire you to build their advertisement algorithm... Suddenly you're breaking housing discrimination laws.
I am all for folks being able to get jobs from their cool projects. But we need ethical standards and educational standards before folks are given large, real-world problems to work with.
We need to take a page out of engineering and medical playbooks and build official education or apprenticeship requirements. We need to have licenses that can be revoked if someone fails to follow ethical or quality standards.
So - love creative people getting jobs. Now give them a high-quality education program along with those jobs.
Accreditation isn't just for enforcing a minimum standard of education - it's primarily so that people can actually be held responsible for bad decisions, and so that they have leverage over their bosses (it's harder to find an actual engineer or a doctor willing to do unethical things for you, when they could lose their license over it).
There is also higher need for programmers ("developers") than for actual engineers.
this is part of the "if you're not unhappy with the first version of your product then you've launched too late" philosophy and it does work in non critical sectors
Of course people learn with experience, and luckily redoing things in software development is cheap compared with when building houses. But that’s the only reason we get away with it.
My point is, we would build better and cheaper systems if we from the start acknowledge that we have to take into account completely different sets of considerations when we move up the scale. A shed isn’t just a big box, a house isn’t just a big shed, etc.
Here in software, we've turned that into a positive thing and made a philosophy out of it. How do you know the toilet shouldn't be in the kitchen? Maybe the users like it? You know what, the data actually shows that in houses where the toilet is next to the kitchen stove, people spend (on average) more time in the living room, thus raising the core metric of happiness.
Building ML projects at home is nothing like building a box to building a shed; it's more like building a shed to building a house.
Sure, building a house is more complicated, but you won't be building that house alone, and if someone hired you to build a house because you were good at building sheds, I'm sure you'll start your job as a junior house builder, not master architect.
You're running the team now. Your colleagues have only ever built boxes. Go get em', master architect!
This is my (hopefully humourous but actually taken from my experience) way of saying that companies will often do what is most immediately profitable rather than what's best in the long run (for humanity or themselves).
(a) Bright young man who built a wooden box
(b) Bright young man who says he has read for years about building houses
I'd pick (a) over (b) and put him on a team where they build houses so he actually learns on the job.
That personal information is then sold to even more organizations that need no scientific or statistical basis, again with no ethical oversight, to make decisions that impact your future and the future of those around you.
Picking up a compiler will never be banned anymore than it is banned to pick up a scalpel. But to call yourself a doctor you'd accept the responsibility (ethical and professional) that it entails. To call yourself a professional engineer you accept the responsibility that it entails. You hold yourself to a certain level of quality and ethics or you give up your right to hold the title.
Without a standard level of responsibility there is no way to build any more of a foundation than we have. We will always have the rickety mess of data leaks, corruption, and general lack of accountability that we have now.
We will have employers that can pressure programmers to go directly against their morals and ethics to get what they want.
I'm not proposing that the poor be excluded from the field. I'm proposing that you not only give them a job but give them a pathway to a quality, independent education that will continue to prepare them for the major challenges they will be expected to face and the ethical questions that they'll have to make a stand on.
No, it doesn't bother me at all.
What does bother me is large companies such as Google have a business model that is based on them gathering as much information as possible about everyone, and then using that information to manipulate them, typically into buying things.
Credentialism will not solve that problem.
> We will have employers that can pressure programmers to go directly against their morals and ethics to get what they want.
Credentialism in programming will not solve that problem either. What will solve it is:
- a reduction in income and wealth inequality, and a basic income, so that employees have more bargaining power with respect to employers
- laws against big companies behaving badly
We see the same with the iron grip that the medical council has on the number of medicine students (imagine if we fixed the numbers of software engineering students and made it difficult to register foreign qualifications).
I agree that educational standards are important - but if that isn't coupled with improving access to higher education (including post-graduate education) then it's basically just saying only the rich should be allowed in.
I feel like the conversation should be
- "We need ethical and professional standards because our current way of doing things isn't working out too well!"
- "But that will exclude people who don't have access to education!"
- "Then we should give everyone a means to get the education. Because it is an unlimited resource that can be freely given without being taken from someone else and enriches all of humanity!"
- "Yeah! Let's do both those things!"
At least that's how I had hoped the conversation would lead.
But we can't throw our hands in the air and say "Accessible education is impossible and so we can never have standards of ethics or quality". That is definitely a dead end for society.
Anywho - I absolutely agree!
Link: http://aosabook.org/en/index.html
A lot of Computer Science graduates, and/or self-taunt engineers are missing the commentary of 'masters' or senior people who can say why and why not this or that way.
Just like the difference between building a bridge across a river and in your backyard is far more than just scaling up, seeing design patterns and common pitfalls writ large is one more step towards proper education needed for a modern programmer IMO.
It is always better to go into the interview, meeting, pitch with projects that are shipped or working to demonstrate.
For instance, if you want to get into the game industry, build games.
Same with any field, when you create/build you can rise above education, experience and more competitive metrics.
Created/functional projects, and especially shipped products, are the best qualifications.
Experience is highly valuable but concrete examples of that which are done individually or completely self propelled, that is the way to be. Creating/shipping/projects to show does demonstrate experience and skills.
There can be people that have experience that can't ship, but if you created/projects/shipped and have less experience it truly demonstrates that you do have experience enough to be worthwhile and improve the place you are going to.
I'd argue even when I have had to interview or been on the hiring side, showing projects beats experience and education, it can also make interviews focused on your projects and what you bring rather than just whiteboard and talking about what you have done.
For instance when I went into gaming I had multiple games made for every place I was aiming for, i.e. a racing game at a racing game company, promotional games at advertising/promotional jobs, and networking/multiplayer shooters/online for more hardcore gaming on console and mobile.
Ultimately projects are the only way to get clients as a freelancer/small business pitching to clients, so that also works when going for full time employment and jobs.
Ending the list with a couple of research/unsolved problems is perfect.
To me, this is the perfect way to learn, almost like feel your way blind in a cave and reaching its extents.
If you have no shipped titles, the second best way is at least game components, networking, game mechanics etc that you can demo.
Personally I only like to work with people that can take a project to ship and are product focused. Gamedevs that don't know the depth and troubles of shipping a title will have to learn that on the company otherwise and usually don't know when to cut/scale correctly the gameplay/mechanic/product etc. A big problem with AAA development and broken systems is the heavy specialization, it is why games come out bugged as there is very little ownership or people that can ship a game solely and understand how all parts connect.
Unless you are only working on triple-A titles, most game companies are small/medium and need shippers that aren't as specialized, especially mobile. Though most people that go into AAA games think they will get to make their game, but in actuality they work on such small parts that they eventually break out into their own smaller studios or indies to build games.
Everyone at a game company wants to build their games, it is why people go into games. For success you have to have shippers who understand the whole shipping adventure that want to come together to get games out the door that each can have input on as a team, but also have the itch scratched that they already have shipped some of their own games and an outlet for creativity if specialization is too heavy on a project. The best game studios and games are developed by smaller teams and key individuals that understand this, especially during pre-production into early production/prototyping/game mechanics, during production and post-production that is where you want more specialization but even then you want people that have shipped.
I agree, I coupled shipping with maintaining as I see that as one. Just getting it out in the world isn't enough, updating it, not breaking it, smooth updates for users with little friction, not breaking profiles, no crashes, upgrade/update testing, library/version updates etc are all massively important.
Definitely better to have someone that has done some production on a live state of the game over someone that shipped and forgot, but both are better than no shipped title or production.
1. Apparently modeling is a solved problem. No need for any knowledge of math/stats folks, just use the latest python packages.
2. Data collection is more important than actually knowing how a model works and even more importantly doesn't work. What happens when he is hired and can't put together a reliable model?
I think his approach is solid (using projects to learn and being ambitious), but it feels like trying to run before you can walk.
Closing those gaps in not just accuracy, but also generalization is the data science portion of the task (and requires much more knowledge than what is demonstrated by this blog post - although they could of left out a lot of detail). They make it seem like if he just had a little bit more time this would of been straightforward. But I am not sure about that.
I am all for giving junior practitioners a chance. But this is like hiring an english major for aerospace engineering because they built a model airplane in my opinion. But maybe I vastly underestimate the amount of extremely low hanging fruit out there for ML projects.
It was the narrative the article was pushing about how knowledge of the field doesn't matter, just strong work ethic and selling yourself visually to potential employers.
However, there's some good advice: - get real data
- clean it, play with it, build models
- focus on the cleaning, as that's what the gig normally is (and damn right too, your models will be way better if you've taken the time to understand your data).
I did find the and then he got a job part annoying, but the post was much better than I expected (relative to other towardsdatascience posts).
When I tried to learn how to golf a lot of time was spent on proper form and club choice, not just "swing!" I swung as much without a ball in front of me as with one.
the club is just the component you need to launch the ball towards the hole using your skill. you don't need or want to know what the club is made of until you think of buying a more expensive club(which might be never)
For me it feels more like "software plumber".
Though I admit - I love "software plumber" and want that as a job title some day! Sounds very cyber-punk
This means that someone can easily claim "this problem is difficult to learn for machines" when they fail or claim "we got X% accuracy look how great we are!" when they do okay. But a really good engineer or scientist would have succeeded in the same task, or have gotten X+10% accuracy with the right models, data, or engineering.
It's important to have stories to tell during an interview, but getting the interview is the hard part.
Prospective employers not only do actually pull up your github profile, many of them will find you because of it.
As an aside, writing this makes me realize how successful Github has been at implementing their motto of "social coding" when none of us had any idea wtf that could actually mean or look like.
So of those 2,832 openings that companies are trying to fill, how many employers are out actively recruiting candidates based on their github profiles?
It seems to me that Github / social coding is an interesting mix of networking (the social kind) and coding (which demonstrates abilities), but like most networking, it's only one of many ways into the door, and it's unclear how many of the 2,832 positions will be filled via that particular flavor of networking. My guess is that it's the minority.
Let's say that the funnel of applicants for one position starts with 250 resumes. How do you proceed? Among the applicants with github profiles, are all of the github profiles looked at?
I did make money in the end however, but only because I was buying bitcoins to gamble with in the 2017 season.
If you show a willingness to scrape resources together to work on difficult problems, that's scrappier than the majority of your competition for the job. Startups love that.
Of course actual technical skill matters, but in my experience, the willingness to reach for aspirational goals is rarer than baseline competence, so I'll happily interview someone who shows that specific trait.
Reading these posts I get the impression people prefer portfolio projects over studying the mathematical fundamentals; "dazzle hiring managers with these three easy tricks!" Seems like a great strategy to getting hired, but one wonders how long people using these lifehacks persist in the role.
I think someone without experience training a model to high-accuracy might struggle at a startup where they're the first hire in the department, but at a larger company, presumably there are senior folks to help newcomers along. I think having motivation is a much better signal, and you need to be motivated to develop a side-project of significant magnitude.
It didn't appear as though these two didn't train their mathematical fundamentals though. I believe the gist here is to find an interesting project tailored to the field you want to work in, and make it visual for the non-technical interviewers. Once you do get their attention, you would naturally need to pass a real DS interview, which would test your mathematical fundamentals.
However if it’s in unrelated technology (or no longer trendy), then I’d argue they should be removed from the profile altogether. These might raise unnecessary questions and trigger biases.
How is this infringement? Why did he oblige? Unless he got paid by them, he should not have obliged with their request.
Lockheed Martin CEO: So what the fuck is this youtube video?
I think it is absolutely immoral (and dumb from an IP risk point of view) for companies to try to get interviewees to solve real problems for free.
This is different though: this is about potential candidates finding creative ways to demonstrate their skills.
The companies didn't even ask people to do this: these people chose to take on projects that would demonstrate their skills in a new space in which which they did no yet have commercial experience. I think that's commendable and a very smart strategy.
A very competent developer who is interested in doing some ML work may have just recently read this article and gotten to work on an ML project because he feels that not doing so will hurt his chances of getting a job.
As a note, I have no idea what the right solution to this problem is, it is good to confirm candidate's knowledge (I prefer creative ways similar to the parent) and part of that knowledge certainly can include open source work, but going down this road too much leads to people feeling obligated to make open source work to put on their resume and even to people faking open source work to try and land a job.
...which makes the world a better place. Why do you object to people writing open source code?
i do have one concern if it ever became an industry prereq, it becomes a filter for those who have more free time to work on side projects.
Doing a unique project is much better for learning cost/benefits of implementing AI/ML, although this post may be overoptimistic on how that can lead to a job offer.
It seems like you would need a very specific level of knowledge to romanticize data science in that way. Most people know too little (So you're basically trying to build skynet?), and most of the rest are either in the industry or know someone that is, and so have a more realistic view.
I don't know, I did maths in undergrad, maybe some of the more clueless CS majors thought this way.
Max is correct to point out the irony in his anti-thoughtpiece thoughpiece as he falls into the same trap of vagueness as those other articles. Specifically, he rails against general “black box” approaches to modeling, then takes a general “black box” approach to the work of operationalizing a model (much harder than building the prototype to begin with!).
The discussion of “pulling data” does not match the practical reality, since pulling via BI tools is not scalable and rarely automatable. SQL may cover this insofar as you dump data from SQL to...what, though? A Python session on your laptop? Automating this process allows a data scientist to scale their impact.
For more specificity on engineering practices required for data science, I recommend Robert Chang’s series of posts: https://link.medium.com/CG7c7mQdyS
For details on how a data scientist can impact an organization, I recommend this from the FirstMark blog by Jeremy Stanley and Daniel Tunkelang: https://firstround.com/review/doing-data-science-right-your-...