My First Year as a Freelance AI Engineer
masatohagiwara.net
masatohagiwara.net
Note that I wrote this post back in February 2020 and there are a couple of things that I would like to add:
- Due to high demand, I increased my rate to $250/hour + some fixed monthly fee in April. I didn't drop any single client :)
- I'm seeing very little impact from the coronavirus. I have a client base spanning between Japan and the US in a little-impacted industry (education). Don't put all your eggs in one basket.
- I would strongly encourage everyone who's considering making a leap to read "The Win Without Pitching Manifesto" https://www.winwithoutpitching.com/the-manifesto/
- Due to a sheer volume of my incoming emails I can't answer all of them, but do let me know if you are interested in working with me!
I did well in Kaggle's highest-prized competition[1], so I wondered if I should explore this kind of consulting myself.
The competition I was in was effectively limited to US residents though, but as a "remote" freelancer you compete world-wide, potentially with people willing to work for far less. Are there reasons your clients prefer US-based freelancers enough to justify the gap in pay?
[1] Top 1% (5th/518), #1 result in California: https://www.dhs.gov/science-and-technology/news/2018/07/09/n...
As you work between Japan and the US, I'm curious about the rate difference. Are you able to charge the same?
How often will the clients find you again to do the follow-up work, e.g. new software feature requests, for the previous projects? If you refuse the requests, will this undermine the business relationship?
"NLP/ML for Asian language processing" covers anything that has text in CJK/other languages. I work with Japanese and lots of things taken for granted in NLP pipelines require entirely different approaches specifically for Japanese. For example, there's no spaces, so word tokenization is actually a complicated issue.
People are so hung up on how much the guy gets per hour, as a business owner all I'm thinking about are the total costs. There are lots of cheaper developers who would take longer to get the job done- how have I saved money by hiring them? And that's before getting into quality of work. I think many commenters are imagining OP as a contractor who works 40 hours a week for his clients, 9 to 5. The whole point of a consultant is that they're project-based, it's not an ongoing expense. I would certainly hire OP for a 20-50 hour project, say, and then an hour here or there to fix bugs and answer questions as needed. It's not a huge total cost, I don't care what the guy gets paid per hour. I literally just look at it like 'overall, this ML/AI project cost me $10k, and delivered x amount of value'.
If HN commenters think $250 an hour is a lot, wait till they hire accountants or- god forbid- attorneys :) I bet OP is much smarter and delivers more value than my attorney that charges $400 an hour for simple contract review, and breaks out specific line items for answering my e-mails and 10 minute phone calls with me....
And these consultants are definitely not Ph.D holders, working on state-of-the-art technology. Just your average java enterprise programmers that get flown in to do slight upgrades.
They might not make $250/hr in salary, but sometimes the stock options will dwarf that number.
I can see how being freelance can be nice though. It keeps you on your feet, you get to see the problem from lots of different perspectives, and it tends to concentrate on the work.
Similarly, a company's CEO could say "Hey, I could simply go to university myself and learn everything about NLP and solve my NLP problem and save 10.000$ for this contractor! Yay!"
That's only scalable up to a certain point, and that point is usually reached when you realize that you need employees.
As an aside, this is one of several reasons I think charging an hourly rate is bad for business: You immediately get compared to other freelancers based on this arbitrary number rather than your abilities.
Given all that, I feel like they are still just selling their skills for short term cash while others make the real money off what they produce. As always with contracting.
If a typical ML engineer makes $250K/year, she costs the company about $500K/year; that means the hourly rate should be at least $250/hr.
It's only a rule of thumb, it doesn't need to be exact, and the more you earn the less accurate it gets.
The less important part of the rule of thumb is for businesses to determine if it makes sense to have someone as a consultant or a full-time employee. But whoever is doing hiring should be able to figure that out for themselves because they should know how much the employee actually costs - but hey, even they might need help sometimes.
And that is the cheapest kind of firm with minimal overhead. For larger firms it is probably an even bigger multiplier.
For example in my country if I lose a job and I have a consultancy that I have made money on in the last year or two (I forget) you have to close it down, and then wait some months, and go through some bureaucratic thing to get them to say ok he really shut down the company before you can get the normal government payment for someone unemployed.
Obviously as starting a company is also not without its work and costs you don't want to do that, which means you need to have the money to last a few months between gigs.
Where does that £40k go? For a start we don't require private health insurance in the UK (we pay out of our taxes). Some of it is estates upkeep (offices, labs, capital equipment, cleaning etc), internal grants, support infrastructure (IT, computing), interview/acquisition cost, legal/HR costs, training, insurance, bonuses, employee benefits/pension matching and employer tax burden. Most universities put the rest into an "overheads" pot which can be used for a variety of things.
Most grants have a separate section for conference travel, publishing expenses, etc. In a smaller company, all of these things need to be added on (unless you're grant funded). So you're also considering expenses on top. If you pay someone to headhunt a candidate, that can be thousands plus a percentage of salary. We have access to a lot of training courses, most of which bill the university thousands per head.
The labour and legal costs alone of hiring candidates can also be very high. Think of a company like Google where your interviewers may well be on six figure salaries. In the US a typical hire costs about $4k. If your company depends on bums on seats to make money, the time between someone leaving and someone starting is costing you a fortune.
Depending where the company makes money, your overheads may also be paying for other staff. For example the university has an HR department, finance, legal, cleaning, estates, etc. None of those people generate any money for the university directly, though they are necessary for it to function. Some of their salaries are paid for by IP royalties, tuition fees and government subsidies, but a big chunk is from research grants.
So basically ML when done right is enormously valuable because it horizontally extends the reach of computation into previously humanlike domains.
But the caveat is that it's hard to do ML right. You need very much to be a generalist, an autodidact (to quickly learn the nature of the domain you're operating in) and I think a background in math and/or physics, or at least some other quantitative field is indispensable. You can do a hell of a lot more with neural networks than sort cat pictures or target ads.
Is this $666 per year? I think you’re missing a comma somewhere.
;-)
(1) a 50 hour contract at $500/hr = $25K
(2) 1 month of paying a full-time engineer = $25K (i.e. $300K/yr including payroll taxes and benefits)
If a freelancer comes in with specialized knowledge for solving a particular problem, it can be quite easy for them to add more value in 50 hours than an average engineer would in a month, and therefore it's worth it.
The company is not that special, I don't even know if we make it to Fortune 500. The point being, technology is a force multiplier, and when applied at the right lever for companies above certain size, the returns can be enormous.
The former category buys you a little bit of flexibility in hiring, etc., so you should be willing to pay a little more than you would fully loaded (e.g. 1.25.x - 2x salary, depending on a bunch of factors) with no complaint, but that's about it. The great thing about capacity freelancers is you can turn them on and off easily as your needs change.
That capability category is a vastly different space though. Imagine I'm a specialized skill/experience sort of consultant or freelancer. If you hire me to show you how to do something you don't know how to do, how much is that worth? Or what if bringing me on for 3 months can pull your time-to-market ahead by a year? Maybe you would love to hire me full time but can't afford it (or I just don't want full time with you) - or maybe you just don't have enough of this sort of work to justify an expensive hire.
It all comes down to value. If 50k worth of my time will save you 3 million dollars, why do you care how much time that is?
Sometimes a 2 hour conversation can save you from a 300k mistake. Should I be charging you for the time, or the impact?
Just for example, I run a team of 10 machine learning engineers at a large ecommerce company. We mostly do NLP and computer vision, some time series forecasting.
I cannot imagine ever paying anything close to $25k for consulting advice, that’s just bananas to me. We recently purchased licenses to use the data annotation tool prodigy from the spaCy creators at explosion.ai. That was ~$4000 and the decision whether to build our own data annotation system or not was excruciating, involved all kinds of business documentation, RFCs, approvals, NDA processes, etc. It was deeply non-trivial to procure that, and building our own was a very serious option we pursued with tech specs and prototypes and everything.
Spending 6x that amount for _advice_ about NLP, which practically grows on trees today, is just totally unrealistic.
It makes me suspect the real target customer for you is not companies with actual ML engineering teams or ambitious data-driven projects, but more like someone looking for McKinsey-lite. Some place that has no serious ML use case beyond drop-in pretrained models and sees $25k as the cheaper path to rubber stamp certification that dissolves internal political feuds. Most likely just selling super cookie cutter NLP models as if they are advanced and represent some sexy leap forward for a company with a couple junior data scientists. Algolia or just some drop-in Elastic tfidf search is more than enough for these companies. Spend the $25k on an intern who can tell you anything you need to know about neural network frameworks.
In reality, the 4-5 ML engineers you already hired are very likely more knowledgeable than the freelance consultant you might hire. They can tell you much more about state of the art and simultaneously know the specific integration path in your company’s web service and data ecosystem. Those folks won’t be wasting time prototyping - they would be pursuing a more efficient way to get the answers you need than advice from a freelancer, even if that freelancer was Bengio for pete’s sake.
I just cannot see the value prop here except for the usual story of paying for consulting as a virtue signal / credential / politics kind of thing.
And again, this wasn't a "freelance consultant" in the sense of the original story posted to HN, but an accomplished person at a very respected company who was able to secure the approvals from their side to help us out. This was novel/niche work for which "drop in pretrained models" don't really exist or apply.
Frankly to judge and belittle someone else who is just sharing their experience by saying they're not doing serious work and don't have serious customers is very rude.
The fact that you went through all of that for a 4k line item means there's something wrong at your end. You would have burned through multiples of that putting together a prototype, only to them spend lots of hours on the procurement process for my team would have just put on a credit card (exaggerating but only somewhat).
Sounds like you are not really accounting for the opportunity cost of your peoples time.
No, we had the prototype we needed for making the decision with 1 team member working on it part-time for 4 weeks.
Your economics are way off. The cost of the prototype pales in comparison to risks around vendor lock-in, security issues, license growth, etc. The legal team vetting the contract and NDAs is the real expense - and well worth it to be quite careful about vendor software.
The cheap quick prototype helped us realize the longterm cost of maintaining that tool was too large, and the upfront procurement costs were worth it.
In a “move fast and break things” shitshow where you just instantly buy the vendor software, the risk of getting burned on a bad / unsafe contract is huge, and you end up playing hot potato with the 3 licenses you bought because you didn’t appropriately plan for license growth, dealing with data breaches, etc.
I think you’re naively reacting to the perception of a bureaucracy unable to do anything, but it’s totally not. This type of vetting is very cost effective.
It’s also why nobody is just quickly dropping $25k on consulting from freelancers.
The parent already explained why he was willing to pay $25k for a consultant, but you ignored his explanation and just started talking right past him and insulting the work he's doing.
" It makes so much more sense to pay $25k to get direct knowledge of systems and techniques versus your team spending a bunch of time exploring different products/methods/algos to find something that might work in the end, or might fail in a few months. $25k is like 5 MacBooks, hardly worth thinking about versus being able to get experienced direction from someone who has done what you're trying to do and saving your team literally hundreds of hours of exploratory work."
>Spending 6x that amount for _advice_ about NLP, which practically grows on trees today, is just totally unrealistic.
If you can't conceive of what useful advice might look like then that's just a failure of imagination on your part. Getting any ML model to work properly involves a lot of esoteric domain-specific tricks. You have to use A,B,C to model the problem domain, apply X to transform the data, clean the data by throwing out Y, impose constraints Z on the model. You can pay your full-time employees hundreds of thousands of dollars to spend hundreds of hours discovering these through trial and error, or you can just pay someone $25k to explain these to you up front.
Of course you can find a billion worthless, generic, baby's first bag-of-words spam filter type NLP tutorials online, but that's not the same as commercially applicable expertise.
Absolutely grinds my gears too because the consultants turn in dreck that we're expected to fix to help the VP who hired them save face.
Not sure why I'm sharing except that the last line of your comment really hit home for me.
If you're working on a project that's driving $1 billion / year in revenue even a 0.01% increase is $100k/year. It's pretty easy to justify $25k on a contractor if they can demonstrate even a very modest impact on your project.
Chances are that 95% of people is what is bringing in 95% of the value of your company.
Sure, but there are plenty of business problems worth solving even if it takes millions. You're not paying for the person, you're paying for someone to solve your problem.
I don't know who they found to take the task, but I have to assume they charged them quite a lot because you're just not going to find very many people to solve that particular problem.
Certainly not 10000 per hour, but a significant amount given the Danish market.
My friend (annoyed) asks what is he going to do. Locksmith pulls out a bent coathanger. My friend gets angry. Says he could do it himself, he has a coathanger upstairs, he doesn't need the locksmith, £150 is a total rip off, calls him every name under the sun.
Locksmith: "Okay mate, you have a coathanger now?". Friend says no, agrees to pay him. Door opened. £150 well spent.
I think too many people here perceive tech as something that meets their needs when it should be about providing a service to others.
A MacBook repair shop may replace a $0.50 transistor, but charge $200+. What you're paying for is the knowledge and experience in identifying and solving the problem. It's unfortunate that so many consumers don't understand this principle.
I agree that 20000 for one hour is pretty unlikely but 10,000 a day for 2 days seems like it might happen.
P.S. It was an inflated amount to get an immediate fix based on his daily rate knowing it'd only take an hour or two - not because sysadmins working on ~30 person networks reguarly earn ~£3k a day.
"Five thousand dollars," the artist replied.
"But, what?" the woman sputtered. "How could you want so much money for this picture? It only took you a second to draw it!"
To which Picasso responded, "Madame, it took me my entire life."
And no...that just isn't correct. It does vary by industry. Very generally, capital-intensive businesses have smaller productivity differences...but even then it is probably far less than you think (in some of those industries, experience curves are per employee). But, given the subject of the post, then yes...productivity can vary that much.
What is perhaps confusing you, and many companies get this wrong, is that they treat every employee like a superstar when (by definition) they can't be. It is far more profitable to hire five below-average people and teach them to be average than hire one superstar.
My point wasn't really about what companies should or should not do but how they should look at those decisions. Wages really don't matter. I will pay someone $1m tomorrow if they can generate $5m in sales. Simple.
But companies should think carefully before they go down this route. It mostly isn't worth it because most companies aren't very good at hiring and will lose the employee if they can't continuously provide a high productivity environment. Some places can retain staff but, again, there is an 95/5 rule about workplaces just as there is employees.
If 20 hours of someones time saves you 2 million though, it's a no brainer to pay them 20k to do it.
Some people absolutely do specialize in this sort of engagement, and do well at it. It's really important to remember that. $N/hour usually doesn't mean $N*2000/year salary. Often hourly rates go up because of the nature of the work means the ratio of billable hours to hours put in isn't great (see, e.g. many independent lawyers).
Also worth noting that most people contracting at this level don't in fact charge hourly but that's a different aspect.
Only if you discount equity, small companies are very unequal because the owners/founders will make orders of magnitude more in the event of a sale.
A more reputable site (that's known outside of the HN-bubble) would be the stackoverflow salary calculator. If I fill in "Data Scientists with a master's degree and 5 years of experience in the Amsterdam area" I get a median income of EUR 51K. In silicon valley it's higher of course, but the same profile in San Francisco still 'only' gets you a median income of USD 157K.
No... but just checking it out, and every single post is about FAANG/uber from people who work in silicon valley. You didn't exactly disprove my point.
As for why. If a large company makes money from advertising and you improve their $10billion/year click model by 0.01% CTR then you've just made them a million dollars. Do that few times throughout a year and a million dollar payout seems low.
Yea, no. Not outside of the HN bubble. I run a data science company myself (in Europe). Rates here are really not higher than 80~100 euros per hour. That's for someone with a master's degree in data science and years of experience. On top of that, we have to very actively look for clients and advertise a lot. It's not like people are en-masse knocking on our doors to throw their money at us. So I claim BS on 500/1000 per hour. That just absolutely does not happen except for maybe the 1% who is a celebrity in ML world.
Switch to an s-Corp.
30% of the gross will be paid as a salary, as an employee of the s-Corp. you must pay social security, etc on that.
The other 70% can be taken as a dividend distribution As the owner of the s-corp. Thats not subject to social security, etc.
Will save you ~15% in taxes.
at your level, it’s Well worth the cost of the cpa to handle the payroll, quarterly, etc.
I don’t think devs realize that a 1099 costs you money.
LLC is a business entity type. S Corp is a way to be taxed.
With LLC, life is simpler (formation, filing taxes, maintenance). With S Corp, you can distribute the profit as dividend (at a lower tax rate) after paying yourself a reasonable salary.
Edit: I just noticed that you'd posted the original comment in this thread. So, LLC electing to be taxed as an S Corp means simpler filing requirements (as opposed to C Corp and the like), coupled with the ability to distribute dividends.
Narrowing your niche down attracts specific types of clients who have specific needs that few people in the world can solve.
My expertise is NLP/ML for Asian language processing and language education. When defining your specialty, I think it helps if you define it in terms of the industry, not in terms of an ML stack. People look for, e.g., “AI solutions for healthcare” and “text analytics for finance,” not for “GANs” or “Seq2Seq models.” You need to be willing to learn a very wide range of ML techniques and models, from simple regression to GANs and RL, no matter what industry you work in.
In other words, define your niche in terms that make it easy for someone with budget authority to say "yes" to hiring you.
This is worth noting also because if you accidentally cross over into talking with recruiters or other gate-keepers to the real budget authority holders, you'll be misled into talking or padding your resume with technical terms that are not the key value drivers for why you'd get a nice-paying contract.
HOWEVER, I also know management consultants who also do work for these corporations and they seem to get the data no strings attached. Apparently all the security/privacy policy is ignored when someone who reports to the CEO says 'Give BCG all the data they want'.
Although I have my doubts maybe this is true specifically for AI area. In general contracting, getting familiar with the large project and "submitting the first PR within a couple of days" is a wet dream. I will exclude cases when one is hired to find and fix bug in some simple, short piece of code.
Because I can't remember a single project where I wasn't expected to contribute something immediately, especially with new clients (that need to be convinced of the value).
This kind of thinking/pricing has been inculcated by so much free-market idealism, and of course on some level human labor can be thought of as a commodity, but that's neo-evil.
If you work in demanding field, and have a client that needs—and can afford—your work, giving them them a discount only takes away from your stack. What possible benefit is there? Repeat business? They will repeat if they need it, not b/c it's on sale.
Assume you are literally putting money in their pocket (ROI) and get paid what you're worth.
If anything more engineers need to be paid even more. Heck even consider a real estate agent that puts your house up for sale, and your house is worth 1m and they get 2.5%, let’s say they walk away with 15k how many hours do you think they spent marketing and managing the sale? In a hot market, where prices are higher and things move fast, even they are reeling in 500$/hr so a PhD level engineer pulling that shouldn’t shock you.
If they aren't 100% set in their ways, I do make them aware that things will move at about half the speed with TF, so they'll effectively be paying twice as much. If they are set in their ways, I do not mention it, since I'm not going to change their mind anyway.
That said PyTorch 1.5.0 is just broken pretty much - tensor permute (which in computer vision you end up doing for every input tensor) is 10x slower than it used to be. There's an issue in GitHub already.
I'm beginning to worry about PyTorch.
I'm working with TF and Pytorch as well, but so far for the later I have found the project to be reasonably reliable (though I did find Chainer considerably more polished). Can you share more about what worries you with Pytorch?
The fact that such obvious, severe bugs make it through the release process likely means that there isn't really much of a release process. And what's in place doesn't even test the release on totally bread-and-butter models like resnet50.
For reference, one of the core devs added more details based on where we are with our investigation: https://github.com/pytorch/pytorch/issues/37142#issuecomment...
I will personally quit my job and sign a contract with you that clearly defines what will be delivered by what date, if you are willing to pay flat rate of $X.
You will not pay until after you accept delivery of the system.