The age of machine learning as code has arrived
huggingface.co
huggingface.co
You're generally tied to proprietary nvidia drivers that are a pain to install, and when they change version, it breaks software such as PyTorch, you're forced to update. PyTorch itself often has breaking changes. Then, most of the code is written in Python, often by academics who know and care very little about good software engineering practices. That also leads to fragile and breakable code. You'll often find yourself trying to install code that's just DOA because of multiple broken dependencies.
The quality of the code can be terrible, because academics are in a constant rush to publish the next thing, and tend to view code as throwaway. Worse yet, it's often written by students, and these students are taught by professors who view software engineering as an inferior, unscientific discipline. I've seen many github repos with no open source license, and no README, just a random code dump with no comments.
I'm personally not super enthusiastic by the idea of replacing a lot of code by machine learning just yet, because not only is the software ecosystem fragile, but the models can be as well. You're likely to introduce weird, unpredictable modes of failure. I can see that there's clearly a lot of people who dream of being able to basically create software without coding, and not have to deal with pesky software developers, but IMO, we're not there yet.
This guy talks about cloud-based machine learning solutions. You won't have to deal with drivers and general "IT" concerns there. The problem is that, besides being very expensive, these cloud-based solution will introduce extra layers of vendor lock-in. If you're training on the google cloud and using TPUs that you can't even physically buy/own, you might have a hard time porting your code to run without a TPU. If Google revokes your access for some reason, because they claim you've violated their TOS, you are fucked.
Last I saw the nvidia driver situation did not improve though. I wanted to come back to some old code, installed a new version of PyTorch. It complained that my GPU drivers were not recent enough, had to do this long tedious process that took over an hour to get newer drivers working.
Brownie points are awarded for persisting the setup of the image in the form of a dockerfile, shell script, etc., but such documents are to be considered non-authoritative sources, and must be expected to break at a rate roughly proportional to their size.
By the way as a seasoned (and actually kind of retired) programmer I started learning ML by myself after deciding that I don't like the code I was getting from hired companies. It is a strange land with many software writing practicies surprising someone coming from different programming culture. I wish I could find a community of people who made a similar path - here is my blog: https://zby.github.io/.
I marvel at the cool machine learning demos but I'm kind of sick working on this stuff, tbh.
As far as I know, AI is Software, so this type of reasoning really confuses me.
I think it is possible that "developers" might have some of their work find automation using AI and ML, but Computer Science will obviously stick around as long as computers are around.
For example, someone has to code the algorithms which are used to teach the computer how to learn. Someone has to code the pipelines for capturing data from the physical world into the digital world. Someone has to code the storage and exchange of that data to the ML models, and someone has to plug the inference into the production processes and systems. Someone has to put alarms and monitors around all these things to assert their operation. Someone has to code the UIs and interfaces that users will use to interact with all those systems. Finally, someone needs to code compatibility and optimizations of all these models and their implementation to continue to work and leverage new hardware capabilities.
If people think that developers were spending 90% of their time hand tuning business rules and coming up with manually implemented inference and decision models they are highly mistaken. That could have been 5% of the job in some circles, but 95% of the job has always been all these peripheral tasks.
If anything, there will be even more work now, since ML models require a lot more periphery to develop, train and ship.
But AI is not replacing processes that rely on software with something that no longer relies on software. What AI is doing is it lets you replace even more processes that were not yet able to be handled by a computer with one that a computer can do, and thus it just helps Software eat even more of the world. Decision making and judgement tasks were not able to be successfully moved to computers prior to the recent advancements in AI, now some of those can be.
Hum, back that up, because I don't think you could ever make a good living with that. No florists is going to pay you the same salaries that going to work for Shopify pays you, small local business just didn't have the means to hire someone to build a website, that's why there was a market for things like Drupal, WordPress and Shopify in the first place. So in reality, those frameworks created jobs, because now you can actually be paid by a local florist to setup and manage a WordPress for them, and the equation works out that for what they pay you and the time it takes you is now profitable. And you have all the jobs opened up from the company developing the frameworks as well.
The florist example was probably a poor choice since they tend to have pretty thin margins - but restaurants (while also not having a bunch of money) tend to have advertising budgets that can float a relatively modest cost for plopping out a website.
52k a year with no time off / benefits?
So I don't, know, maybe their data is wrong, but it seems to be an even better gig today.
I think you'd be very hard pressed to automate systems programming, and most forms of programming for that matter. So, as long as we develop new hardware, new products, new platforms, new programming languages, then there will be lots of programming work to be done.
The jobs that machine learning is replacing right now (if any?) are fairly repetitive jobs that don't really require any reasoning. Maybe machine learning is also helping automate phishing schemes, but, is there any programming job out there that's really repetitive and thoughtless, where the work is predictable and akin to factory work? Do you think a transformer model is going to be able to handle debugging?
Websites used to be made by hand. Then came frameworks and then came code-free e-commerce sites like Shopify. Github is testing the Copilot AI. How much longer before some architect tells an AI to stitch together code that fits some sort of functional spec. REST API's are already defined by their model and have code generated to match it. Is it that much a stretch to say that in the near future (15-25 years), AI might take over a significant part of software engineering?
> How much longer before some architect tells an AI to stitch together code that fits some sort of functional spec.
Would that person not be, essentially, a programmer? And what would they do if they needed to interface with external hardware that's not yet supported, or some network protocol or piece of software that's not built into AI-code-generator-3000?
> Is it that much a stretch to say that in the near future (15-25 years), AI might take over a significant part of software engineering?
In my opinion, people who probably think little of software developers, keep imagining that software engineering is this primitive, simple, reducible thing, and they are simply wrong.
Can there be a use for AI-generated code, and can that take over things like generating user-facing forms and website? Sure. However, what I think is likely to happen is that in 15-25 years, the world will be changing just as fast as it is now, and the software engineering discipline will just expand. Companies that hire "architects" who only know how to use AI-code-generator-3000 will be at a disadvantage compared to companies that hire more fully fledged software engineers.
Imagine you run a business and you only have architects who know how to use some code generator code that runs on the Google cloud. You pay some monthly bill to Google and hefty fees to use that tool. You're locked-in to that ecosystem. If you need a new feature or access to a new API, you may need to wait for Google to add support. In contrast, your competitors who hired pesky software engineers are paying higher salaries but save on Google cloud fees, and they don't have to wait for Google to fix problems to implement new features.
Sure, but...I'm not convinced there's any advantage over writing deterministic generators for things like that.
On one hand, you have something that produces a predictable output for a given input (modulo bugs in the generator, which can be corrected over time).
On the other hand, you have a black box that could misinterpret some non-obvious aspect of its input and produce completely unexpected output. Its training data will need to be carefully curated and maintained over time, similar to how a deterministic generator would need to be maintained, but, again, without the same degree of certainty as to what it's going to produce.
Honestly, the black box approach just seems like a nightmare to me.
Farming, auto, journalism, and marketing fits your model. But medicine, finance, and tech would suggest otherwise. I've been getting advice about how there is no money in software and I should study hardware since 90s.
Yes, maybe this is a freefall and we haven't made impact yet. But drawing parallel without further evidence seems to be too simple of a analysis.
What cause the jobs in Detroit to be outsourced? What skills was replaced? How easy was it to train those skills? What was the potential pool of workers? What was the demand of workers?
All I'm saying is that this question is a lot more nuanced. I'm a software engineer, so I'm biased. But I'm guessing that you might be biased in some ways too. The real answer, perhaps boring, is that we really don't know. Even if you're right - you could easily be off by 200 years.
Higher profit margins at the cost of lower quality and enabled with NAFTA. In the 90s until the mid 2000s, US cars were mostly top notch, now they're not that great.
You can read what the market thinks of the quality of a vehicle by its resale price relative to its new price. Used Honda and Toyotas are pretty close to a new price. Chryslers, not so much. This isn't exclusively a US problem though. You can get a used BMW or Mercedes for cheap, but you probably shouldn't if you want to keep it for a while and not spend a lot of money. Back in the 90s, you could expect to get 10 years out of a vehicle relatively hassle free, now it's about 3. YMMV.
It's how I got my job originally.
Not much of a stretch at all. But we've seen this sort of transition before: lower-level languages (microcode, assembly) are now largely generated by other software yet the amount of software development work needing to be done by humans has only expanded, and I don't expect there to be much more than a bump in the road in terms of demand for software developers this time around either.
Though I will say that the bump will be larger than the non-event that 4GLs turned out to be: https://en.wikipedia.org/wiki/Fourth-generation_programming_...
The most disruptive scenario I can imagine will result in a slew of businesses hit with absolutely massive cloud computing bills (due to naive generated code that works, but very inefficiently, or perhaps just code running for naively specified tasks that HAVE no efficient solution as stated), leading to a standard requirement of having an expensive human software developer sign off on it before deployment like architects signing off on blueprints (which might be the thin edge of the wedge for professional licensing, but that's a different conversation).
As a secondary effect, there might also be a proliferation of new (or newly popular) languages (for which insufficient public code exists for ML systems to learn from), as developers focus on domain+skill+technology combinations that haven't been automated YET.
You should register for the Copilot beta. It's pretty eye opening.
Basically it generates bog standard terrible code, because thats the majority of code. It can "Hello, world!" in most languages. Maybe it's useful for particularly boilerplate heavy languages or librairies, but for the most part it's more like StackOverflow copy-pasting except the code is worse.
So quite a long way to go.
Of course there is bound to be an end to this, a peak-software if you so will, where productivity increases will stop being offset by demand increases. I might live to see that day, but this AI wave is unlikely to be it. Yes, it brings some new tools that make programming easier, but it stimulates more than enough demand to offset that.
It's just libraries, really. Or libraries are 80% of the gain. So it's less about automation and tools and more about code reuse (sometimes it's even just binary reuse - you surely do that everyday by calling dynamic libraries).
The progress you note in the last 20-30 years is just Internet that became widely available. First at work and Unis, and then at home. We switched from sending disks by mail to downloading stuff from FTP servers. Recently WWW Git front-ends gave it another serious boost. That's what happened.
> But demand turns out to be incredibly elastic. A 50% increase in productivity just means that there's 400% more work, because so many new things have now become cost effective to try.
I would credit Moore's law for that instead. Really, if one can do e.g. affordable, real-time machine vision stuff today, it has more to do with CPUs and GPUs gaining power every year, than software getting better.
How can you say that seriously when, for the vast majority of people, a computer vision project consists of `import cv2`?
Not really. AlexNet was a watershed moment for computer science, and the kind of things that were possible by 2015 with AI that weren't possible in 2010 far outstrip the gains in hardware advancement during that time.
Yes I've heard it for 25 years. It's mostly low hanging fruit and it hasn't been done with auto-coding per-se, but more data drive architecture where a layman can do a basic website. Anything custom or sophisticated requires programming knowledge though, and since they are proprietary systems, it's a lot more expensive.
I think there might be a bit of a misconception of what existing software engineers did. Very rarely were they focused on designing inference models or performing data analysis. Most of the time the business side of things, like a business analysts, or people working on the business problem would perform analysis manually, and come up with business rules and logic. Software engineers could help them leverage more powerful techniques by building them tools that help for data analysis, like BI platforms, or data stores that can do bigger and more complex queries faster. Or they could help them with letting them know of techniques, generally known as old-school AI, such as edit distance algorithms, graph algorithms, logic rule engines, expert systems, etc.
So nothing has really changed, except that there is even more demand from businesses who want software engineers to help them setup tools and pipelines for them to experiment with ML and have data scientists analyze their data.
What it would take to get rid of the demand for software engineers is to be able to automate their work, and ML is far from able to do that. It could one day, but we are talking about a computer that can reprogram itself successfully, and that can also understand the requests of the user in how it should reprogram itself. And we're talking not just reprogramming, but it should also be able to replicate itself to other machines and all that, since most use case today involve distributed system, as no one computer has the compute needed for the scale we operate at. Things like GPT-3 show some promise, but as it stands today, you'd need a software engineer to inspect and review everything it generates, which would take almost as much time if not more.
I think what is far more likely is that the offer increases, that is, that there eventually is just a lot more people with the ability to perform the work of software engineers. This will probably happen as a combination of more people joining the field, and the barrier to entry becoming smaller. The latter could be helped a little bit by ML, if it can deliver better auto-complete for example, but I think it largely is just driven by frameworks, PAAS, SAAS and IAAS, library, and all other type of code reuse we've been building for years now.
I don't really see how. Software development is about translating arbitrary requirements into a rigorous form without ambiguity (ie the developer has to fill in the blanks). Unless we invent AGI we would need the 'clients' of software development to be able to unambiguously list their requirements (without there being any blanks that need to be filled in). Effectively, they'd have to become devs themselves. It might not look like the software development of today of writing text, but they'd still be doing a similar type of job.
Also, automation really changes the nature of a product, and the quality is often poor.
I think we begin to see software construction design oriented towards "agents". Analogous to previous abstractions like "daemons" or "running programs". They have permission to "self-assemble" intelligently.
If you mean people who's work involves developing software by typing instructions into a computer via a keyboard interface... and this is what entirely defines a software engineer. Then possibly.
If you mean people who's work involves developing software by instructing computers in various other ways (e.g. via talking, feeding it pictures, videos, audio, or other data)... and this is what defines a soft-... er I mean data scientist. Then I'd argue data scientists are just a newer software engineer.
If you mean engineers in the true and general sense? Never. Devising new technology & solving problems are skills forever employable.
There was a good quote either from Joel or someone like that (or someone he was interviewing on a podcast), along the lines of "I went into software because I like tinkering with intricate clockwork; not because I like training a puppy to not pee on the carpet". On top of that, you tend to not even spend most of the time on the puppy, most of the time is spent collecting and organizing carpet samples.
Of course, we have not figure out how to reliably writing dumb code yet, but we are going to replace that with ML. Now everyone involved can truly shed the responsibility of program failures. After all, no one understand ML anyway...
I surmise "being applied" means outcome of an ML model is used to augment the decision taken by a human. I doubt they are used to handle everything automatically for critical infra, since ML models are neither 100% accurate and fail miserably when they fail. But the insinuation is that the field has already been won by AI.