Small language models are the future of agentic AI
arxiv.org
arxiv.org
I don't know whether Amazon relies on LLMs or SLMs for this and for similar interactions, but it makes tons of financial sense to use SLMs for narrowly scoped agents. In use cases like customer service, the intelligence behind LLMs is all wasted on the task the agents are trained for.
Wouldn't surprise me if down the road we start suggesting role-specific SLMs rather than general LLMs as both an ethics- and security-risk mitigation too.
We're going to be so messed up in a decade or so when only 10-20-30% of the population is employable in decent jobs.
People keep harping on about people moving on with their lives, but people don't. Many industrial heartlands in the developed world are wastelands compared to what they were: Walloonia in Belgium, Scotland in the UK, the Rust Belt in the US.
People don't really move on, they suffer, sometimes for generations.
The LLM, here, is the opposite; additional human labor to build the integrations, additional capital for chips, heavy cost of inference, an additional skeuomorphic UI (it self identifies as a chat/texting situation) and your wasted time. I would almost call it "make work".
The product maker cecotec, from whom I hope never to buy a product from, uses the following repairing process: you have to create an account to submit your data and then when you try to login into the caretaker page the system announces that the account you created does not exist (I have tried several times, several days, both with mine and my wife email and personal data). Furthermore there is no way someone should take the telephone on cecotec.
Another step that failed is to ask the Spanish postal service to give me some option to try to send again the product from its current location, stopped expecting sender order, to Amazon Storage avoiding the product come back step: they informed me that they can not respond to my email because the privacy law forbid it, perhaps this is because the product was send with my wife name and address. I don't suppose the air conditioned system to have some personal information attached to it.
Buddishm says that you can learn from any experience and maybe the karma of this product is to never be repaired and the fate has decided to condemn the old lone woman to suffer the hot wave, perhaps to expire some past life bad karma.
Fortunately all that comes goes by, and in this case I am happy to be able affording to buy a new machine that produces cold air, so, kind reader, be quiet there is no real problem. Furthermore, this experience can expand my empathy: it could be that for some people life don't work as it should, for them this anecdote is the normal course of actions where one problem calls another. For those a stream of problems is the only repl. To those I, most sincerely, wish peace of mind and hope their fortune reverse.
In this anecdote or episode human intelligence is not producing correct results and that could create a hope: that those small LLM models enhanced with intelligent agents could provide better support.
Today, in my current mental state I envision that the contrary could occur: those system could convert the bad things that happens once into the bad things that happens every day. So be careful with what you wish.
My hope it is that the greatest agents, ourselves, get together to solve whatever problem we have to cope with. But don't fool yourself that require real human deep intelligence and human hard work.
while life:
problem = read(bureaucracy, bad_luck, bad_ai)
response = eval(problem, resources = few)
print(response) # Spoiler '404 Solution not found'Processing returns is pretty standardized - identify the order and the item within it being returned, capture the reason for the return, check eligibility, capture whether they want a refund or a new item, and execute either. When you have a fully deterministic workflow with 5-6 steps, maybe 1 or 2 if-then branches, and then a single action at the end, I don't see the value of running an LLM in a loop, burning a crazy amount of tokens, and hoping it works at least 80% of the time when there are far simpler and cheaper ways of doing it that will work almost 100% of the time.
Extending the example you gave, nicely packaged, fully deterministic workflows work great in demos. Then customers start going off the paved path. They ask about returning 3 items all at once, or a whole order. They get confused and provide a shipping number instead of an order number. They switch language part of the way through the conversation because they get frustrated by all these follow-up questions.
All of these absolutely can be handled through traditional NLP, but require system designers to account for them, model the conversation, and design their system accordingly to react accordingly. And suddenly the 5-6 step deterministic workflows with a couple of if-branches… isn’t.
The LLM told me what sort of information they need, and what is the process, after which I followed through the whole thing.
After I went through the whole thing it reassured me everything is in order, and my request is being processed.
For two weeks, nothing happened, I emailed the (human) support staff, and they responded to me, that they can see no such request in their system, turns out the LLM hallucinated the entire customer flow and was just spewing BS at me.
We're supposed to think "oh it's an LLM, well, that's ok then"? A question we'll be asking more frequently as time goes on, I suspect.
Reason number one being that when the rep feels you are going to hold them accountable to the point of requesting such a number, you might not be the type of client to pull shenanigans with. Maybe they suspect me of being a cooperate QC agent? Either way, requesting such a number demonstrably reduces friction.
https://www.bbc.co.uk/travel/article/20240222-air-canada-cha...
We need "cumulative cases" that work like this: you submit your complaints to existing cumulative cases or open a new one, these are vetted by prosecutors.
They accumulate evidence over time and once it is a respectable sum, a court case is opened (paid by the corporation) everyone receives what they are owed if/when the case is won. If the court case loses, is appealed, and loses again, that cumulative case is banned.
Cumulative cases would have greater reprocussions to large corporate entities than "single person takes to court for several months to fight for a $40 discount".
And the people who complain rightfully eventually get a nice surprise in their bank accounts.
At most that could be done with pre LLM chatbots like classic Alexa - or the AWS equivalent Anazon Lex (which has Sentiment Analysis built in)
If anyone working on support bots reads this, you should realize it’s very easy for a bot to tarnish your companies reputation which can add up over time. You MUST have a way to tell if interactions are deeply frustrating to the humans. Since people are used to being patient with humans, we have a higher tolerance before we become frustrated with a support person. (And support peeps know this isn’t even that much true.) Not when we know we’re talking to a bot.
Seriously, fuck all the executives who bought into chatbots for support without bothering to make it a thoughtful experience. (Which in my experience so far, is all of them. Maybe Amazon is better, but they’ve had easy refunds with a simple form for years and years!)
Anyway really love the idea but many years of experience with decentralization / security / privacy projects makes me think it probably won't happen. Their description of how to incorporate SLMs at the end gives the game away: it's a description of a large, complex project that requires fairly good data science skills e.g. they just casually suggest you autoclean the data of PII then run unsupervised clustering over the results to prepare data for model fine tuning using QLoRA, and then set up an automated pipeline to do this continuously. Sure. We'll get right on that.
The history of computing is pretty simple: given a choice between spending more on hardware or more on developers, we always prefer more hardware. For NVIDIA this is a good thing modulo the fact that nobody buys their hardware directly because it's too overpowered. But that's the way they've chosen to segment the market. Given a choice between using a sledgehammer to crack a nut, or to make a custom hammer for nut cracking, we're gonna spend the money on outsourced LLMs every time.
Naturally, perhaps in future LLMs will create these SLM factories for us! If you assume software is mostly written by LLMs in future then past experience about expressed preference of software teams might not apply. But we're not there yet.
Specialized, fine-tuned models sit somewhere in between LLMs and traditional procedural code. The fine-tuning process takes time and is a risk if it goes wrong. In the meantime, the LLMs by major providers get smarter every day.
Sure enough, latency and cost are a thing. But unless you have a very specific task performed at a huge scale, you might be better off using an off-the-shelf LLM.
Are they though? Or are they just getting better at gaming benchmarks?
Subjectively, there has been modest progress in the past year, but I'm curious to hear other anecdotes from people that aren't firmly invested in the hype.
They are getting much much better.
That said, I also think the "Unix" approach to ML is right. We should see more splits, however currently all these tools rely on great language comprehension. Sure, we might be able to train a model on only English and delegate translation to another model, but that will certainly lose (much needed) color. So if all of these agents will need comprehensive language understanding anyway, to be able to communicate with each other, is SLM really better than MoE?
What I'd love to "distill" out of these models is domain knowledge that is stale anyway. It's great that I can ask Claude to implement a React component, but why does the model that can do taxes so-so also try to write a React component so-so? Perhaps what's needed is a search engine to find agents. Now we're into expensive market place subscription territory, but that's probably viable for companies. It'll create a larger us-them chasm, though and the winner takes it all.
A server needs energy to build it, house, power and maintain it. It is optimized for throughoutput and can be used 100% of the time. To use the server, additional energy is needed to send packets through the internet.
A local machine needs energy to build and power it. If it lives inside a person's phone or laptop, one could say housing and maintenance is free. It is optimized to have a nice form factor for personal use. It is used maybe 10% of the time or so. No energy for internet packages is needed when using the local machine.
My initial gut feeling is that the server will have way better energy efficiency when it comes to the amount of calculations it can do over its lifetime and how much energy it needs over its lifetime. But I would love to see the actual math.
The local machine is usually also highly constrained in computing power, energy (when battery driven) and thermals, I would expect the compute needed to be very different. The remote user will happily choose a large(r) model, while for the local use case a highly optimized (small) model will be chosen.
Once you've settled down on a few well-known paths of action, wouldn't you want to freeze those paths and make it 100% predictable, for the most part?
These sorts of “heuristics are surprising incapable while the semantic flexibility of language models are powerful” are surprisingly large. Even flexible validation mechanisms that take human entered semi fixed form inputs and reformat them to the expected input in a really reliable way and is much less frustrating to end users. Essentially any situation where abductive logic or natural language comes into the picture a small model does really well.
Both of those things - abductive logic and natural language - were largely unavailable as tools until recently. This pretty nearly rounds out the complete toolkit for making really robust and powerful (and usable) systems. You sacrifice a perceived determinism by admitting abductive logic and non determinism, but in my experience this warm blanket of the mathematically inclined wasn’t particularly robust in reality and systems often deterministically failed in complex and difficult if not impossible ways to avoid that a little bit of abductive reasoning could make it remarkably simpler and more robust.
It’s not 2023, there isn’t really a question of whether LLMs are going to stick around. They’re too useful to ever truly disappear, and in all likelihood their usage will grow if they can continue to eke out even minor incremental improvements.
This is not a minor oversight - it's arguably, in my experience, the most prohibitive technical barrier to this vision. Consider the actual context requirements of modern agentic systems:
- Claude 4 Sonnet's system prompt alone is reportedly roughly 25k tokens for the behavioral instructions and instructions for tool use
- A typical coding agent needs: system instructions, tool definitions, current file context, broader context of the project it's working in. Additionally, you might also want to pull in documentation for any frameworks or API specs.
- You're already at 5-10k tokens of "meta" content before any actual work begins
The "context rot" phenomenon is real: as the ratio of instructional/tool content to actual tasks content increases, models become increasingly confused, hallucinate non-existent tools or forget earlier context. If you have worked with these smaller models, you'll have experienced this firsthand - and big models like o3 or Claude 3.7/4 are not above that either.
Beyond context limitations, the paper's economic efficiency claims simply fall apart under system-level analysis. The authors present simplistic FLOP comparisons while ignoring critical inefficiencies:
- Retry tax: An LLM completing a complex task with 90% success rate might very well become 3 or 4 attempts at task completion for an SLM, each with full orchestration overhead
- Task decomposition overhead: Splitting a task that an LLM might be able to complete in one call into five SLM sub-tasks means 5x context setup, inter-task communication costs, and multiplicative error rates
- Infrastructure efficiency: Modern datacenters achieve PUE ratios near 1.1 with liquid cooling and >90% GPU utilization through batching. Consumer hardware? Gaming GPUS at 5-10% utilization, residential HVAC never designed for sustained compute, and 80-85% power conversion efficiency per device.
For a paper positioning itself as a comprehensive analysis of SLM viability in agentic systems, sidestepping both context limitations and true system economics while making sweeping efficiency claims feels intellectually dishonest. Though, perhaps I shouldn't be surprised that NVIDIA Research concludes that running language models on both server and consumer hardware represents the optimal path forward.
>Chester Hunt is the growth manager at Legitt AI, where he oversees all organic channels. With several years of experience in tech startups and scaleups focusing on B2B, Chester brings a wealth of knowledge and expertise to her role. Outside of work, you can find her socializing, traveling, engaging in extreme sports, and savoring local desserts.
That almost seems too bad, like it's a false flag or something.