Considering that LLM can hallucinate in different ways unpredictably, not sure whether it will work in practice.
Also commonly referred to as the robustness principle; “be conservative in what you do, be liberal in what you accept from others“. An approach that is now considered an anti-pattern, for such future compatibility reasons.
The AI of tomorrow will be trained on the output of AI today.
(Somewhere in my memory, I hear an ex-boss saying "Well that's good - isn't it?")
There might be other ways of achieving the same goal. Also,LLM hallucination is changing/improving quite rapidly.
That said, "Designed for LLM" is probably a very productive pursuit. Puts you in the right place to understand the problems of the day.
In my opinion, using code from LLMs, which might see your program come to life a bit quicker, will only enhance the time needed for debugging and testing, as there might be bugs and problems in there ranging from trivial things (unused variables) to very subtle and hard to find logic issues which require a deeper knowledge of the libraries and frameworks cobbled together by an LLM.
Additionally, it takes out a lot of the knowledge of these things in the long run, so people will find it more and more challenging to properly do this testing and debugging phase.
Let's say you've got some rest API. Are they trying to send PATCH requests but you only support POST? Do they keep trying to add pagination or look for a cursor? Do they expect /things to return everything and /things/id to return a single one but you have /things and /thing/1 ?
None of this is to say you must do whatever they are trying to do but it's a very cheap way of seeing what introducing your API/library to a new user might be like. And, moreover, you can only ever introduce someone once but you can go again and again with a LLM. Review the things they're doing and see if actually you've deviated from broad principles and if a change/new endpoint actually would make things much easier.
Fundamentally you can break this down to "if there's someone who can do some coding but isn't a top class programmer, how much do I need to explain it to them before they can solve a particular problem?"
I would recommend API fuzzers here, which cycle through all available possibilities, and generate 'garbage' inputs as well as structured inputs regarding the specifications the APIs implement (http, and the underlying structures provided to the api endpoints).
an LLM would likely not perform an exhaustive test.
There are several projects freely available to run such fuzz tests. Eventhough that does eat up considerable resources, and 'resetting' state during such tests can be problematic at best, the same would apply for LLM based testing. (database gets screwed in some way, and the next request is not on a 'clean' state.)
Originally, it wasn't. From the GP post:
> If there is a common hallucination API call, just add that API. This will make the API more "friendly" and resemble common similar APIs.
If it's just users, you wouldn't be adding the APIs.
I write API calls. I am a user of apis. If you want to be more pedantic than this I simply do not care and you should evaluate what benefit you are bringing to this conversation.
- sorry if the comment was too silly. i'll try to keep it for myself.
It's not testing the API for bugs, it's asking whether the API you have follows typical norms and isn't missing things. Is your API confusing or incomplete?
Imagine you have a library dealing with chat rooms, and LLMs keep trying to call "room.getUser(id)" and "room.getUsers()" when your API is actually "room.user(filter:{id:id})[0]" and "room.user(filter:None)". Maybe that's a sign that your API, while totally functional is going to be confusing to other programmers. Maybe you don't have anything that lists all current users, or there's a flag or option somewhere that would make sense or be more typical.
A fuzzer won't tell you that.
For the original post, you could imagine that a user might want the price of a token in a different currency, but maybe it currently returns everything in dollars. An LLM might try and add ?currency=GBP . Maybe an LLM keeps expecting to see in the response data the last updated timestamp to check data freshness, but it isn't something you add right now.
it seems like trying to facilitate bad things. make good things instead.
If it breaks a lot of conventions or norms, no it's probably not good.
But also the point here is *how much do you need to really explain?". If you have apis that require a lot of re-explanation to llms to force them to do the right thing "no getUsers doesn't exist use users(filter) where... YOU MUST USE users(filter)!" That's a sign your API might be going counter to a broad estimation of existing code.
> it seems like trying to facilitate bad things. make good things instead.
I have seen so many people make something that is obvious to them and so few others that "simply write good things from the start" is pointless advice.
That being said, its not always the case and you are right, 'make something good' is a bad advice, essentially it's not advice at all.
I would say as advice - don't do something alone if you want others to use it. Try to get feedback as early as possible from people you trust, and from people you'd expect to be your consumers/users. And always listen to that advice, don't throw it into the wind. I think that's the best advice if you want to make something 'successful', because the label of successful ultimately will be granted to your product by your users, not yourself.
Sorry to put a 'bad advice' if it was taken as advice :-). I don't mean to make bad things even in advice ^^.