Comment by selalipop | Hacker News Reader

Comment by selalipop | Hacker News Reader

Parent Full thread

What you're describing as pre-programming is a little misleading if the "pre-programming" doesn't need to change for each specific request: a real product would provide that "pre-programming" for the user.

Prompting for task performance is fine as long as you're not expecting the end user to have to replicate your prompting. Your goal is to change model activations for a given input, the end user is similarly affected regardless of if you used a prompt or fine-tuned.

-

This task doesn't require fine-tuning though, zero-shot performance is enough:

I generated a mock schema from Home Assistant's API (https://data.home-assistant.io/docs/states/) and explicitly gave the model the option to ask for clarification, but it has no problem translating non-obvious commands into actions without asking for details:

https://chat.openai.com/share/fc5b972f-4641-47a1-9842-2e0d69...

Note those objects mirror Home Automation, you could hook that up today without any song and dance. Combine that with RAG and you'd have something that's a lot more useful than Siri and capable of improving performance over time.

We're back to "how is it different from Siri".

You had to provide two pages of text and do manual mapping between human-readable names and some weird identifiers to provide the simplest functionality.

Funnily, this functionality is also completely unpredictable.

I ran your prompt and first request, and got "Identify the area with the lowest observed request volume and increase the brightness of the light in that area to improve the lighting." ChatGPT then proceeded to increase brightness in the garage.

---

It's also funny how in the discussion about context the context of the app is forgotten.

Indeed, the context is "people using natural language to make requests". No soul on earth would consider/use your phrasing. I (a human) have clue what your request is for - "lowest observed request volume"...??? Try "raisr the lights where we usually aren't asking you for much" and you might get tge same result. As far as I can tell, brightness increase in the garage (where, I'd guess, you've made the least requests), the AI apparenyly understood better than you or I what you meant.

That JSON isn't something you'd type, it's something that you can programmatically generate if you have a Home Assistant setup.

With super primitive wake word detection and transcription, the most you get is:

- What the user said

- How loudly each microphone in the house heard it.

If you take a look at the mock object in that transcript, that's what it maps to...

```json { "request": "I'm finding it hard to read" "observedRequestVolume": [ 3eQEg: 30, iA0TN: 60, h1T3y: 59, 5Qg1M: 10 ] } ```

The only part that would be human provided is: "I'm finding it hard to read"

The invented challenge was to see if using a suboptimal set of inputs (we didn't tell it where we are) it can figure out how to action.

It's zero-shot capability that makes LLMs suitable for assistants: traditional assistants can barely handle being told to do something they're capable of in the wrong word order, while this can go from hastily invented representation of a house and ambiguous commands to rational actions with no prior training on that specific task

I think you fundamentally don't understand the topic if you're talking about two pages of text?

The end user would never type in a word of that: they'd say "[Wake word] play me some music"

A piece of software running on a device would transcribe what it heard, and fire off a request to the LLM with all of that text wrapped around their statement.

For ease of sharing I used the web interface to provide the instruction, but you'd use the API with a prompt which also dramatically increases determinism.

No one is writing out the state of each light bulb: you trivially query that information programmatically and bundle it with the request.

—

In a real product there'd be explicit handling of detecting where the request came from, that's already a problem that's been worked on, but I wanted to demonstrate the main difference vs Siri: zero-shot learning

The LLM wasn't told what those volumes mean, but it was flexible enough to infer the intent was to provide a form of location, rather than ask.

It's a forced example so if you want to get caught up on the practicality of audio for locating people be my guest, but it's to show LLMs are great at "lateral applications" of capability:

You give them a few discrete blocks of functionality and limited information, and unlike Siri they can come up with novel arrangements of those blocks to complete a task they haven't yet seen.

—

Honestly the fact you keep going back to "look at all the text" feels a bit like if I showed you the source code for an email messaging app, and you told me: "No one will ever use email! Who would write all that instead of just writing a letter and mailing it?!"

Reply on news.ycombinator.com