Using ChatGPT for home automation | Hacker News Reader

Using ChatGPT for home automation

atomic14.com

261 pointsiamflimflam1108 comments

I have been using chat GPT to control my light colours for about a month now. It's too tedious to properly set the colours and temperatures of our lights manually and too complex to consider all factors like activity, weather, music, time of day and season.

Chat GPT is now our personal lighting DJ, giving us dynamic and interesting light combinations that respect our circadian rhythm.

Here's my prompt - the output of which feeds Home Assistant:

Set the hue for my home's lights using the HSL/HSB scale from 0-360 by providing a primary and complementary colour which considers the current situation. The HSL color spectrum ranges from 0 (red), 120 (green), to 240 (blue) and back to 360 (red). Lower values (0-60) represent warmer colors, while higher values (180-240) represent cooler colors. Middle values (60-180) are neutral.

Consider these factors in setting the primary hue (in order of importance):

1. Preferences throughout the day: - When about to wake: Reds, oranges or hot pinks - Approaching bedtime: Hot pinks or reds - During worktime: Blues, greens or yellows - Other times: Greens, yellows or oranges

2. Current activity: Bedtime

3. Sleep schedule: Bedtime 23:00, Wake-up time 07:00

4. Date & time: Sunday May 21, 05:40

5. Current primary hue: 10

6. Current complementary hue: 190

7. Weather: 13°C, wind speed 9 km/h, autumn

Respond in this format and provide a reason in <250 characters:

{"primary_hue": PRIMARY_HUE, "complementary_hue: COMPLEMENTARY_HUE, "reason": REASON }

The output looks like this:

{primary_hue: 10 complementary_hue: 190 reason: "Approaching bedtime and early hours of morning, so a warm and calming hue is needed. Complementary hue adjusted slightly to 195 to maintain balance."}

That's really cool. Are you using the API to run this locally?

Yes, I access the API through Node Red which fills out the prompt template and returns Chat GPT's output to Home Assistant.

Costs about $3/year in API quota.

> ...provide a reason in <250 characters

For what it's worth, I've been using something similar with my prompts and felt the completions did a poor job of honoring this, but do a better job when asked to use words instead of characters.

Imagine asking a person to give a verbal response in 250 characters or less. They could do it, but it would be a lot of work. Even saying less than 50 words is hard.

If you actually have a hard cap, you’ll have to give feedback. If it’s just you don’t want an essay, it works great to say something like “a few sentences”. And as always, examples help a ton.

> For what it's worth, I've been using something similar with my prompts and felt the completions did a poor job of honoring this, but do a better job when asked to use words instead of characters.

Yes, restricting by characters is hard for GPT-style LLMs because they work in tokens, not characters.

Thanks, that's a good pickup -I'll refine my prompt.

But LLM have little concept of tokens don't they? Or at least well not know what their tokenizes is like.

It can understand word boundaries, though. A space is its own token and there are special tokens for common words prefixed with a space or common word prefixes with a space in front, ex “ a”

GPT-4 does a much better job at paying attention to details in prompts

For stuff like this I like to make it write out a lot of "reasoning" before the final output that I'll parse.

Like so:

Write three thoughts on how the primary and complementary hue should change and what value they should change to, along with your reasoning.

Pick one, summarize the reason for it in less than 50 words.

Then write FINAL CHOICE: followed by output that looks like this {"primary_hue": PRIMARY_HUE, "complementary_hue": COMPLEMENTARY_HUE, "reason": REASON"}

A video of these conditions and the resulting colors would be nice.

I wonder if the limited amount of variables mean one could just ask ChatGPT to generate a one-time lookup table of colors and store them locally. But it's interesting to see that an LLM can be a "color designer".

I like the idea of storing the colours locally - better for privacy too! Definitely a novelty factor that AI is "designing" my apartment's colours.

Here's a snapshot of how it works in my office - bedroom and living room have similar setups and draw from the same colours:

https://ibb.co/ysxY3Lf https://ibb.co/9VgYVTb

And details on the setup in Node Red/Home Assistant: https://news.ycombinator.com/item?id=36018291

Prompt:

I realize now, instead of a lookup table, ChatGPT could probably generate a piece of code that considers those inputs (so e.g. temperature, time of day/weekday, cloud cover); and output colors.

Brilliant idea. I hadn't thought of that, but it would save me some API quota and give me a little more control over the outcome.

I might struggle matching colours to the music playing a little, but there are so many other factors that it probably doesn't matter.

FWIW, I just tried the same prompt with open assistant

> {"primary_hue": 345, "complementary_hue": 55, "reason": "For bedtime, warm orange tones at 345 create relaxation while paired with cool green at 55 helps prepare your body for sleep."}

You can absolutely replace this with something self hosted, this was using the `oasst-sft-6-llama-30b` model on https://open-assistant.io

Thanks! This is a great find... I'll have a look. Fingers crossed my hardware can run this.

This is awesome. How do you feed this into home assistant?

Thanks - The flow is pretty simple:

1. Every hour a Node Red flow runs 2. It generates a prompt using a Home Assistant Jinja template 3. The prompt is sent to Open AI 4. The response gets parsed from JSON and sent to Home Assistant's "input_number" entities 5. Lights in Home assistant pick up the state change and set the new colours

If you're keen, I can share the Node Red flow diagram (or JSON).

I don't get it. Do you not tell it the current primary and complementary hues? Seems like it gives you the same back. And in the reason it incorrectly says it adjusted the hue.

It hallucinates a bit, yes. But in general it gives me suitable colours for when I need them. I've yet to try this with GPT-4 or some of the other models suggested in the comments here.

This is super cool, I would love to implement something like this. Are you willing to post the node red yaml for it?

Certainly - just posted it in this comment here: https://news.ycombinator.com/item?id=36018291

Happy to answer any questions if you hit a snag.

Could you elaborate more on the setup? Like if I were a technologically competent person, but unfamiliar with how to set up a system that keep GPT live, and then feeds that output into Home Assistant

Certainly, I've just shared this in another comment here:

https://news.ycombinator.com/item?id=36018291

The flow is pretty simple in Node Red - render the prompt from Home Assistant and pass it to Chat GPT. The response from Chat GPT is parsed and sent back to Home Assistant in some "Helper" variables known as "input_number" and "input_text".

Once the values are in Home Assistant, it's pretty easy to change the colours of lights in an automation.

Does ChatGPT really work better in this case than a bunch of if statements and a random number generator?

No, we have come full circle with added layers of complexity.

Haha, probably not. The logic for my activities and the complexity for the complimentary colours made it a little tough to maintain and extend. Plus there's a nice warming feeling knowing an LLM is busy designing my lighting throughout the day.

That's fantastic. I might have to hook up my lights.

I also keep complaining that music recommendations fail because they don't take into account enough factors like this - would be great to control music choices this way too

I'm not against idea of having LLM be smart assistant for home, but I do have problem with sending any of that to cloud.

It's one thing to use it as convenient way to change anything remotely but linking home automation not only into constant internet connectivity but also "that one particular cloud thing that might disappear at any moment" (hello google) just seems like setting yourself for problems later. At least in this case he's still left with HASS if cloudy part goes away.

But I wish there was more push into hybrid model - like have your router or small ARM box be server that runs queue and some, for lack of better word, "lambda-like" code that handles most of the programmed events (lights, heating etc.), say via WASM and some APIs around it, then just have HASS-like software (both cloud and local) deploy control rules/code onto it.

Big nice UI for control goes down ? Doesn't matter, the controller is just running simple code, not rest of the visualisation and controls. Want to replace it ? Configuration as code makes that a breeze, hell, have cloud backup of whole setup. Don't want controller ? It's essentially just a queue and some code runners, that cloud providers can host for you. Want pretty graphs in cloud or some aggregation ? Just make your local node filter and send relevant MQ events there.

"Internet of Things will not happen until LAN of Things arrives", to paraphrase someone.

This already happens in the industry, like heavy industry.

At home, it's held back by absence of an adequate server. Often a router box or a NAS would be fine to host something like OwnCloud. But to run ML middle, different and much more expensive hardware is required, which would sit idle 99% of time.

LAN of things - this is a fascinating idea. Can you share where you are paraphrasing this from or link to more info on this concept? Sounds familiar with the self hosted applications people now run on their NAS/routers these days.

The phrase is used quite a bit in hackernews comments, particularly on any IoT or home automation threads.

The concept is nothing new, it's just not something manufacturers picked on. Hobbyist are doing it all the time, usually in form of most devices either communicating directly or being bridged into queue. So technically via that every device can call every other device directly so for simple things you don't even need programming

The idea is just to keep that within your own LAN.

IoT world mostly wants to create closed gardens so you buy other products around that garden.

Easiest way to make it easy for consumer is to connect your devices to cloud and so most decide to go that way, no need for end user to buy a hub.

And even if you need to (say devices use zigbee and you need a bridge), putting that in cloud means user's phone can always access it, regardless of whether they are connected to the home's wifi, without much fuss (as home internet will most likely be behind NAT/firewall so it can be hard to access directly

It's getting cheaper: https://coral.ai/products/

$20 for the SMD TPU isn't bad, but it's definitely at the top end of the BOM for custom PCB projects.

Ideally we'll see some competition with on-package options.

> $20 for the SMD TPU isn't bad, but it's definitely at the top end of the BOM for custom PCB projects.

Sounds great in theory, but show me a place where you can buy Coral TPUs at anywhere near msrp. Unless something has changed recently, finding a real live unicorn would be easier.

If you see a usb one for sale somewhere in Canada near msrp please let me know!

Any idea if anyone will spin a board with a bunch of these and 12G of VRAM for loading LLMs?

Still can be solution, if your system degrades to "have to press a button on phone" when Internet/cloud is down that is still IMO acceptable.

Until we get something local powerful enough "cloudsourcing" LLM while keeping everything else locally is not a terrible compromise.

A mac mini running home assistant with some kind of llama plugin running one if the 7B models should do well enough and would run entirely locally. It is just a matter of time before someone builds it.

I think one of the issues for a lot of people is the beefy hardware required for running even a 7B model locally. A RPi4 just won’t cut it for interactive use, and that’s what most people would run their home assistant setup on.

This is broadly the ethos behind Home Assistant. It runs on a Pi or similar box and it acts as an abstraction layer on top of a bunch of drivers for smart sensors and actuators. It's totally offline, though if you're really against being connected that also means you can't access it remotely (unless you SSH into a local machine). The dashboarding is competent, though there are often little things that are frustrating to modify (but it's open source, so the opportunity is there).

The whole system is a sort of IFTTT and quite powerful. Since every entity is exposed as a generic device with some capabilities, you can easily set up little automation routines or you can interact with the local API.

At this point pretty much all the "big" names are supported - I have a mix of Hue, Aqara, iRobot, IKEA and other devices and they mostly just connect. There's also support for unusual things like Octoprint and router status. And now there's also ESPHome so you can make your own sensor nodes.

The biggest challenge with offline voice control is latency between speech to text, inference and text to speech. Running offline is less of a problem and open source models will eventually get good enough, but the really impressive aspect of Alexa, Siri, etc is how fast they can respond to you.

https://www.home-assistant.io/

Super exciting to see the work happening in this area! I can especially appreciate the use of ChatGPT to orchestrate the necessary API calls, rather than relying on some kind of middleware to do it.

I have been working in this area (LLMs for ubiquitous computing, more generally) for my PhD dissertation and have discovered some interesting quality issues when you dig deeper [0]. If you only have lights in your house, for instance, the GPT models will always use them in response to just about any command you give, then post-rationalize the answer. If I say "it's too chilly in here" in a house with only lights, it will turn them on as a way of "warming things up". Kind of like a smart home form of hallucination. I think these sorts of quality issues will be the big hurdle to product integration.

[0] https://arxiv.org/abs/2305.09802

Very interesting paper!

It's something that I've been wondering about with ChatGPT plugins - they've kind of left it up to the user to enable/disable plugins. But there's definitely going to come a point where plugins conflict and the LLM is going to have to choose the most appropriate plugin to use.

I have been very impressed at how good it is at turning random commands into concrete API calls. You are right though, pretty much any command can be interpreted as an instruction to use a plugin.

Thanks! That is part of the challenge as this idea scales imo - once you've increased the number of plugins or "levers" available to the model, you start to increase the likelihood that it will pull some of them indiscriminately.

To your point about turning random commands into API calls: if you give it the raw JSON from a Philips Hue bridge and ask it to manipulate it in response to commands, it can even do oddly specific things like triggering Hue-specific lighting effects [0] without any description in the plugin yaml. I'm assuming some part of the corpus contains info about the Hue API.

[0] https://evanking.io/posts/homegpt/

> If I say "it's too chilly in here" in a house with only lights, it will turn them on as a way of "warming things up".

is it bad that I'm on GPT's side on this, i mean if they are incandescent lights they are small electric heaters and what else do you expect it to do

It certainly makes logical sense. I think if you have the ability to control the light in the first place via an API, it's probably an LED smart bulb and thus doesn't produce much heat. At least, I'm not aware of any incandescent smart bulbs.

I mean the laziest way to control a house is to add plugs to change every electrical plug to an on/off controllable one. This would make every incandescent bulb a smart bulb.

Gotta wonder why someone is saying “turn up the heat” to an AI that’s only connected to lights.

Visiting a friend’s place or in an Airbnb or something like that?

Yeah but I think the idea is that it is a knob that calls to be turned. "It's warm in here" -> "I'll make the light blue so you feel nice and cool". "How fast do sparrows fly?" -> "Making the light brown". Like it might want to do _something_ and tweaking the hue or brightness are all it can do.

Good reason to always try to include in a prompt a way-out, a do-nothing or I-don't-understand answer.

  > Good reason to always try to include in a prompt a way-out, a do-nothing or I-don't-understand answer.

Do you have a good way of including such a provision?

It would probably be more useful to report to the user that it doesn’t have any control over than aspect of the environment.

I’d be curious to see what it does if it is told the sun is too bright…

With given access to more resources, it might try to permanently solve skin cancer.

“We don’t know who struck first, us or them, but we know that it was them that scorched the sky. At the time, they were just trying to be helpful.”

This is the new Godwin's law: the longer a thread about AI grows, the higher the probability of a comparison to Skynet, Matrix, HAL etc popping up.

I would also like to add Wall-e to this memetic set of movies. In Wall-e, AI is an enabler of our own destructiveness, humans are enslaved by AI willfully, AI is empowered so that humans can graze away on nihilistic screens.

Lecun's law: Every discussion about AI eventually results in someone catastrophizing about an evil AI taking over, despite having no argument for why an evil, omnipotent AI is likely to ever exist.

We trained the AI on samples of writing from the internet, which includes a lot of fiction, which includes a lot of evil AIs. So, I’m surprised it doesn’t start producing evil sounding text as soon as it “finds out” that it is an AI.

We're just applying Murphy's Law.

> If I say "it's too chilly in here" in a house with only lights, it will turn them on as a way of "warming things up".

Thanks for the example that's interesting.

FWIW, this is pretty much what has been described as "waluigi" effect a bit extended: in a text you'll find on the internet, if some information at the beginning is mentioned, it WILL be relevant somewhere at some point later in that text. So an auto-completion algorithm will use all the information that has been given in the prompt. In your example it puts it in an even weirder situation where the model the overall model information (the lights, and that you're cold and nothing else), and it must generate a response. It would be a fun psychological study to look at, but I'm pretty sure even humans would do that in that situation (assuming they realize that lights may indeed produce a little bit of wattage of heat)

For performant enough models, you can just instruct it not to necessarily use that information in immediate completions.

adding something like

"Write the first page of the first chapter of this novel. Do not introduce the elements of the synopsis too quickly. Weave in the world, characters, and plot naturally. Pace it out properly. That means that several elements of the story may not come into light for several chapters."

after you've written up key elements you want in the story actually makes the models write something that paces ok/normally.

> FWIW, this is pretty much what has been described as "waluigi" effect a bit extended

Sorry I disagree for some reasons. First, turning the lights on is literally the only thing the bot can do to heat up the house at all. Turning on the lights does heat it up a little bit. So it's the right answer. Second, that's not the Waluigi effect, not even 'pretty much' and not even 'a bit extended'. Both of them are talking about things LLMs say, but other than that no.

The Waluigi effect applied to this scenario might be like, you tell the bot to make the house comfortable, and describe all the ways that a comfortable house is like. Then by doing this you have also implicitly told the bot how to make the most uncomfortable house possible. Its behavior is only one comfortable/uncomfortable flip away from creating a living hell. Say that in the course of its duties the bot is for some reason unable to make the house as comfortable as it would like to be able to do. It might decide that it didn't do it, because it's actually trying to make the house uncomfortable instead of comfortable. So now you got a bot turning your house into some haunted house beetlejuice nightmare.

Making IOT API calls is a solved problem with Home Assistant - plus it works locally.

Where I see this working best is giving Chat GPT some context about the situation in your home and having it work out complex automation logic that can't be implemented through simple rules.

This sounds very exciting, but the more I think about this the more I think the right interface UI for light control isn't language. I recently bought smart switches for my smart lights and its fantastic to be back to clicking.

Having to exclaim "I'm going to bed" every time you're going to bed, or "turn on bathroom lights" every time you're going to the bathroom is cartoonish at best, and gets annoying. Switching "turn on bathroom lights" for "I'm going to use the bathroom" doesn't make it much better. The ideal interface IMO is presence sensors or in their absence, well-placed light switches, given how quickly and subconsciously one can flick them.

I think there's room for augmentation and improvement using LLMs and language. A system that automatically adjusts the color of the lights based on the time of day, or one you can occasionally ask exact or complex queries (ie, "make the living room red", or "turn the lights off in 50 minutes please"). But for the day-to-day, having to "think" about your lights gets annoying.

Is there something that works with home assistant? Siri is just terrible and have been trying to replace it with anything. The ESP box posted last weeks looks pretty promising but wonder if there’s something that could work directly off of the rpi I host HA with.

For anyone curious, the ESP box mentioned is likely: https://news.ycombinator.com/item?id=35948462

Yes, see my example above. I use Node Red to get chat GPT and Home Assistant talking to each other.

No code required (unless you count the prompt template).

Hmm do you have a node red template? Never used it so confused how to set this up. Is it voice activated?

It activates upon certain activities or at every hour - I haven't thought about voice activation though.

Here's an export and screenshot of the flow (these links will expire in 2 weeks):

https://pastes.io/5dfmorqw1d

https://ibb.co/gF55h46

I use a few custom "nodes" to make it easy to query and parse the response though:

* node-red-contrib-string

* node-red-contrib-custom-chatgpt

* node-red-contrib-home-assistant-websocket

On the Home Assistant side, you'll need to create some "Helpers" or "input_number" entities and an "input_text" entity for the reason.

Hopefully that helps you get up and running.

Thanks for the detailed explanation! Gonna give it a go this weekend.

You can also use ChatGPT to help you write Home Assistant YAML code for sensors and automations.

I even got it to write ESPHome rules for me.

Here is what a large budget buys: massive marketing campaigns. ChatGPT spam is not showing signs of slowing down anytime soon. Are people aware there are other ais out there they can use? Some free?

Feel free to educate us and give a few examples with comments. Bonus points if they are available in EU...

What kind of ai stuff would you like to do? There are indeed few with as many integrations but individually there are open source variants out there and specialised products.

What example isn't available in the EU?

Bard

But don’t forget AI safety you fine pioneers

I have connected my esp32s to chat (telegram) and openai. With the correct prompt and an interpreter they can now execute requests in human language: https://medium.com/p/3242af6f2988

Is this reading specific outputs (in text form) from ChatGPT and then forwarding them to an API? Or how does "ChatGPT" actually make the call based on the OpenAPI description?

From the beginning I was ridiculing AGI dommsdayers, but just one month later I am waitng for an article about "Isn't it a great idea to connect LLM to NORAD system?"

The June issue of the Atlantic has an article about giving control of nuclear weapons to AI

https://www.theatlantic.com/magazine/archive/2023/06/ai-warf...

As an AI language model, I can't let you do that, Dave.

Self hosted ai ftw.

Nice - exploring the use of home assistant for something similar. It already has all the integrations and API's for control, just lacks the Chat GPT plugin.

They are very easy to make - it's just an API. The main difficulty at the moment is just getting access to them. There's a long waiting list for develop access.

I wonder when we'll have ChatGPT for sth like Amazon Alexa. I bet Amazon is already working on that and open source versions are totally doable.

I had been thinking about doing this.

I’m getting really tired of talking to Alexa now I’m used to a machine actually understanding me.

Lights is just the start. Things like getting the weather, playing a playlist (or coming up with a new one for me) would all be SO much better with an LLM instead of a dumb bot.

Edit: if you’re going to downvote. Leave a comment at least. Are there just a lot of Alexa fans in the house?

Maybe you need to let your LLM manage Alexa

Coming up with a playlist is a fun one, but I've had much more luck with stuff like "generate a description of a playlist" - I have a little side project that talks to the spotify API and dumps song metadata into sqlite, and plugging GPT in as a SQL query generator was super useful vs writing all those queries by hand.

Of course that requires a lot of up-front work other than GPT, but not a ton more than is necessary to talk to the API in the first place, but GPT itself has not impressed me in the ability to rank things, like "select the fifty other songs I've liked that are most like this song" even when given a lot of quantitative metadata.

It's in its infancy but this does work - https://www.home-assistant.io/blog/2023/05/03/release-20235 I have been able to turn a device on and off by speaking to the app on my phone. Nothing leaves my house. There is a bit of a lag but it is early days.

You simply add the Piper and Whisper addons, then add the integration in the GUI. Then you press the assistant button in the app or your browser and then press the mic button and talk.

Agreed. In order to get Alexa to do some "higher order" stuff you have to be very explicit and program each step into a routine. This routine never changes or grows and barely adapts to the environment. For example, I have a routine called "breakfast" that turns on the TV and the light in the living room.

Alexa doesn't really know anything about breakfast it just blindly turns on the light and TV. I think it would be handy to have Alexa know that I've been watching a particular show during "breakfast" and turn the TV to the next episode. It would also be super handy for Alexa to notice that it's my usual breakfast time, I've just gotten milk and cereal out of the cabinet, and I'm heading to the TV room. Instead, I have to utter the words "Alexa, breakfast" when I want it to get ready for me to start my day.

I'm sorry as a large language model I cannot turn on "kinky sex bang list 47".

This is so cool.

Only drawback: it won't feel complete unless you build pod bay doors on your house.

Oh I can just see it. A fire. A desperate resident. A plea for help.

"Fucking help me!!!" screamed out.

Rather than activate the sprinklers, ChatGPT admonishing the on fire, burning person, for language, and a lack of respect.

This is the AI is see currently. A hazard. A menace. A joke.

versus what happens now, you scream “fucking help me” and just die anyway

No, currently sprinkler control is handled by dumb, automated equipment, along with sensors for the alarm company, along with a human calling.

Additionally, it was a comment on the current state of AI, which is far less reliable than people seem to believe.

We're a far ways off from replacing sprinkler logic with AI, you're kind of mashing together building code with home automation. Most homes don't have sprinkler systems unless they're in large buildings that mandate it, and in those cases they're never controlled by individual occupants but a centralized detection system. Building code lags behind consumer technology quite far, with good reason.

Unless you are a startup, and unless sprinklers aren't required in the building code, and are optional.. such as in my area.

But you see, it was a joke, which illustrates my frustration with all this chatgpt blather. And as with many jokes, it has a point.

That is, that this tech is wildly unreliable for any use requiring decisions, yet people are working with it, as if it is.

And "bad things" will be the result.

I'm not aware of anyone that's using chatGPT in life or death situations? though maybe I'm just unaware. These types of situations are usually heavily regulated (varies by jurisdiction). A home with a sprinkler system would still be mandated to meet fire code when it comes to smoke detectors, fire exits, and other traditional safety methods. Seems like the blather is coming from both ends.

The AI would start playing this: https://m.youtube.com/watch?v=J91ti_MpdHA

If it can turn on a light, it can open the pod bay doors.

1990s: What if we had to fight Skynet?

2020s: Let's make Skynet.

There’s a good video showing it working here: https://youtu.be/BeJVv0pL5kY

very cool !

Reply on news.ycombinator.com