balloob

Thanks! We're continuously improving it and nowadays have two people full-time working on it.

If you want to hear about the future of ESPHome, make sure to tune into the State of the Open Home next week: https://www.youtube.com/live/oa__fLArsFk

Today at Home Assistant we announced the release of microWakeWord. It is a new wake word model that is fully open source and runs on the ESP32 S3 microcontroller. It is integrated into ESPHome and now powers the latest generation of Home Assistant's privacy focused voice assistant.

Overview of Home Assistant's voice work: https://arstechnica.com/gadgets/2024/02/can-you-manage-your-...

More information about todays release: https://www.home-assistant.io/blog/2024/02/21/voice-chapter-...

Create your own voice assistant directly from your browser: https://www.home-assistant.io/voice_control/s3_box_voice_ass...

Video of it in action compared to streaming the audio to Home Assistant where it is passed to openWakeWord: https://www.youtube.com/watch?v=oSKBWtBJyDE

Everything we make is accessible via APIs and integrating our Assist via APIs is already possible. Here is an example of an app someone made that runs on Windows, Mac and Linux: https://github.com/timmo001/home-assistant-assist-desktop

That's something we've been building towards to all of last year. Last iteration can be seen at [1]. Still some checkboxes to check before we're ready to ship it on ready-made hardware.

[1]: https://www.home-assistant.io/blog/2023/12/13/year-of-the-vo...

That's a good one.

We might take it one step further and ask the user if they want to add a rule that certain rooms have a certain level of light.

Although light level would tie it to a specific sensor. A smart enough system might also be able to infer this from the position of the sun + weather (ie cloudy) + direction of the windows in the room + curtains open/closed.

That could be solved by using something like Anthropic's Constitutional AI[1]. This works by adding a 2nd LLM that makes sure the first LLM acts according to a set of rules (the constitution). This could include a rule to block unlocking the door unless a valid code has been presented.

[1]: https://www-files.anthropic.com/production/images/Anthropic_...

As a user of Home Assistant, I would want to easily be able to try out different AI models with a single click from the user interface.

Home Assistant allows users to install add-ons which are Docker containers + metadata. This is how today users install Whisper or Piper for STT and TTS. Both these engines have a wrapper that speaks Wyoming, our voice assistant standard to integrate such engines, among other things. (https://github.com/rhasspy/rhasspy3/blob/master/docs/wyoming...)

If we rely on just the ChatGPT API to allow interacting with a model, we wouldn't know what capabilities the model has and so can't know what features to use to get valid JSON actions out. Can we pass our function definitions or should we extend the prompt with instructions on how to generate JSON?

Streaming responses is definitely something that we should look into. The challenge is that we cannot just stream single words, but would need to find a way to learn how to cut up sentences. Probably starting with paragraphs is a good first start.

I think that LLMs are going to be really great for home automation and with Home Assistant we couldn't be better prepared as a platform for experimentation for this: all your data is local, fully accessible and Home Assistant is open source and can easily be extended with custom code or interface with custom models. All other major smart home platforms limit you in how you can access your own data.

Here are some things that I expect LLMs to be able to do for Home Assistant users:

Home automation is complicated. Every house has different technology and that means that every Home Assistant installation is made up of a different combination of integrations and things that are possible. We should be able to get LLMs to offer users help with any of the problems they are stuck with, including suggested solutions, that are tailored to their situation. And in their own language. Examples could be: create a dashboard for my train collection or suggest tweaks to my radiators to make sure each room warms up at a similar rate.

Another thing that's awesome about LLMs is that you control them using language. This means that you could write a rule book for your house and let the LLM make sure the rules are enforced. Example rules:

* Make sure the light in the entrance is on when people come home. * Make automated lights turn on at 20% brightness at night. * Turn on the fan when the humidity or air quality is bad.

Home Assistant could ship with a default rule book that users can edit. Such rule books could also become the way one could switch between smart home platforms.

Founder of Home Assistant here. Great write up!

With Home Assistant we plan to integrate similar functionality this year out of the box. OP touches upon some good points that we have also ran into and I would love the local LLM community to solve:

* I would love to see a standardized API for local LLMs that is not just a 1:1 copying the ChatGPT API. For example, as Home Assistant talks to a random model, we should be able to query that model to see what the model is capable off.

* I want to see local LLMs with support for a feature similar or equivalent to OpenAI functions. We cannot include all possible information in the prompt and we need to allow LLMs to make actions to be useful. Constrained grammars do look like an possible alternative. Creating a prompt to write JSON is possible but need quite an elaborate prompt and even then the LLM can make errors. We want to make sure that all JSON coming out of the model is directly actionable without having to ask the LLM what they might have meant for a specific value.

I hope you meant Zigbee dongle instead of Z-Wave. Hue devices use Zigbee and can be directly paired to Home Assistant using a Zigbee dongle, but not using a Z-Wave dongle.

We’re working on making a voice assistant with Home Assistant[1]. We have all parts done except for wake word, which Hopefully we’ll be able to ship soon.

[1]: https://www.home-assistant.io/blog/2023/04/27/year-of-the-vo...

We’re super stoked to launch the Home Assistant Green[1]. It will be $99! The first 1000 are available today at Seeed, and 14000 more will be available end of October. More to come afterwards.

With Home Assistant Green we hope to make it very easy for people to join the Home Assistant community and experience a smart home focused on privacy and local control.

[1]: https://www.home-assistant.io/green

Some feedback to make your project easier to install and integrate better with Home Assistant (I'm the founder):

Home Assistant is building a voice assistant as part of our Year of the Voice theme. https://www.home-assistant.io/blog/2023/04/27/year-of-the-vo...

As part of our recent chapter 2 milestone, we introduced new Assist Pipelines. This allows users to configure multiple voice assistants. Your project is using the old "conversation" API. Instead it should use our new assist pipelines API. Docs: https://developers.home-assistant.io/docs/voice/pipelines/

You can even off-load the STT and TTS fully to Home Assistant and only focus on wake words.

You will see a lot higher adoption rate if users can just buy the ESP BOX and install the software on it without installing/compiling stuff. That's exactly why we created ESP Web Tools. It offers projects to offer browser-based installation directly from their website. https://esphome.github.io/esp-web-tools/

If you're going the ESP Web Tools route (and you should!), we've also created Improv Wi-Fi, a small protocol to configure Wi-Fi on the ESP device. This will allow ESP Web Tools to offer an onboarding wizard in the browser once the software has been installed. More info at https://www.improv-wifi.com/

Good luck!

Once we get our hardware done, this is feature that we definitely plan to include (but probably as a later software update)

1. We don't currently have anything for that besides looking at the database directly.

2. We offer VM images if you want to keep flexibility but also benefit from all our work on an integrated system. You can also do a Supervised installation, but then it's your own responsibility to keep the host OS running the requirements of HA.

In Home Assistant a voice assistant is a combination of a speech-to-text, conversation and text-to-speech engines. A user can create as many combinations as they want. For a conversation agent, it's text command in, text response out. We already have the option to use OpenAI GPT 3.5, so it's possible in the future to hook it up to a large language model and allow it to control your house.

Fun fact: we supported Almond (later Genie), the IoT AI from Stanford which they released a couple of years ago. It would convert natural language into their ThingTalk language, which could do things like you mention. It was not popular at all and Stanford ended up shutting it down.

Yes, that's the plan.

We're still exploring different ML on the edge chips to help with wake word and audio processing. One hard requirement that we have is that the tooling is openly available, so that anyone can convert an AI model and run it on the chip. We don't accept tooling to be available as a website or under NDA. That challenge is harder than we expected!

Founder Home Assistant here. Let me know if anyone has any questions.

Edit: if you want to keep in the loop of the work we're doing, subscribe to our free monthly newsletter @ https://building.open-home.io/

It’s possible. We’ve published a couple of example apps for ESP32 variants build from the official SDK. You can find them at https://nabucasa.github.io/matter-example-apps/

Yep, these are the way to go. Small and with a case, so ready to put around the house.

Using just your browser you can transform these into Bluetooth proxies for Home Assistant with https://esphome.github.io/bluetooth-proxies/

There are enough products and projects available to build a smart home based on just open source code: Home Assistant, ESPHome and WLED for example. And soon you can add Matter to that mix. Product manufacturers could publish their source if they wanted to.

For the Raspi images: the build pipelines are often hosted on GitHub and all the build logs and source is available. If anything, open source projects that are big enough to offer their own OS often receive more scrutiny than most closed source products do.

ESP32-H2 supports Thread: https://www.espressif.com/en/news/ESP32_H2

That's also the chip that Espressif has build it's reference implementation Thread Border Router around: https://openthread.io/guides/border-router/espressif-esp32

But you can also use ESP32s for Thread. If you have one that we support laying around, we have an installer website here that we use for testing https://nabucasa.github.io/matter-example-apps/

Brought to you by our $20k/year annual membership fee that Home Assistant, an open source project, pays to be a member of the CSA alliance and be able to contribute to the open source Matter reference implementation.

They will because there is an open source reference implementation that they can use: https://github.com/project-chip/connectedhomeip

At Home Assistant we're very excited about Matter.

Here are some highlights for us:

Open source reference implementation: Google, Apple, Home Assistant, we're all going to be running the same code to be a Matter controller. Chip manufacturers like Espressif and Nordic maintain implementations for their boards in there too, so anyone can now have the software to produce Matter compatible light bulbs. https://github.com/project-chip/connectedhomeip

It will be cheap: The software is freely available and works with a big audience. It's the same reason Android TVs from some manufacturers are cheap, the same will be the case for Matter lights and switches.

Multi-fabric: each Matter device is required to support 5 fabrics. A fabric is a Matter network. This means that you will be able to run multiple home automation controllers at the same time. So when run into the limitations of Google Home or Apple Home, you can try out Home Assistant without taking down your old system.

Easy sharing of devices: Because of multi-fabric, it will be as easy as hitting a share button to get a device added to another fabric. See this example of Android https://twitter.com/home_assistant/status/157703612255503564...

Local: all communication for Matter is happening local between a device, a thread border router (if thread-based Matter), and the controller. Note that your controller can still decide to store your data in the cloud (ie Amazon, Google).

Supported by major systems: Amazon, Apple, Google and Home Assistant are all building the open source Matter code into their systems. It means that for manufacturers it will be easy to pick Matter as the protocol they want to support to reach most users.

Works over IP: Matter works over IP and doens't care how that IP-based device communicates. It means that you can have Wi-Fi based and Thread-based devices co-exist on your network. Thread is not required if you don't care about such devices.

Bridges are part of the standard: Devices like Philips Hue hubs are going to get an upgrade to expose all the Hue lights over Matter via the hub. This makes integrating a whole ecosystem at once into Matter very easy.

Thread: Thread is a mesh networking standard that connects to your Wi-Fi network via border routers. Where Zigbee and Z-Wave need to mesh communicate all the way to your controller, Thread messages will be delivered via Wi-Fi/ethernet as soon as possible. This means it is a lot more reliable and less traffic is going over the mesh. Expect cheap border routers (open source reference implementations) but also expect them in your future Wi-Fi routers, voice assistants, Wi-Fi connected TVs etc.

Paulus /Founder Home Assistant https://www.home-assistant.io

Give it some time. Since the software for devices is all available as open source, I expect a lot of commodity products (lights, switches) to be made available as pure Matter, without any apps, because it reduces the software development costs. The Matter early adopter companies are all companies with more expensive product lines.

Yes! It's maintained by the members of the CSA alliance (which authors Matter). It means that Google, Apple and Home Assistant will all run the same controller code and benefit from bug fixes. Chip manufacturers like Espressif and Nordic maintain implementations for their boards in there too, so anyone can now have the software to produce Matter compatible light bulbs.

https://github.com/project-chip/connectedhomeip

Because they didn't. The only thing that Thread shares with Zigbee is the radio. It's a completely different standard.

Matter certification requires that controllers do attestation (validate that the device says they are who they say they are) but you are allowed to offer an option to users to bypass it (like Apple does today for HomeKit today).