S.e.p.i.a. – Digital voice assistant
sepia-framework.github.io
sepia-framework.github.io
I'm still looking for a privacy / offline voice assistant solution, so I'll dig a bit on this.
But, still, an exciting project. I would love to rid my house of Google Home devices which are training my kids to think Google is god.
My parents would simply scream with a loud bang from a foot stomp or fist slam "Enough!" OR "shut up, the both of you!" which worked very well. Did not require purchasing anything, accounts or internet connection.
A lot of deployed deep learning is in Python so I doubt this will matter. Looking at the docs, it looks like the main speech to text is python heavy, it leverages elasticsearch and calls libraries that are bindings to code written in C++. As none of that overhead will ultimately matter for anything approaching the fidelity you get with cloud services, cost benefit trade off favors Java in this scenario.
> lower profile computers
When it comes to deep learning, this just isn't the direction things are headed. If you want the best results with low latency, your home server is going to have to be fairly beefy with a decent GPU and lots of memory.
> When it comes to deep learning, this just isn't the direction things are headed.
You're not going to be training the models on the raspberry pi, just running them. There's a huge difference in requirements there, and lots of progress has been made towards squeezing models into phones etc. ( https://pytorch.org/mobile/home/ ). There are also efforts like like https://github.com/XapaJIaMnu/translateLocally , https://neural.mt/speed/2021/
As someone who never uses Java _and_ still dislikes it, it's because i've had to install that damn runtime a thousand times. At least Python has the simplicity in coming pre-installed on many systems. I can't count the number of times i've installed/uninstalled Java.
Also, back in the day i was used to seeing some Java systray icon. I imagine that built some dislike towards Java too.
At any rate, Java isn't a client language. It's very much geared toward server development. It's not impossible to write client software with it (see Minecraft), but it's really clunky and not really suitable. They've basically done everything they can to make it as awful as possible to use for desktop users, basically ceding the ground to C# almost more through self-sabotage than losing a fight.
I don't want to be conspiratorial, but it can be suspected Microsoft and Oracle had lunch together and came to some sort of an agreement.
Think of it in terms of human languages.
C and C++ (and many - but not all - compiled languages) get compiled into Native program, so they get compiled into instructions that talk the computer's language, and when they get run they talk directly to your computer... albeit with a thick accent. So a linux machine can only talk to your machine via the linux kernel, a windows machine talks with a windows accent, macs... you get the idea.
Because of this thick accent, programs cannot be shared between different operating systems (though there are some amazing projects out there that either do some accent translation, or more recently, do clever things to avoid any accent at all. There are almost always limitations to what can be done here though.)
That's not to say that sometimes, they don't need help. These "helpers" (or Shared Libraries - aka DLLs or .so files) might be absolutely necessary for a certain program to run, because the developers don't want to reinvent the wheel. So as someone else pointed out, even with native programs you sometimes need to download extra bits to get them running. But as you've noted, this is a lot more rare, and it doesn't usually change HOW you run the program, merely what needs to be installed first.
PHP, Ruby, Bash etc are Interpreted - so they need a special program that does the talking to the machine (again, via the kernel). This means they don't need to be compiled, which makes them quicker for developers to write, and easier to run on windows / linux / mac using the same codebase. But like with a human interpreter, it generally takes at least 2x as much energy to get the message across, which usually means a sometimes significant slowdown.
Note that this is not the same as having a Shared Library - and in fact these programs sometimes do use Shared Libraries directly. With interpreted languages, the interpreter starts first, it gathers together some core resources and ideas that it will need to be able to do its interpretation, and then it will load the human-readable program. Some shared libraries will be loaded by the interpreter, and some will be loaded later once the interpreter has seen the code.
Java is in a special category. It is both interpreted and compiled - that is to say, it compiles to something that looks a bit like a native program, but it still has this interpretive layer that gathers together core resources and ideas that the compiled program can then use - we call this part of the interpreter a VM, or Virtual Machine. It then loads a program that is very much not human-readable into that VM, which then does all the translating.
The theory is, Java could be the best of both worlds. It can have speed closer to a compiled language (and in fact, can sometimes be a little faster in places, as the VM contains some nifty tricks to move things around on certain machines) and yet the compiled languages can be run on any machine that has the Java Runtime installed (I believe they coined the phrase "Write once, run everywhere" which very soon got corrupted to "Write once, test everywhere" because the first few versions of the Java VM were spectacularly buggy)
As for that dislike of Java, I will point out that I spent 5 years writing Java programs. Once I was no longer forced to write Java programs, I stopped, and haven't touched it for at least 10 years. Because while Java could be the best of both worlds, it wasn't then. And it might be much better now... maybe. I cannot forgive it for the years of my life that it stole; and am now very happy in the arms of pretty much any other language.
That's not even getting into the Java ecosystem which is vast, but it's full of nightmares like remote debugging ports that allow unauthenticated remote access to the JVM's memory space.
I feel like the meme that Java is slow and bloated needs to die. Java can be very fast, and also runs on fricking SIM cards that have extremely limited processor and memory.
Granted, often enough Java is fine, it doesn't use too much overhead, especially once the JVM is warmed up. But the bloat is real and it's partially related to the language itself but plenty of it is also systemic.
* It's way faster to say "hey Google, set a 10 minute timer" than to use any timer I've ever owned.
* I can have multiple timers running side-by-side that remember what they were for. "Set a timer for 30 minutes for the potatoes."
And, yes, as you mention, I can set a timer while my hands are still dirty or while moving on to the next step.
https://www.amazon.com/Lux-Minute-Minder-Mechanical-Markings...
in kitchen when cooking stuff
Kitchen Timer
Set Alarm
Play Music
Check Weather
Open or close Garage door
Turn off everything in the loungeroom (theater)
Broadcast to the house when i want some ones attention
Send text messages (not very often though, this interface is very clumsy and i detest having to speak. like. a. robot. to. be. understood.)
- To adjust lighting throughout the house, and to shut off all the lights across the entire house when we go to sleep
- To turn on the oven or air fryer, so it's preheated when we need it... can even be done from the car on the way home
- To turn off the oven when you have that thought "did I forget to turn off the oven?" 30 seconds after driving away from the house
- To adjust the thermostat without walking upstairs to adjust the physical thermostat
- To get an announcement when the clothes washer is done so I don't forget to move the clothes to the dryer
- To send a quick voice message to my wife somewhere else in the house using the "make an announcement" command
- To adjust the volume on the TV when I don't have the remote nearby
- To start pre-heating or pre-cooling my electric car in the driveway while I'm still getting dressed
- To reorder common household things like pet food or toilet paper, and it shows up at the door the next day
- To check the weather before leaving the house, to decide if I need a jacket or not
For example, displaying a rain cloud symbol and temperature seems just as effective and quicker than asking alexa.
The little speakers can be put into every room, almost invisible, and for a fraction of the cost of a smart display.
Also, at night, smart displays suck. They always turn on, when you do not need them. Waking you up on the way to the toilet.
Lots of these tasks are completely automatic (e.g. locking doors, announcements, turn things off when I am gone, motion sensors....) or simple inputs, e.g. volume buttons or a slider for oven temp.
> The little speakers can be put into every room, almost invisible, and for a fraction of the cost of a smart display.
My phone is in my pocket and thus in every room I am in.
So let me rephrase the question: If I use my phone, for which things is voice input so much better than just clicking on widgets?
Thus far I think hands-free while cooking and the UX for music/video are the standard examples. The question above was: Besides this, what are great use cases?
Not everyone does this. Eg. My parents use Alexa but keep their cell phone by the door or on their nightstand.
> So let me rephrase the question: If I use my phone, for which things is voice input so much better than just clicking on widgets?
You already answered one: Cooking, etc. But also some people have other household members. The communal nature of the device matches the communal nature of a household. If I hear an announcement saying to put laundry in drier, I can act on it even if I didn't start laundry. Or if there's an timer to take something out of oven, I can act even if the cook is in the bathroom.
There's also no tangential distraction when looking up information. If I'm going to travel I can ask for the forecast (daily or weekly) at places along the way without getting bogged down with unrelated web browsing or advertising.
But to be honest, only very few use cases would be applicable in my standards.
For example, never would I want remote controllable locks.
That said, home automation is pretty neat.
Reminds me the opening scene of the 2022 movie Scream
95% of my usage is in the kitchen, and I think both amazon and google are hugely under-valuing that space.
My ideal voice assistant would be something like the Fire TV cube (streaming shows along with voice control) where the thing is able to show timers/recipes/unit-conversions picture-in-picture while the show is playing, as I ask.
Frankly - Amazon acted like a chicken with its head cut off with the Fire TV cube, it should be exactly what I wanted, but the device is almost laughably bad at basically everything it should excel at, and I still hate them a little for it.
- Reminders (managing, etc)
- Navigation (mostly in the car, but Siri kinda sucks at this)
- Music
- Lists (grocery, etc)
- Speech to text entry/messaging if typing is inconvenient (walking dogs, etc)
- Quick math/unit conversion
- Quick searches
- Seeing what entertaining things it can do
- Managing apple health stuff
I do have some home automation stuff (lights, thermostat, etc, but no locks), but Siri is not configured to manage any of it as of now.
I dream about “assistant” like Focus in “Horizon” – so in a more passive way.
For me, the best “assistant” is the one I don't pay attention to.
“Alexa, set an egg timer for 12 minutes”
“Alexa, set an alarm for 6:45am”
“Alexa, play Stories Podcast on Spotify”
These or variations of these are the most used commands at my house. Passive assistants can’t really figure out that I or my kids want to play a specific podcast or when to set a timer when cooking. Lighting is another tricky one. Sometimes I’m reading and want bright lights. Sometimes I want scenic lights. Sometimes I want very low lights while falling asleep.
I suspect most people don’t use their smart assistants for shopping. I do occasionally ask it a random question like “what year was Einstein born” when I don’t feel like whipping out my phone or play an occasional round of 20 questions with my kids. But that’s rare compared to the typical usage.
What I want is an assistant that will handle all my bookkeeping for me. Right now, I have to log in to the account, then download all the statements for the last tax year. Evidently, no bank has such an option. Nope. You gotta do it month by month, click, download, save, back, back, click, download, save, back, back, until you scream.
Then ya gotta download the transactions for the last year in a format that quicken can use. You can download transactions for last week, last month, last 12 months. No option for "last tax year". So you have to manually enter 1/1/2021, then 12/31/2021, then confirm, then download, then save, then do it all over again for the CVS version.
Then you gotta poke around looking for the tax documents.
Then I gotta get the transactions for a credit card I canceled 6 months ago. But since I canceled it, I can't download the transactions.
It must be only me that has these problems. Why is that? Does nobody else need to pay taxes?
No, I doan need no help setting a $%^&# egg timer :-) Maybe if I become bedridden I'll need help flipping a light switch. Until then, nope.
When walking into a dark room carrying a heavy box it is often easier to use a voice assistant to turn on lights. It’s also frankly more fun.
What you are looking for is not a home assistant. You want an accountant + QuickBooks. For $1000/year or so you can get both and be happy. Intelligence is often cheaper than artificial intelligence.
As for turning on lights when carrying, that's why I have an elbow! The switch is at elbow height.
> It’s also frankly more fun
What, this isn't fun? https://www.amazon.com/Clapper-Sound-Activated-Switch-White/...
For one, my grandparents, didn't have nearly the number of light switches even with a comparable house size. And "just teaching" kids to turn the lights off is quite laughable. My kids know how to turn off lights and know they should turn them off when they leave the room. But knowing and always remembering to do it are two different things. I also forget to turn them off sometimes. I remember my dad grumbling quite a bit about having to go around turning off the lights or having to run back in the house after you get outside and see the lighted windows. Just one less little hassle.
> But knowing and always remembering to do it are two different things. I also forget to turn them off sometimes.
What is "sometimes"? You wrote "I'd have to spend half an hour a day traveling around the house turning off light switches countless times" which is definetly not normal and a clear case of your parenting not working. I also "sometimes" forget a light. About once a year, tops. I don't see how this isn't your brains failing to manage a very basic every day task.
Most of the time Google needs “Set a timer for 3 minutes”.
It's much more difficult than home automation, but I think this is the direction we're heading into. The incentives are there, because access to such data is valuable.
Slow steps
BTW, Quicken is the most rube-goldberg-esque program I've ever encountered. There's no UI consistency anywhere, the screen constantly clears and flashes and redraws, some check boxes cannot be clicked on, you have to Alt-key them, some places Paste does not work, and it blocks you while it updates itself every week or so.
What a mess of a program. I bet it was written by squirrels and poodles.
One insanely simple thing that'd be useful to me: local tide times. Can Siri or Google do this for me? No, not a chance. Best I get is "here's a web page", ie just a higher friction version of the thing I could have done myself.
I'm ok turning my music up and down or choosing a song. Let's make these devices actually useful and then I might sit up and start using them.
Alternatively, try calling you bank and asking you can get the statements emailed to you automatically. Most banks offer this in one way or another, although you usually get PDFs so some programming might still be necessary to transform the data.
See where this is going? Spending a weekend automating a program that you use only once, and have to recode every year, does not work.
Also, quicken cannot import PDFs. As for getting the statements via email - it's just as much work getting them one by one out of my email and into the proper place in my accounts.
What I want:
Download in one operation ALL the data in one zip file.
Now that would save me significant time. I've never had an account that allowed this. They're all different, but all about 20 minutes or so of clickety-clickety-clickety.
BTW, at least they finally stopped downloading each statement with the same file name (named "download.pdf", naturally). So I had to rename it as an additional step for each statement. It lead me to wonder didn't anyone who worked at that bank on the web site have an account at the bank?
But nothing ever changed, so I stopped bothering them with suggestions.
Some regular banks support it, you can use your existing credentials, and there are Node.js libraries for it. Downside is the metadata is lean. I think there’s a ~10char limit on the description? Like you get “2021-11-01 14:35, LONGNAMECO, $500” and that’s it. Don’t quote me.
A friend of mine does a shopping list on his "add eggs to shopping list".
It’s not that it looks useless. It’s sure looks pretty cool to be able to switch on the light with the sound of my voice.
But the privacy cost is extremely huge for something that isn’t that much revolutionary. I’ll just keep pressing the button. That’s less cool, I admit that and I’m even somewhat jealous. But I’m never going to allow Amazon (or any company) listen to me constant just not to press a button.
I respect and have no issue with your choice but it feels wrong justifying the utility without mentioning the drawbacks.
Not always possible, I heavily make use of Alexa while both my hands are dirty from baking or cooking. Both dumb units to metric, and setting timers.
But I am pretty happy with the voice recognition, so I’m currently waiting on Pi Zero 2 W’s becoming available again and replacing her with Rhasspy.
What did you do before your voice assistant? Did you leave the light on until such time that your hands became clean?
I don't think it's really a comparable to a voice assistant. It's more "Google Glass" than "Google Assistant".
So, scroll down a ways, then click "getting started" at the top right of the page, then there's a "quick start" guide.
It's self hosted, so you probably won't find an online demo. If you don't want to try it without an online demo, then this probably won't be that interesting for you anyways, since custom voice assistants necessarily require customization.
I’m interested in what platform you’re using, because I wasn’t aware that any made a visual distinction of the error they’ve made. Specifically, they have the root element hiding overflow (well, it’s #site-window that does the actual `overflow: hidden`, but it and its ancestors control their height to 100% of the viewport), and an element inside that (#site-main-views) being the scrolling element. I noticed this because keyboard navigation (Up/Down/PageUp/PageDown/Home/End) failed to work, since the initial containing block doesn’t have any scrolling, and you have to focus the scrolling element (e.g. with Tab or clicking in it) before keyboard navigation will work.
It's a hard problem. From my understanding, homepods use bluetooth to coordinate who is going to answer. Google Home, at least in my home, is terrible at this problem. Often devices in another room will answer.
Once someone has solved this problem, I'm on board!
Depending on the phone's proximity is still annoying, though. It seems weird that they don't seem to use voice proximity to determine which one is closer.
Most of those things are about one step above "press 1 for sales, press 2 for tech support". You just get to say "sales" instead of pressing 1. So if you want to convince people yours is better, a good demo is essential.