Android Audio Latency
androidpolice.com
androidpolice.com
That the Android designers are having to play catchup with this speaks a lot, I think, of how significant design-decisions were influenced by the territory from which the originated - the web, managed systems - and the realm which they hoped to capture - the embedded system in your pocket. Android was not designed up-front to be an amazing media system; iOS, was. Thus, iOS has more synth-plugins and a vibrant, thriving Audio-Application eco-system (seriously, its million-dollar) whereas Android has a lot of new players, but so far: not such a great playground, really, for "Musicians".
In fact, while iOS has really impinged upon the Music-Instrument (Digital) industry, Android has yet to make a dent.
So it is an interesting time to be having new solutions to what have been, relatively, frustrating and troublesome aspects of Android realtime Audio, as a developer.
I could be confusing them with Palm OS, which did the same for the kernel of their first Pilots...
To me it's not surprising that iOS was the go-to for audio, based on historical prescedent that both MS and, from what I understand, Linux platforms didn't really have strong native audio processing for low latency. I mean, yes, it was totally do-able (MS via ASIO4ALL, I know there were a few Linux based music systems even that ran on Atom netbooks), but plug-and-play wasn't quite the same. Even audio cards had some driver issues in the MS realm, speaking from personal experience.
It's rather sad because the MS environment allowed for a lot of homebrew plugins - the VSTs. Those won't run in Apple's ecosystem. Other plugins do (ex. UAD) but for the longest time those required (still do?) their own hardware processing cards. $$$$$$$.
iOS bridged the gap in a lot of ways between the 'average' and 'pro' musician being able to access nice digital sounds on reasonably priced hardware. I'm a genuine PC guy (Ableton Live, Reaper) but get a lot of mileage out of a second gen iPad 2 - specifically Propellerhead's Figure software, which is flat out fantastic. I've made an entire EP in GarageBand while traveling.
Sure, some devices won't work with iOS devices (my LPK25 will run on the iPad 2's port via camera connector USB dongle, but the MPD8 won't), and yet some are outstanding (looking at the Line6 / iK media products for guitarists and amp simulation). There's so much of a head start with iOS that honestly I kind of think there's little to no point in Android or MS trying to play catch up in the mobile sphere. MS has a winner with the Surface line because they run real software, but in the 'pure' mobile space, iOS has all the goods in my opinion.
iOS inhereted a strong foundation for this as OSX has a history of strong support for this sort of content creation. Linux does not.
Such as? Remember we're talking Linux in the 2005-era (~2.6.10-ish), not Linux today. JACK was still fairly new at that time. ALSA was still mostly broken at the time. Most stuff still wanted OSS. Basic "does sound come out of the speakers?" was often broken, and god help you if you wanted two things to play audio at the same time.
And you still need to solve the driver problem, which JACK, etc... don't help with at all.
Um. If we're talking about 2005, we're talking about Android.
If we're talking about Android, we're talking about brand-spanking new hardware on a brand-spanking new phone.
This means that the audio driver didn't exist, which means that it was being written from scratch. This makes the "driver problem" a non-issue.
> Remember we're talking Linux in the 2005-era (~2.6.10-ish), not Linux today. ... ALSA was still mostly broken at the time. Most stuff still wanted OSS.
I never tried to do any "pro audio", [0] but that's not how I remember it. ALSA worked just fine. Anything that wanted OSS worked well enough with ALSA's OSS-compat API. I can't remember if the software mixer was around or not at the time, but I remember that when it did arrive, it eliminated that problem with single-stream sound cards.
[0] But -as I remember it- most folks doing pro audio used specialized, standalone hardware for it back then.
However, none of the folks I knew at the time who doing professional recording were using PCs or Macs to do the recording and mixing, they were using standalone gear.
It's entirely possible that the folks I knew were not a representative sampling of all sound engineers.
Yes, Android's designers had a very different mentality from the traditional RTOS-for-a-handset designers. But that does not mean they had a "data center' mindset. Indeed one of the most common error or people who do have a "data center mindset" is that they load up on libraries that do various useful things for their Android development and then blow out the method and heap limits.
Understanding Android requires a unique design and implementation mentality that is neither RTOS oriented, nor "small *nix in a phone" oriented, nor server Java oriented. It's a clever multi-processing managed language runtime with aggressive memory recovery that's capable of running many managed runtime processes, none of which can hog memory globally.
Without having been designed for real time audio from the beginning, retrofitting that capability was hard to do. Apple made a lot of hay out of lack of real time audio processing. But only in one domain of apps. These are the kinds of trade-offs that get products out the door.
I don't care about the audio latency this article is talking about and I never used to have this problem either. But recently, maybe just since I flashed latest CyanogenMod my phone sometimes responds instantly, but sometimes doesn't respond to the headphone button press until I press the power button or wait 30s+. It's incredibly annoying. I'm using CM on a OnePlus One with Xiaomi Piston 3 headphones.
Considering the power button works, I suspect this may have something to do with sleep states?
Similarly stupid response when unplugging headphones:
1) Oh, he unplugged the headphones!
2) Let's play that audio out of the speakers instead! At whatever volume the speakers happened to be at last time he used them!
3) And keep playing it for a half second or so!
4) I guess he might have wanted it to stop when the headphones came out? Fiiiiine.
Google music does have the habit starting music playback when connecting to bluetooth regardless if the app is running or it's been a week since connecting to bluetooth or the phone has been rebooted since connecting. It just spams my car stereo with music AFAICT.
Another audio bug: when Google Maps gives audio directions, it pauses my podcasts. Half the time it resumes playing afterward, half the time it doesn't. I can find no pattern to this.
PocketCasts and Google Maps are both pretty well polished programs, and I have no idea whether I should blame one of them or blame the OS. Ah well. At least I got an SD card slot instead of paying $200 for 48GB more storage.
Phone notices headphones are unplugged, phone switches to blasting over the speakers for a moment before pausing it. Really stupid problem.
I should probably wipe the thing and start fresh.
I think the issue you're describing is something that is happening in AudioFlinger and processing the input events.
All over the place, apps and devices get faster and faster, and take longer and longer to actually do what you tell them to do.
</rant>
The manufacturer could get even smarter and use something like CRIU [0], along with opportunistic hybrid suspend [1] to dramatically speed up start times.
[1] Which would work like this: A little while after you power your TV off, it suspends to RAM, and disk. If you power your TV on, the system TV restores its state from RAM. If AC power goes out, when AC power is restored and stable, the TV performs a background operation that uses the on-disk suspend image to get back to its hybrid suspend state.
What?
No feature works out of the box. Engineering effort has to be expended to make something work when designing and building any new product. Did you maybe misphrase what you were trying to say?
> It's worth noting here that nobody buys TVs based on how quickly they turn on.
[citation needed]
A big part of my criteria for purchasing a display device is device and control responsiveness.
My non-technical family members who care about TV made "time between the time you turn the thing on and time it becomes usable" their #3 priority when purchasing their second "smart TV". [0]
It's pretty clear that many (most?) techies are really disappointed in how shitty, slow, and poorly designed most "smart TVs" are.
[0] #1 Picture quality. #2 Overall value per dollar.
This makes me wish BeOS had survived.
My 2012 MBA boots in 2 seconds from power on to login. And it's a pretty humble device that is almost 3.5 years old.
Software developers are to blame. An accumulation of people calling code they don't understand, ignoring constant factors, depending on Moore's law, sacrificing performance to save a little bit of effort.
I feel like responsiveness, low-latency and high-performance especially when it comes to UI stuff is where it comes down to taste. Frankly most people are unlikely to be bothered by a laggy UI. This applies to programmers as well. IMO, the only people I would rely upon to write performant (consumer oriented) code primarily exist only in the games industry. I feel like being forced to "do more" on a hardware platform that is going to be static over the course of the next five years self-selects people who are good at saving cycles and extracting performance.
You can't expect a hardware company to write good software - All smart TV's and such are going to suck big time. The IoT is going to be a goldmine with outdated libraries, unpatched vulnerabilities and other goodies. But you should expect more from professional software developers. Unfortunately, I can't think of the a single Google product that I used that was "highly responsive". Certainly the ones that I use regularly - Chrome, Gmail, Maps, Search have consistently all gone downhill in terms of memory bloat, cpu consumption and responsiveness. I haven't used Android recently so I can't comment, but the last time I had an android phone it got replaced with an iPhone 4S within a week.
curious what phone and headphone type you guys use?
I use a Nexus 6 with a single button headphones.
I know on some device I have used there is a delay in interpreting a single press because the device has a longer window to allow for a double press. Although this is never much more than a second or two.
I will try to reproduce the issue ... I am pessimistic though, since I use bluetooth headphones daily without any issue.
Even though the Linux kernel is less than ideal for audio, it can still provide a latency low enough for music, ie. 10 ms or less.
This is an unbearable situation. There are lots of interesting music apps on the iOS that I would like to use, but only a handful are available on Android. And even if they are available, they are not necessarily usable.
There is a diagram lower down in the article with the components (ALSA, audioflinger etc).
http://www.androidpolice.com/wp-content/uploads/2015/11/nexu...
Of course many phone vendors use ALSA to implement Android HAL layer. It seems that Samsung uses its own sound drivers and therefore they have small audio latency compared to other Android vendors.
So this is one of the "good" Android audio stacks, getting 36ms latency. But half of that is "wasted" in user space, in AudioFlinger as well as the app ring buffer (which seems pretty big to me). I'd like to see the comparison to a "bad" Android device.
Doing some internet searches, I found someone who had plugged in PulseAudio to an Android phone and got 20 ms latency instead of the 170+ ms latency from AudioFlinger on the same device [0]. That's pretty huge.
Ideally, there should be no user space middleware component processing audio in between, this could half the latency (according to the diagram), bringing the latency to acceptable levels. But ALSA isn't really intended for routing audio on the fly conveniently (e.g. when plugging/unplugging headphones), so there is some room for improvement too.
Overall, the situation is pretty sad. Linux can do better than that but a modern multimedia device (desktop or mobile) has several audio outputs and hot-plugging, which makes the situation rather difficult using ALSA only. And the userspace audio components just aren't great.
http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinger-f...
tl;dr: it's easier to increase buffer sizes to fix underruns than to fix driver foo randomly disabling interrupts for too long.
That, along with not having fast-paths to bypass large parts of the stack (mixer, resampler, audio effects, etc...), and bam, high latency.
Nonetheless, I avoid Apple products on principle, and also enjoy playing with audio stuff (so much so that I started an audiovisual company a few years ago, which I only recently extricated myself from, leaving it in the hands of one of the co-founders), so I'm pleased to see the issue getting attention. I'd love to be able to make real music on my tablet. I have a Gameboy that I use for "on the train" music tinkering, because it has an immediacy that is lost in most modern devices (not merely latency...just the general composition process on something so simple it's impossible to get bogged down in bullshit like selecting the perfect snare sample or something). It's be cool to be able use the Nexus 7 for that purpose.
It probably provides the best insight to the problem Google engineers were and probably is still facing in regards to high performance audio on Android. Even when the latency is brought down, there's still other audio related issues/glitches (like buffer underruns).
Even now that Android is measurably better than it was previously, things like buffer underruns can still affect the overall usability. Increasing the buffer (to fix the underruns) render the device less suitable for such things as using it with external MIDI controller. You can see from Superpowered's audio latency measurements:
http://superpowered.com/latency#cta
that the buffer for iOS is half that of the best Android (the dual-core Nexus 9)- 64 instead of 128 for Android.
I
That being said, it's not clear whether the improved latency on newer platforms can be achieve without using Android NDK. I haven't used it, but I know Android 4.0+ supports OpenSL through NDK, and I'm guessing that if you want low latency you need OpenSL. By skimming through the docs, it looks like that's the case, isn't that so? http://source.android.com/devices/audio/latency_app.html
So, maybe it does solve the latency issue, and I would gladly use NDK because I like C++ stuff, but I think most developers wouldn't want to do it.
"How hard can it be?"
At the end of a year, imagine someone who has spent a year in the Alaska wilderness, strangled three grizzly bears with his bear hands, lived on berries and nuts and fish caught in frigid glacial streams, and generally been up and down the audio stack in the OS and a couple pieces of firmware, involving audios APIs, a USB software stack that we re-wrote, a close examination of USB hardware and how chipsets are fucked up, and how exactly I2C microphones really work. My wife described me as "broken". It took a while to recover.
How hard can it be? Isochronous, low-latency audio can be pretty hard.
It currently runs on iOS and OSX.
Initially I thought about doing an Android version as the code is all in C++ and it would be easy to port, but if the latency is as bad as described here I might not bother.
You heard it here first folks, in a few years Apple is going to come out with the revolutionary "Cochlea sound", which has such a small delay your ears are unable to perceive that there is any delay at all (while listening from 10-12 inches from your ears of course)!
It will be magical and delight us, and something we never realised we needed until we were told we did. It will become the new standard and a household name, and we'll all have to catch ourselves when we say it and try to use some obnoxious generic term instead (LoALPS perhaps: Low Audio Latency Per Sound).