An update on Android's audio latency
android-developers.googleblog.com
android-developers.googleblog.com
(Disclosure: I work for Google, speaking only for myself)
Edit: I've been thinking about this and a touch-to-tone latency below 10ms seems imposible, even with the 120hz digitizer on newer iPhones. I would be very interested in seeing some experimental data as well.
[0] https://juce.com/discover/stories/Mobile%20performance%20ind...
The audio processing latency seems to be single digit on both systems.
In any case, I don't think these differences are big enough to argue that "Android can't be used for instruments".
Well financed developers like ROLI already have instruments on android. I think the incredible instruments available on iOS are also there because the App Store promise actually kind of works for them. They write a great application, upload it to server and people buy it for more than 2.99$, upfront. And unlike VSTs, no one even pirates it. The general idea/stereotype is that no one wants that on android. An open source port of VCV Rack would be feasable and wonderful though. That's the only free instrument I have installed on my music iPad.
Edit: nevermind, that was not free either.
We should not forget that our brain is great at adjusting to latency. In the old days, when I was excessively playing Quake 3 Team Arena it took me just a couple of minutes to adjust to network latency depending on the server I was connected to.
Professional musicians do this constantly and are much better at that.
There’s this video around that puts people in a VR helmet playing A/V with a delay, even the lowest setting completely destroys any form of efficiency.
Anything below 60Hz in VR has a significantly increased chance of giving motion sickness due to latency.
Playing Super Mario Bros in an emulator is strangely subtly harder than on the real NES hardware, just because of a slight lag added (can’t recal on NES but on SNES buttons literally bit-flip the RAM, hard to beat that). You can train all you want on an emulator, when you get back to the real thing it’s obvious it was just so crippling.
Try Guitar Hero / Rock Band with various audio / video latency settings. 10ms is obviously impactful, anything higher borderlines on unplayable, and even 5ms has a subtle impact increasing missing notes.
As for pro musicians, they absolutely loathe latency. This is why bassists (tempo) / drummers (rhythm) / conductors (tempo) exist: they serve as a single source of truth to sync on. In fact, we’re so sensitive to latency that the slightest deviation is unbearable, hence why following them put you back in sync. Same for effect pedals, amps, feedbacks... any latency is destroying.
Try playing on Zoom, it’s absolutely impossible (unless you assume latency is fixed, which it is not, and play in a tempo matching latency exactly, and follow the lead but time-warp your play by one beat, which I imagine is possible as a sort of canon but absolutely horrendous) unless you completely ignore the other part and find a common time source to sync upon.
Now you mention Quake 3, and it might just be you were severely misled by several critical innovations that game introduced, like network prediction and frame interpolation, negating most of the practical effects of latency.
The fact that the effects are laggy doesn’t mean you need to sync on them, you’re mostly listening whether the sound envelope is correct (e.g adjust for overshoot/undershoot a bend). Checking the output of effects only matters at timescales of whole beats/measures in terms of how you have to react, definitely not like keeping up with tempo, which has to be precise across the band.
You don’t get delay on drums, and exceedingly rarely so on bass, which is what you have to react agains efficiently, plus the instrument you’re playing provides its own subtle and not-so-subtle haptic feedback. On a guitar, this means the moment you pluck the chord is based on your internal clock synced by listening to the tempo giver (usually bass).
Source: amateur musician myself, wife is pro.
Lag comes from video. If you do not use FPGA emulators like Mister the video delay is enormous.
I don't understand what you are calling 10ms lag. Have you measured the delay between your computer and your screen? It is way bigger than that, in any computer or NES console(console-TV).
Are you maybe referring to 10ms NETWORK packet delay? All TVs or projectors use buffers, and it takes time for information to travel around serial cables.
Network delay is inconsistent, but electronic delay is not.
That’s exactly what I said ;) By virtue of being a different platform, emulation introduces a lag that is just not present on the real hardware (and can’t be lower by virtue of literally being a wire to memory). USB alone often introduces an astoundingly high minimal floor on latency.
> I don't understand what you are calling 10ms lag. Have you measured the delay between your computer and your screen? It is way bigger than that, in any computer or NES console(console-TV).
Yes I have measured it (on all my devices, accurately so) in order to minimise it as much as possible (I can work with but can’t stand latency, even in a terminal).
Rock Band, which I used to be an avid player of, allows you to adjust a relative delay (audio and video separately), timewarping A/V so that all three are perfectly in sync relative to real-world input.
> Network delay is inconsistent, but electronic delay is not.
I agree (and agree to buffers as well), but by and large any modern device is far removed from being just electronic, from USB (USB 2.0 has a 125us polling loop for multiplexing, USB 3 is point to point so can fare much better at 30us, but even then there’s kernel context switch to read and process that data) to HDMI (packet based with loads of multiplexing), many things end up being surprisingly subject to soft-real-time firmware/software before it hits the display itself. Hard real time stuff probably exists because the specs now allow for it but good luck finding that in consumer space where it’s much easier (read: cheaper) to push a best effort implementation.
But my point isn’t about the minutiae of lag source, it’s that the human brain is severely impacted by lag, can adjust to some extent but the cost is paid heavily at multiple levels and with various efficacy crippling consequences; and that Q3 and subsequent games have increasing numbers of facilities to create the illusion that everything happens in sync so that we’re not so impacted by it.
The brain can compensate for lag/latency up to a point. Church organ players often have to deal with >100ms of delay between pressing a key and hearing the note produced through those big pipes.
If you have an app which produces a sound when you tap on the screen, and there's a delay of say 70ms between tapping and sound being produced, it's easily possible to play in time with an external clock source, like a metronome, as long as the delay is fixed.
In the 1968 paper "Response time in man-computer conversational transactions" actions which occurred within 100ms the human input were perceived as instantaneous. In my experience this holds true with musical instrument apps, although one must first learn how much latency is present in order to compensate for it when synchronising with other clock sources.
I think there is a subtle difference between fps (avg) and latency. I would say that a 'stable' latency is always better than a high fps.
> Try Guitar Hero / Rock Band with various audio / video latency settings. 10ms is obviously impactful, anything higher borderlines on unplayable, and even 5ms has a subtle impact increasing missing notes.
Do you mean that you set an additional latency for both video and audio? Or you delay only the audio by 10ms?
Yup, I only mentioned 60Hz, because 16.667 ms is more obvious WRT latency but less known.
> Do you mean that you set an additional latency for both video and audio? Or you delay only the audio by 10ms?
No, RB allows you to set up the audio and video latencies, and since the track is fully known, it’s able to apply negative latency to everything but the input (which, being physical user input, is the only thing that can’t be altered).
So you set up input->audio latency and input->video latency values and the game will internally play them ahead, so that they appear exactly in sync in the physical world.
Zoom's latency isn't fixed, so that won't work. On the other hand, it is possible to do something like this over the internet by measuring latency and then adjusting for it: www.jefftk.com/p/bucket-brigade-singing
A physical piano with a 1.2" hammer blow distance played at soft velocity (~2.2mph), assuming distance of closest resonating surface to ear is half meter, would have around 35ms latency.
The latency(different type I believe) of Spotify applications and even YouTube applications is pretty bad compared to an Apple device. Sometimes the audio just fails to respond for half a second or so. This is not an interface bug from my experience.
My impression is that Android has more latency variance than iOS. If that's true then it's important to be clear what the reported numbers represent.
Also from talking to an engineer who worked on earlier generations of iphones, who said that every team had a latency budget that they could not exceed.
Sad to see Apple take such a direction - musicians appreciated having proper hardware/software and Apple definitely made money on all the paid content. Why let this arrangement fall by the wayside?
What makes you believe this?
Sometimes I wonder if these increased levels of abstraction and even digitization itself (or at least the Internet-ization of things) is a mistake for human-machine interface. Watching drone racing pilots use analog controls and analog video headsets in spite of the potato resolution because WiFi/digital/etc just add too much latency is kind of eye opening and makes you sort of reconsider the last couple decades. (Improvements have been made so digital isn’t so high latency and VR has served to address a lot of these latency concerns, but it’s still kind of interesting...)
With any android device openly spying on you while locking down all the open source apps you might want to install.
Not that it counts for much in the grand scheme of things, seeing as 99% of users will never install a custom rom.
I'm running my own Android ROM without cheating Safetynet and I still have access to everything I use using their original unmodified uncheated app: Netflix, Spotify, my banks, all my connected objects (withings, Philips hue, bluetens, neato), or some other random apps.
There are things that don't work, it's true. The only I have witnessed are Pokemon Go, and NFC payment apps (like g pay). This is very far from the "go all floss RMS-style"
Of course you can always pick up a Librem 5 or a Pinephone, but I'd argue it comes with a lot of drawbacks.
Of course Android is already world's better than desktop, where 200ms is seemingly closer to average than an outlier.
It is also kind of cool that perusing that chart, it seems headsets connected to Android have, on average, lower latency than when connected to iOS!
https://www.rtings.com/headphones/tests/connectivity/bluetoo...
Doesn't change the fact that the same device can have a 3x latency difference between platforms with the same codec.
Google went and developed VP9 as a royalty free codec in the AV space, they should do something similar for Bluetooth. Right now if I'm on a video call with someone and we both have BT headsets, it is very possible that there is a 500ms audio latency!
If there are multiple parties on the call, this breaks down, because if two people want to speak, they both start talking and don't realize it for a little while.
On video calls this also breaks down, because either video and audio get out of sync (like with zoom) or you get odd artifacts in either the video or audio as they are kept in sync.
It turns out that the device that's meant for phone calls has to, first and foremost, make sure that there are no dropouts, crackles and other buffer underrun articles. Latency comes afterwards.
Is it intentional to ensure there's plenty of buffer so audio packets can be re-sent when they fail, and the audio is seamless? In other words, is 100-200ms just inherent at that frequency with expected interference?
Or is it actually just terribly designed?
I don't immediately see the relationship however -- is there anything you could point me to, to explain the mechanism?
You're mixing up two different things. Transport latency of WiFi is 2-3ms, but for bluetooth you're suddently talking about a transport, codec, receive buffer and playback code latency. That's not an apple-to-apple comparison.
If you want comparable numbers, measure the latency of audio being streamed from one computer to another via PulseAudio and you'll quickly see that it's more than 2-3ms. And there's a reason for that: most people strongly prefer their music to not stutter and work properly with all kinds of headphones instead of having absolutely no latency. Most people prefer to shove their phone in a separate room, back pocket and walk around RF saturated areas and still listen to music without interruption.
Hence the manufacturers err on the side of buffering some audio so it's more resilient to connection dropouts. You wifi, after all, doesn't need to move around busy highways and public transport systems.
On Linux, I have to restart pulseaudio after using Bluetooth audio, otherwise retroarch is unplayable.
BTW where are you getting numbers for 200ms latency being average on desktop? My understanding was that latency is typically around 40-50ms on desktop if you are using analog speakers.
Of course, it probably makes the experience just worse if the delay is silly long..
And there is no "touch" on PC generally speaking.. input latency would be on mouse or keyboard.
I think you're talking different thing from the topic.
As the article says, there is room for improvement, so I guess it's good to see that the long term goal is 10ms round trip.
But then the video takes a strange turn and starts talking about "Stability" of mac vs PC, and I had to turn it off. It made the argument that "I haven't seen any professional producers use anything except macs" as its main argument.
It would have been nice to hear some more technical information like mentioning core-audio, or something else.
Regarding the article itself. I'm excited that Android is target less than 20 ms round-trip latency. I'm hoping that this work can be brought back to the Linux Desktop. Messing around with Jack and dealing with xruns is something I wish I could stop doing.
Yeah that is both totally irrelevant to the topic at hand and also a bit strange. I have only seen PCs in the pro studios I have used but that doesn't make me believe nothing else is used.
Have you looked at Pipewire?
That no producers are using windows PCs is an obviously false claim. There are DAWs that don't even run on mac. There are also live production tools that don't, like Notch, so there must be someone running windows on stage, anyway.
That macs are more stable in day-to-day operation is not really a bold claim though.
It absolutely is. Do you have any data to back up the claim? Anecdotally windows 10's stability seems vastly superior, especially with every release of MacOS seemingly regressing further and further.
You won't get a sudden I/O or CPU load during important operations for example.
I used to use windows, for a decade, and certainly never had any experience like that.
Updates alone put my old macbook pro at lower availability than any of my windows or Linux systems since MacOS manages to have a uniquely slow update system. I don't get how major MacOS updates manages to be several times slower than fresh installs of Windows or Linux.
More to the point, one of PulseAudio's claims was that they consumed less CPU than Android's audio pipeline. With the recent hyped PipeWire, I wonder how that compares to Android's Oboe.
I haven't played with it myself (it's been a few years since I was actively involved in Android audio latency), but from what I've seen, AAudio is pretty good, probably close to what the hardware is capable of. I'd very much encourage people to do empirical measurements against, for example, PipeWire (which also looks good). Do keep in mind that constraints are different on mobile, and things like power management can and do get in the way of down-to-the-wire latency.
A rule of thumb from reading linux audio mailing lists but which probably works generally-- completely disregard any claim of measurement of latency that isn't prefixed.
For example, the Google blog labels the Y-axis with "round-trip time in milliseconds." They're clearly and explicitly measuring round-trip latency. They then piggy-back on that well-understood concept to introduce the concept of "tap-to-tone latency" and show measurements for that.
Almost to the case, the times I've read a back-and-forth conversation where the word "latency" is used unprefixed, the commenters might as well be describing the warmth of vinyl recordings. Often they are simply describing an integer they typed into a widget or config file.
On several occasions I've witnessed users insert Jack between a single application and ALSA because "that's what the professionals use." These are knowledgeable users who want as little latency between the system and their ear as possible.
Hearing any claim that pulseaudio uses less cpu than some alternative makes me blink. For years I could not use pulseaudio while playing games, because it would eat 99% cpu and I only had one cpu core. Things are better now that I have eight cores, but I don't know if that's due to the extra cores or if pulseaudio has improved.
I tried jackd, but the documentation is overwhelming.
I remember ESD and OSS working just fine, but I was young and maybe I was just happy to get audio at all. I grew up with blips and bleeps coming out of the PC speaker.
The improvement is great, but I think the audio professionals jumped ship long ago to Apple, and they're not going back. I know of several music professionals plugging into their Macbooks or iPads as part of the pre-amp, but I don't know if any using Android. Google sat on this for way too long while Apple courted the creative professionals, and for any that have already made the upfront buy-in for Apple hardware, I don't see them taking the risk of switching ecosystems to save a few dollars for performance that still won't match what they're used to.
Disclaimer: Googler, but opinions are my own.
I think Google is doing a better job with their Pixel devices. I honestly hope they try to lead the Android industry more with them.
It absolutely does, in part at least, fall on Google to establish how the Android label can be used. Their decision has (apparently) been that it can be used by anyone even if their devices become (practically) obsolete in just a year or two from various perspectives (not in an absolutely obsolete sense, but a lack of many/most/any updates is a crucial element of currency for mobile devices and hastens obsolescence). That's on Google, not the OEMs who take advantage of it.
I'd like to say the OEMs should do better, but the market has clearly convinced them that they can be profitable and sell soon-to-be-obsolete devices.
I won't support Android 11 and later in any of my apps. If someone has an issue with that, they're free to send me a device getting the Android Preview releases for the version they want me to support. I treat Android just like iOS now: Want support? buy me a devkit.
In the past it was possible for me to just build AOSP master from source and test with that, but 3.0 was the first release breaking that assumption and after 5.0 the assumption was broken entirely. Now I just won't support it anymore.
Reaper (free during COVID, $60 discounted license otherwise) + free plugins is pretty great though without going the unlicensed route.
Reaper also runs on Linux, natively and in WINE.
Just because the pros have merged towards a single brand doesn't mean that the only other smartphone monopoly won't benefit greatly from offering other high quality solutions.
If Google cared about niches, they wouldn't have such a large graveyard of dead products - some of which were profitable and some of which were widely loved.
Ever since Google killed Reader I have no faith in them to ever maintain anything that isn't directly selling ads.
Also if it's part of Android, I bet it will unravel a bunch of hidden bugs that haven't been exposed before. Like I said, Android is already buggy as fuck. My Pixel 4a freezes/crashes regularly and Google told me (paraphrasing) "it just does that."
Once the tech is capable the software will come.
It might be a niche case, but it's really frustrating using the various sleep-related audio apps at night with earphones and not being able to select a minimum volume that is both loud enough to hear but not so loud it keeps you awake (example: Audible with a sleep timer).
I'm sure there are other uses for the finer control as well.
Have you tried pestering Audible? Would probably take a while, but it feels like the kind of thing that would be hard to "un-justify" once you started arguing for it.
Obviously any major app supplier may be reductant to invest significant dev time to offer a workaround for an OS limitation (and Audible is just an example), but yes it's worth raising I suppose, thanks.
I'm not even sure if it would be straightforward with a rooted device either.
I don't know if there are any programmatic system-wide ways to do the same (every app can implement it in their software mixer though). I haven't had great experience with third-party solutions that obviously cannot integrate with the OS at the same level.
I had the impression the problem was physical limitations with the dial. With fully digital controls, it seems like this wouldn't be an issue. But obviously you need more than the handful of steps the volume buttons on the side of the phone give you.
10ms would start to be useful for music, but I wouldn't call it good until it was below 2ms, which is what I run my jackd setup (which requires linux-rt) at.
I'm afraid we won't see good latency until Android moves away from Linux. The good news is that this is bound to happen, sooner or latter, as Fuchsia exists.
40ms is 40 feet. Bands play with that kind of separation all the time.
Why you would be considering doing that on an android though I'm not sure. Though I guess it may be a reinforcing problem, there isn't an ecosystem because the tech isn't there.
Shortcutting this latency is one of many reasons musicians wear earplugs on stage.
Here's a corny video from Google about audio latency and pops from several years ago.
I read through the whole article and that's the most detail given about any testing methodology; I don't see any more description of how these numbers are being measured.
My rule of thumb for computer speed measurements and efficiency has always been in terms of CPU instructions; the goal of 10ms, while seeming very tiny, is still roughly ten million instructions of a 1GHz CPU. The other numbers mentioned are proportionally more, which then makes one wonder where all the time is being spent.
Relatedly, I've actually written audio drivers (HD audio for Windows 3.x/9x, long story but I won't digress...) and the output latency measured from the app writing a buffer to the signal appearing at the headphone jack is probably in the dozens to hundreds of microseconds range (thousands of CPU instructions, including virtualisation overhead) at most --- but those OSs have extremely thin abstraction layers, and the call stack between the app and writing to the hardware registers is very shallow.
The fact that newer hardware/software seems to have more audio latency reminds me of this: https://danluu.com/input-lag/
The details usually aren’t that interesting. What you can do is record an input signal, pass the signal through to the output, and wire the output back in to another input.
You end up with two copies of the input signal, one delayed by the total round-trip time relative to the other. It’s fairly easy.
I think we can give the benefit of the doubt and assume they are not trying to actively deceive.
It's a simple test: produce a tone (either on built-in speaker or over wired headphones if using a loopback dongle), measure the time it takes for that tone to reach the audio input.
For devices which I didn't have access to (our team has a limited number of test devices) I used the figures from Superpowered, but with some assumptions/rules:
#1 If AAudio was available on the device I used the measurements from that rather than OpenSL ES #2 No measurements from custom ROMs #3 I used the measurements from the latest version of Android which the OEM had released for that device. For example, if a device was originally released with Lollipop but it was possible to upgrade to Marshmallow then I used the figures for Marshmallow.
For touch screen latency measurements I used a WALT device (https://github.com/google/walt)
# audio latency
From your comment and the original "An update on Android's audio latency" article, there seems to be 2 different ways to calculate "audio latency".
* 1. play a well-known sound (a tone), listen for it in real-time
IIUC, the exact operation of this would be something like this:
open recorder
loop:
timestamp1 <- getclock()
play audio file
keep listening to input frames until detecting audio file signature
timestamp2 <- getclock()
sample := timestamp2 - timestamp1
Operation should be as follows:
open recorder
play audio file
loop:
keep listening to input frames for 1-2 seconds
# analyze captured file
look for 2x consecutive appearances of the original audio file
sample := timestamp2 - timestamp1
# downlink/uplink latency
Many of the latency measurements break the latency down between audio downlink and uplink.
Q2: How is this measured?
The android doc discusses using a GPIO to have a zero-latency signal. You swap the signal on a well-known GPIO, and then play a well-known audio file. You then connect the GPIO to a speaker, and measure the latency between the GPIO-fed speaker, and the actual device speaker. As an optimization, you plug both the GPIO and the audio jack that feeds the actual device speaker into an oscilloscope, and get the distance there. Is this how audio downlink latency is measured?
Q3: If this is how downlink latency is measured, how do you implement it in prod devices (no access to the board)? Is this something that can be simulated using the usb-c plug?
# tools
The android doc/some googling points to several tools. I found oboetester, splatency, drrickorang, and google walt. I tested all but the walt one.
The first interesting thing is that oboetest, drrickorang, and walt seem to need an external jack-based dongle.
Q3: Why is this needed for? The audio latency approaches mentioned above should only need a speaker and a mic.
The experience has not been great for any of the tools. Operationally, I'd like to script these tests. Instead, I need to click on GUIs. Also, all the tools I tested seem to fail often, so I have to repeat the experiments multiple times. Also, I miss better documentation on exactly what they do.
Thanks for any help
edit: s/markdown/formatdoc/g
Thanks for your interest in the details of Android Latency measurement techniques.
Some of the measurements were collected by third parties. We cannot describe their techniques but they are probably similar to ours.
We use OboeTester to measure latency. You can find a description of how to measure Tap-to-Tone Latency and Round Trip Latency in this doc: https://github.com/google/oboe/blob/master/apps/OboeTester/d...
We do not use the Larsen Effect any more because it was too sensitive to variations in gain. We now use a random encoded bit stream that sounds like a short noise burst. We can get a better correlation peak with that signal.
> Many of the latency measurements break the latency down between > audio downlink and uplink.
It is very hard to separate the input and output latency without special hardware (like the WALT device). You can measure combined input+output latency using a loopback test. Input latency tends to be much lower than output latency. This is because, when the full duplex stream is stable, the input buffer is close to empty and the output buffer is close to full. Then, if there is a preemption, the input buffer fills up and the output buffer drains, providing glitch protection.
You can measure touch+output latency using tap-to-tone. The screen touch latency is about 15-30 msec. If you use a hardware MIDI controller (like a keyboard or drum pad) instead of tapping the touch screen then you can get a “touch” latency of about 1 msec (MIDI is a lightweight and low latency protocol). Then you can get a better estimate of just the output latency.
> The android doc discusses using a GPIO to have a zero-latency signal.
We don’t normally use that technique because it requires special hardware.
> I found oboetester, splatency, drrickorang, and google walt.
Our group supports OboeTester.
> The first interesting thing is that oboetest, drrickorang, and walt seem to need an external jack-based dongle.
The lowest latency path is usually over the headphone jack (either through 3.5mm or USB dongle). To test this path you do need a "jack-based dongle" aka loopback adapter.
The reason why wired headphones usually give you lower latency is that OEMs often introduce digital signal processing for the speaker to improve the acoustics/quality, which can introduce additional latency. Side note: this is why you will often see "best with headphones" on games and music apps.
That said, OboeTester will work over the speakers and mic in a quiet room. That is because the new random bit technique is more robust.
> I'd like to script these tests.
OboeTester can be scripted. We use it for continuous integration testing. https://github.com/google/oboe/blob/master/apps/OboeTester/d...
* I find interesting that you are getting better correlations when you add a well-known noise burst instead of a less chaotic signal (e.g. a tone or a chirp). In retrospect, it makes sense
* for the uplink/downlink breakup, I see in the Usage.md file in oboetester that you're isolating the downlink measurement with the tap-to-tone experiment. For this experiment, the doc suggests to use (a) the jack to avoid the speaker processing extra latency, and (b) a USB-MIDI input device to replace the touch screen latency (15-30 ms).
2 questions here:
* Q1: I assume that, if instead of the jack, you use a usb-c audio adapter accessory mode, there should be no extra latency either, right? (I'm using late pixel phones)
* Q2: which device are you using for the USB-MIDI input?
Thanks again!
edit: s/markdown/formatdoc/g
That's correct, by using either the 3.5mm jack, or USB-C adapter you won't incur any additional latency introduced by DSP to improve the speaker acoustics.
However, the USB path typically does have a few ms higher latency than the 3.5mm jack path.
> * Q2: which device are you using for the USB-MIDI input?
We've used a variety of devices and found the latency differences to be negligible. At the moment I test with an old AKAI LPK25.
Uh.... It needs to be at least 10 times faster to claim realtime use
Samsung
Redmi
Oppo
Huawei
Vivo
Which probably account for the majority of Android smartphone sales.
I'll take it.