Sampling: What Nyquist didn't say, and what to do about it (2018) [pdf]
wescottdesign.com
wescottdesign.com
Why FIR? It had a super sharp cutoff and of course linear phase. If I'd known about Bessel filters at the time I'd have tried those out. Live and learn and ship it.
Nifty story, just wanted to validate my understanding.
The dense interview question I use to assess this area of knowledge: "How do you choose the stopband attenuation of a filter?" You can assess a lot from the interviewee's response. Stopband attenuation, with respective to the input signal magnitude at the stopband frequency, or start of the stopband, is the most relevant term for determining the magnitude of noise in the sampled signal. And that is the upper bound on the performance of the downstream algorithm.
That is, to expect a specification for the desired S/N (signal-to-noise) ratio and bandwidth, and/or bit rate at which the downstream algorithm is expected to work. This doesn't have to be a perfect estimate/evaluation/measurement, but it helps to get the ballpark amount of computation power, delay, etc. needed to actually make the system work. All too often I see people react to a noisy signal, by "putting a filter on it," without knowing how much attenuation/gain/phase shift, etc. are actually needed to make the system work acceptably.
E.g. You don't want to spend too long designing the perfect filter... And you want to know if there's any hope at all cleaning up the signal to make use of it!
Good luck with your filtering projects!
https://en.wikipedia.org/wiki/Undersampling#:~:text=In%20sig....
I mostly deal with this stuff in the realm of audio where the matter of analog vs digital rages regularly. The reason one hears "Nyquist says" so often from people defending digital audio is that the pro-analog people are imagining stair-stepped signals in which the information that is "lost" results in a noticeable degradation in sound quality. This (11-year-old) video is the gold standard for addressing this concern:
https://www.youtube.com/watch?v=cIQ9IXSUzuM
Curiously, no one ever seems to mention the actual problems (like aliasing).
We have periodic steady state analysis to simulate sampled systems and harmonic transfer functions by the way. It is extremely useful especially to stimulate noise aliasing.
from "24/192 Music Downloads ...and why they make no sense" (2012) [1]
Audiophiles demand streaming services to provide 192/24 because they see the music being originally mastered at high sample rates, and from that conclude that listening to 48/16 is a loss of "original recorded quality".
I can totally envision audiophiles picking up the Wescott's article and using it as "scientific" argument to distribute more Hi-Res music, and some half-technical manager buying that. They won't even read it.
Are human ears even capable of hearing 24 bits of dynamic range?
We know that 2^10 ≈ 10^3, so 2^24 ≈ 10^(3/10*24) = 10^7.2. This 7.2 is about amplitude, but what we hear is power which is quadratic in amplitude, so take a factor of 2, and you're talking 14.4 bel, or 144 decibel for 24 bit (and 2/3 of that, 96 db, for 16 bit).
(In reality it's a bit more complicated, but that's a good rough guide, see https://en.wikipedia.org/wiki/Audio_bit_depth#Quantization)
As far as I can tell, though, it doesn't mention what may be the most important reason (especially to the folks here at hackernews): resampling and processing.
This is why professional grade audio processing operates at a sample rate many multiples higher than human hearing. It's not because of the quality difference between, say, 192 and 96 kHz, but rather if you're resampling or iterating a process dozens of times at those rates, eventually artifacts will form and make their way into the range of human hearing (20 kHz).
Which makes sense I suppose.
It's worth noting it's a tradeoff, even in pure processing: almost all non-linear transfer functions will create an infinite number of overtones, so it's impossible to avoid aliasing completely: you can only reduce them to some threshold which is acceptable to the application.
Your signal contains all kinds of frequencies: Those you care about and those you don't want in your recording. You can't just sample at the Nyquist rate of the interesting frequency and expect all the other frequencies to vanish. They will mess with the frequencies you are actually interested in.
In my opinion, having a high sample rate only really matters during the production phase and does not have a noticeable effect on the final form factor. If the producer uses high sample rate during the creation process, I see no reason why the listener would care if the file they’re listening to is higher than even 44.1kHz unless they are planning on using it for their own production.
However, any type of subsequent processing in the digital domain, even just a volume change by the listener if it's applied digitally in the 16 bit realm (i.e., without first upscaling to 24 bits), completely destroys the benefit of dithering. For that reason, we might say that additional processing isn't confined to the recording studio and can happen at the end user level.
I'm unsure whether this same logic applies to sampling frequency, but probably? I guess post-mastering processing of amplitude is far more common than time-based changes, but maybe DJs doing beat matching?
The real benefit is not using 6x network bandwidth, storage, memory, processing power and more battery of the mobile device. That benefit is not going anywhere, no matter what.
Post-processing is applied to the signal which is physically impossible to distinguish from the source. It is true that it often needs higher resolution, and DSPs will upsample internally and then back and operate on floats. But to claim without evidence, that post-processing may give human listener back the ability to tell apart whether 192/24 medium was used instead of 48/16, would be to reintroduce the same quality-loss paranoia, just with an extra step. If one couldn't hear the difference before an effect was applied...they won't hear it after.
As for DJs, they do use high-res assets when producing mixes. That's still mastering stage, technically.
That’s why producers (mixing many tracks in a session) want to use high bit rate stems, because they are summing the noise from n tracks.
It’s a pointless exercise for DJs or anyone listening to a single source to use a higher bit depth.
Dithering (or more bits) does solve for this. A fade out of the song also lowers the captured noise floor, but the dither function keeps going.
It's akin to noticing occasional posterization (banding) in very dark scenes if your TV isn't totally crushing the blacks. With a higher than recommended black level, you will see this artifact, because perceptual video codecs destroy (for efficiency purposes) the visual dither that would otherwise soften the bands of dark color into a nice grainy halftone sort of thing which would be much less offensive.
I think that "if" is doing a heavy work here.
All because Sony and Philips wanted 80 minutes of stereo audio on CDs decades ago.
It's very likely that the 44.1 kHz rate comes from the PCM adaptors that were designed to take PCM audio and convert it to something that a video tape recorder would accept.
I watched a YouTube a few months ago about these adaptors and the presenter did the calculations showing how the 44.1 kHz 16-bit sample rate lines up with the video fields. There was a valid engineering reason for this sampling rate.
However, the stories about one of the Sony executives having a particular piece of music in mind are true, and have to do with the diameter of the disk being enlarged compared to what Philips originally had in mind. By that time the bitrate was already decided.
I still agree that 48 kHz is a better choice today, especially after reading this paper.
> Kees Immink, Philips' chief engineer, who developed the CD, recalls that a commercial tug-of-war between the development partners, Sony and Philips, led to a settlement in a neutral 12-cm diameter format. The 1951 performance of the Ninth Symphony conducted by Furtwängler was brought forward as the perfect excuse for the change,[76][77] and was put forth in a Philips news release celebrating the 25th anniversary of the Compact Disc as the reason for the 74-minute length.
https://en.wikipedia.org/wiki/Symphony_No._9_(Beethoven)#Com...
Everything you said about sample rate applies more to bit depth. Higher bit depth (bits per sample) results in a lower noise floor. When audio is digitally processed or resampled, a small amount of noise ("quantization distortion") is added, which accumulates with further processing. This can be mitigated by working at higher bit depths - which is why professional grade audio processing routinely uses 24 bit formats (for storage) and 32-bit or 64-bit floating point internally (for processing), even if the final delivery format is only 16 bit.
Sample rate, on the other hand, affects bandwidth. A higher sample rate recording will contain higher frequencies. It doesn't have any direct effect on the noise floor or level of distortion introduced by resampling, as I understand. (It could have an indirect effect - for example, if certain hardware or plugins work better at particular sample rates.)
A survey of ~2,000 professional audio engineers done in May 2023 showed that 75% of those working in music use 41.1 kHz or 48 kHz, while 93% of those working in post production use 41.1 kHz or 48 kHz.[1] These are the basic CD-derived and video-derived sample rate standards.
From this it's clear that even in professional audio, higher sample rates are a minority pursuit. Furthermore, the differences are extremely subjective. Some audio engineers swear by higher sample rates, while others say it's a waste of time unless you're recording for bats. It's very rare (and practically, quite difficult) to do proper tests to eliminate confirmation bias.
[1] https://www.production-expert.com/production-expert-1/sample...
EDIT: add link to survey.
Meaning, if an IMU sensor is mounted on a very slow moving, such as 2cm/s, RC vehicle then the sampling rate can be very slow. But if the sensor is on a fast moving drone, we need to estimate the highest frequency of the motion and make sure our sampling rate is at least double that?
Given a CPU runs at many GHz but SW sampling profilers run at ~1 or even maximum 10khz, it’s really hard to write software if you’re targeting processing at MHz rates.
Basically a 1ms sampler can pick out a signal that’s 2ms or longer in periodicity if sampled once (all faster signals will get aliased). To get to 1ghz (once a nanosecond) would require capturing 1 million times more samples and you’re still dealing with aliasing screwing up the picture you’re getting (although maybe with stack sampling you get disambiguation to combat aliasing? Not sure).
Surely for this to happen you'd have to be putting a lot of effort into getting a perfect 1ms sampling rate, and even a little bit of variation in that would be more than enough to handle aliasing issues.
It sounds like one of your concerns is being catastrophically unlucky with the sampling rate: > This is because you run into aliasing with the samples being collected at unhelpful points.
I interpret this as you saying "we sample at times T, 2T, ..., but there's a hotspot that hits at T+0.001, T+0.003, ..., T+0.999, 2T+0.001, ..., and we never get to visit it". I'll grant that this could happen, altho it seems contrived, but my claim is that by sampling at "T+/- epsilon, 2T +/- 2 epsilon, ...." sooner or later you're going to start hitting that hotspot. And before too long if, say, 5% of the time the CPU is executing that code you're going to hit it, on average, 5% of the time.
It'll be aliased, sure, but in a way that smears across all frequency bins instead of getting missed. You won't be able to recover the true frequency (at least not without fancy sparse methods) but why do you care? The important question is "where is the CPU spending its time" and not "is this function being entered 100000 times per second or 100001 times".
Here's another general objection. The things being sampled are square waves: a function is either in the call stack or it's not, the program counter is either at a particular location or it's not, and so on. That means you're going to have energy at all odd partials, which you'll have to account for somehow, but however you do it it's not going to reflect the underlying behavior.
Remember - profiling tools aren’t even giving you a frequency bin and indicating which samples you see there. It’s giving you a sample and estimating frequency. Most people are not optimizing code that’s running at millions of times per second so it’s not a common problem, but all sorts of wrong conclusions start to get made.
Only when your sample rate is perfect!
Let's say we have a 1GHz system, one instruction per Hz, a signal at 1MHz and we're measuring at 1kHZ. If we're rock-solid at 1kHz then, as you said, we can't distinguish 1MHz from 2kHz, but if we're off by a little bit, then things change.
Let's say the 1MHz events are at times 50, 1050, 2050, and so on, and the 1kHz sample rate triggers at 0, 1000000, 2000000, and so on. You'll obviously never see that event.
But suppose there's a little bit of noise in the sample rate timing and we're just under 1kHz, so now the samples are 0, 1000001, 2000001, 3000005, etc. Sooner or later we're going to hit that 1MHz event, and we're not going to hit it at the same rates as we would a 2kHz event, a 10kHz event, and so on.
We might not hit it that often, but we'll hit it sooner or later, and we'll also hit a lot of its neighbors.
> The difficulty with the Nyquist-Shannon sampling theorem is that it is based on the notion that the signal to be sampled must be perfectly band limited. This property of the theorem is unfortunate because no real world signal is truly and perfectly band limited.
whose relevance I'll get to below.
(I'm assuming that by "sampling profiler" you mean the usual practice of recording on a cadence the call stack / program counter / etc of some code; if this is not what you meant please clarify)
If you approach sampling profiling from a signals viewpoint, what's the signal that you're sampling? I see it as a collection of pulse waves of varying (and irregular!) duty cycles, each pulse wave corresponding to a yes/no answer to "is the program inside this function / executing this instruction / etc". At any given sample we collect our data and that tells us which waves are high; all others will be low.
Nyquist, as the above quote points out, only applies to perfectly band limited signals. Not only are pulse waves not perfectly band limited, they're actually extremely not band limited, with energy at harmonics going all the way up. Right away that should tell you that Nyquist is the wrong way to be thinking about things!
And furthermore, Nyquist tells you what you need if you want to reconstruct the original signal, but why do you want to do that in the first place? Do you actually care about the phase of the wave and the timings of the pulses, or do you just care about how often the wave is high? (i.e. how often any bit of code is executed). I don't think I've ever cared about the phase and timings when profiling, but I do care very much about how often the wave is high.
It doesn't have big factors?
I don't know what I'm supposed to be surprised about.