Ask HN: Why isn't VoIP better? | Hacker News Reader

Ask HN: Why isn't VoIP better?

143 pointsscandox138 comments

I'm very sensitive to bad call quality. I'm very easy going about most things, but poor sound quality or an intermittent connection on a call, make it completely impossible for me to maintain normal communication and also makes me extremely cross.

As a result I am very attuned to the general quality of different call services. I always know when I'm on a PSTN that terminates to another PSTN - yes maybe the voices are a bit trebly but the connection is so consistent and wonderful. I actually appreciate the quality of those calls.

Skype, Viber, our own corporate VOIP system (whatever the hell it is) all drive me completely nuts. I have 240MB/s at home, but still VOIP lets me down: about 1 in 3 calls is of a quality I am happy with.

I guess my question is: will this ever be solved to my satisfaction, or is it simply the nature of the beast and "good enough" for most people?

View on HN

I'm surprised that no one has mentioned bufferbloat yet. In most home routers, people can see seconds of delay when there's significant other traffic on the link. (And remember, "other traffic" can be ordinary web browsing, where pages average 2 mbytes these days...)

The latency (and jitter) caused by bufferbloat has been a solved problem for almost five years: fq_codel, and the newer cake qdisc's are are in the Linux kernel, and people's home routers now could install LEDE/OpenWrt (https://lede-project.org). Two links:

What can I do about bufferbloat? https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_a...

Test your network for bufferbloat: http://dslreports.com/speedtest

Bufferbloat is a side affect of congestion or microbursts, and not the primary cause of poor voice calls. Either packets get dropped (shallow buffer) or queued (enough buffer), and in both cases the quality of the call will suffer.

Mitigation: Don't congest. If you need to, mark VOIP traffic with a DSCP value that has queues that are serviced prior to all others (e.g., EF, DSCP46).

Applications do it already, but edge routers are adept at ignoring these flags or applying them only to their own service.

So that you use it and not competition.

Net neutrality would be the antidote.

You're correct, most applications do.

QoS/CoS services are best when clearly spelled out in service agreements. Providers may clear markings, but likely because either they don't want to have a free for all when it comes to their queues, or because they didn't purchase the gear that offers the rich queuing needed.

If those flags provide better quality of service, why am I not applying them to all of my traffic? Any attempt to "know" if my use is legitimate would undermine network neutrality.

Sure, you could make yourself crazy trying to prioritize all the traffic with varying kinds of flags.

Or you could just let an algorithm (SQM - fq_codel or cake) automatically determine which flows are sending more than their fair share of traffic into the bottleneck link, and offer backpressure to slow those applications down, so other applications work well.

re: net neutrality. That applies to whether my ISP (or any provider upstream in the Internet) should prioritize packets/data from various sources toward (or from) me. The answer should be, No. My contract with my ISP is that I get X mbps of data. The contract doesn't say the ISP will provide higher priority or data rate for service Y over service Z...

SQM has no effect on net neutrality: I'm simply configuring my own router to prioritize the data that I'm requesting (or sending). My ISP should handle them in a totally neutral way...

Yes. I'm a Republic Wireless (WiFi VoIP phone) customer. They mark their RTP packets with DSCP 48 (CS6 / ToS 192) and their SIP packets with DSCP 56 (CS7 / ToS 224). I limit ingress & egress bandwidth on my home router (a Mikrotik), and prioritize flows with those DSCPs above all other traffic.

The result is that even when my pipe is completely full, VoIP packet latency is practically unaffected. Compare to ~3 s latencies I would incur by Comcast's awful traffic shaping.

I have almost no bufferbloat when downloading (about 1ms) but about 300ms when uploading.

What does this mean?

I didn't look at that tool, but just reading this suggests that you're being shaped/policed more aggressively on egress (in the direction of your provider) than you are on ingress. Shaping and policing (typically) both use the same mechanisms as interface buffers to temporarily store packets in order to prevent congestion/discards. The distinction between them is that shaping actively buffers packets, where policing will more aggressively drop them (and aim to trigger a TCP congestion event) - in most cases, policing will involve a burst bucket that is somewhat similar to a shallow queue.

This would make sense if you have a higher bit rate from your provider for down vs up.

What speed connection do you have, and what speed connection did that tool report?

If your connection is fast enough that that tool is unable to max it out for whatever reason (say you're on a poor WiFi connection), you will experience no bufferbloat.

Yes, Don't Congest. But you can spend your life marking various kinds of traffic (many apps already do mark it, as someone down-thread mentioned), and pray that your router (and remote equipment) handles the markings properly.

Or just use SQM with Cake qdisc. It actually looks at the "sojourn time" for packets in various streams/flows of data and offers backpressure for sessions that are generating queues of data: all the other flows "go right through" without a lot of other settings.

I believe Ubiquiti's routers also implement fq_codel: https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-E...

> Add new "smart queue" feature providing FQ-CoDel + HTB function and this can be configured in the Web UI to provide better QoS experience for the WAN connection.

Thanks for these! I have the Omnia Turris with Openwrt at home, which has everything needed to fix this. I'd also hope that problems playing music in our home network from the NAS when moving big files in the lan could be solved with these settings.

Anyone know if Mikrotik's RouterOS suffers from this?

Bufferbloat is typically a disease of your ISP (which throttles traffic), not of your router. A Mikrotik in its default configuration, like most other routers, will do nothing to address – nor exacerbate – bufferbloat caused by ISP throttling.

However, it is fairly simple to address ISP bufferbloat using RouterOS's "simple queues". In the most basic configuration, just add a simple queue targeting your WAN interface, with bandwidth limits in each direction slightly (~5-10%) lower than what your ISP throttles you to. If you have a slower connection (<10 Mbps), select "default-small" as the queue type. That's it, bufferbloat solved. (You can get fancier with multiple queues or different queue types, but the above alone gives you 90% of the benefit.)

Awesome, thanks!

Not so - Bufferbloat can occur everywhere there's a bottleneck link.

Your home router's link to the ISP is likely to be one, and most don't have any mitigation, and suffer from high latency. (And various ISP's equipment has bufferbloat for data coming toward your home.)

Some sites suggest that you knock your bandwidth down to 70% of the link speed to "leave some headroom" for other packets. That's fine if you want to give away 30% of your capacity.

SQM (implemented through fq_codel or cake qdisc's) takes off a few percentage points from the rated link speed, and achieves great results with minimal setup and configuration. Check out https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_a... for more info.

That is… exactly what I said? Bufferbloat occurs where there's congestion. That's generally on the ISP's router, since they're the ones throttling your connection to 10 Mbps or whatever.

Yes, if you have a gigabit fiber connection and your home router can only handle 50 Mbps, you can get bufferbloat on your home router. But most people aren't in that scenario.

SQM is great, but not available on RouterOS, which is what the GP was asking about.

Yes, but it can occur on both ends of the bottleneck. Your ISP's equipment has a bottleneck link toward your home, and can have bloated buffers on that side. But if your router doesn't handle the uplink well, it'll be bloated as well.

Fortunately, SQM is available (from LEDE) for many MikroTik routers, so there may be a way out for the GP.

That's not how the Internet works. If your ISP is throttling, both upstream and downstream traffic is buffered on its router.

This is a consequence of how congestion is signaled in IP networks: by dropping packets. If the ISP is the bottleneck (it almost always is, due to throttling in both directions), it signals this to upstream equipment (i.e. the source of the packets) by dropping packets. They aren't sent back to some random router to be queued up.

Bufferbloat comes in because the ISP is trying to make use experience a little less crappy by hanging on to bursts of packets which would otherwise be dropped, with the intent of forwarding them on at the throttled rate. The consumer's router does not do this because it has no way of knowing this is happening. (But see below.) This is fine, except ISPs make their buffers HUGE.

There are two ways around this: one is, as you suggested, to use fq_codel, which monitors flows to infer packet buffering or loss at the ISP. So then, it is able to avoid sending packets which would be buffered or dropped by the ISP. The other is, as I suggested, to make your router the one dropping and buffering packets, which it normally does NOT do. It has no reason to: the links to the modem and the client are typically much faster than what the ISP throttles to, and those links are all it can see. (Though it's not unlikely that the user has subscribed to a higher speed than their wireless can handle, in which case inbound traffic is queued on the router, NOT outbound traffic.)

Yes, if your router is underpowered and you have a high-speed connection through the ISP, then your router could become a bottleneck and show bufferbloat. That is almost never the case, and when it is, it can happen in both directions.

The GP has "a way out", it is exactly to perform the steps I indicated. fq-codel is nice to have but NOT necessary.

I'm sorry, but I have to disagree. The laws of physics (of the internet) dictate that the piece of equipment at a bottleneck is responsible for handling bloat/congestion on its own end. Your ISP's head end/DSLAM/etc. has an opportunity to prevent download bloat (with data being sent to the home), but it has no way to know what's happening on the consumer end of the link.

Your router at home needs to have the same logic: it needs to control the buffering of data being sent toward the ISP. fq_codel/cake actually measures the time packets dwell in the queues (their "sojourn time") and drops/marks with ECN some of the packets of flows that build up a queue.

A cool feature of fq_codel/cake qdisc in LEDE is that it can control bloat in both directions. It imposes a bottleneck that's slightly (a few percent) below the actual ISP download link speed so that traffic queues up within your home router (instead of at the DSLAM/head end). Consequently, it can do the fq_codel/cake algorithm on the download (as well as the upload) direction, keeping your link unbloated at a very small loss of link speed.

You seem to be working under the fundamental misunderstanding that the link is the bottleneck. IT IS NOT. The ISP's router is the bottleneck, because it typically throttles the connection.

For example, my home network has a 100 Mbit bidirectional Ethernet connection between my router and my modem, and my modem has a 152 Mbps downlink (4 bonded DOCSIS channels) and an 81 Mbps uplink (3 bonded DOCSIS channels).

Neither of those is the bottleneck, because my ISP throttles my connection entirely on their router to 3.5 Mbps downstream, 1 Mbps upstream. I, like most in the US, cannot afford a 100+ Mbps home connection.

What you said would hold true IF, say, I had a gigabit connection (but still used a 100 Mbps Ethernet link). Then, yes, upstream traffic would build up in the upstream queue on my own router or modem. But that's not the case, and neither my modem nor router see anywhere near the 100+ Mbps of traffic needed to saturate the link.

If you do not understand this, I suggest looking at the queue depth on an actual router, with all traffic shaping turned off, while running a speedtest. You will see the interface queues in both directions be completely empty.

Yes, but... A "bottleneck" occurs wherever there is a high speed link going to a lower speed link. In your case, the 100 Mbps Ethernets in your home feed through the router to the 1 Mbps upstream your ISP provides. That's another bottleneck.

At this point in a conversation, I always recommend people measure their actual network, to see if they're happy with the situation. If it's good, then everyone's happy.

What results to you get from the DSLReports Speed Test (www.dslreports.com/speedtest)? It measures latency (lag) during the download and upload parts of the test, and will show if your router (or your ISP's router) is buffering too much data, and giving you undesired latency. Best regards.

The 100 Mbps is from the router to the modem. There is no such thing as a 1 Mbps "link" to an ISP. That bottleneck is imposed by traffic shaping on the ISP's router and nowhere else. (An exception would be something like ADSL, where the uplink truly is the bottleneck. But even then, the queue builds up in the modem, NOT the router!)

Here's a picture, since you seem to be ignoring my words:

     150 Mbps WiFi
           |
      home router
           |
    100 Mbps Ethernet
           |
         modem
           |
    152/81 Mbps DOCSIS
           |
    +-ISP router-----------------+
    |      |  <-- QUEUE          |
    | 3.5/1 Mbps traffic shaping |
    |      |  <-- QUEUE          |
    +------|---------------------+
           |
     10 Gbps fiber (or whatever)

Or, in the case of ADSL:

     150 Mbps WiFi
           |
      home router
           |
    100 Mbps Ethernet
           |
         modem
           |  <-- QUEUE
     8/1 Mbps ADSL
           |
    +-ISP router-----------------+
    |      |                     |
    | 3.5/1 Mbps traffic shaping |
    |      |  <-- QUEUE          |
    +------|---------------------+
           |
     10 Gbps fiber (or whatever)

In either case, neither queue builds up in the home router -- that is exactly the problem, since we can't control the queue size in the modem or ISP! But by traffic shaping to 3.25/0.85 Mbps in the home router (either manually or automatically with fq-codel), we force the queues to build up there, thus giving us control.

> In either case, neither queue builds up in the home router -- that is exactly the problem, since we can't control the queue size in the modem or ISP! But by traffic shaping to 3.25/0.85 Mbps in the home router (either manually or automatically with fq-codel), we force the queues to build up there, thus giving us control.

YES! I understand what you're saying. (I had been treating the cable/DSL links and their terminating equipment as a black box; your description was being far more specific. Thanks for sticking with me to make yourself clear.)

And as you point out, the fix in all cases (at least, until our ISPs get serious about solving bufferbloat) is to force the home router to take control of the buffering by shaping the traffic a bit below the actual link speed. Thanks.

Not in Germany, kernels and routers need! To be locked by law.

I accidentally updated without know. Now I'm locked...

citation needed. If that were true, quite a few router models available would be illegal, so it probably isn't.

(TP-link started locking down at least some models, in response to FCC rules, and some manufacturers don't allow you to downgrade, but there is no legal requirement to do so and often ways around it. Other manufacturers like Linksys openly advertise their routers for use with OpenWRT)

Oh boy, direct down vote instead of looking on your own or asking...

http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELE...

http://www.tp-link.de/download/Archer-C7_V2.html#Firmware 1.For Archer C7(EU)V2. 2.The EU firmware was specialized for CE certification and can't be downgraded to other version, please click here for choosing your region and selecting the most suitable firmware version to upgrade. 3.Old firmware's configuration file can be imported into this new firmware; 4.Your device's configuration will be lost after upgrading, which means you need to configure your device again

All routers are doing the same. EU law.

Sorry, was a bit to quick and I misread your post as describing the current situation, where this not (yet) applies.

And last I've heard about it they extended the validity of the old rules for undefined time (due to most of the standards not being ready), but I can't find a good source for it now, so I might have gotten that wrong. My mistake for not checking better.

> All routers are doing the same. EU Law.

Citation still needed.

a) I reviewed your eur-lex.europa.eu link, and didn't find anything that prohibits installing other firmware. Please cite the chapter/article that's relevant.

b) re: TP-Link firmware. It is true that TP-Link has added a technical means that makes it impossible to install certain other TP-Link firmware (the "downgrade" they mention). However, there are other firmware distributions (LEDE/OpenWrt/DD-WRT/others) that have the capability to install over the TP-Link factory firmware.

Lots of good answers, but there is another one that's important too: echo cancellation. Most telco equipment has very good echo cancellation hardware in it. It's not impossible to do a decent job in VOIP, but not all of the software does it well. This can be very important when you have high latencies or if you are doing conference calls.

The other main overlooked problem is automatic gain control in software. This one drives me crazy. Most people are too lazy/clueless to set the gain properly on their microphone, so VOIP client makers tend to use automatic gain control to adjust the input. However, it works very differently that what you are used to in a telephone. If someone laughs loudly, or coughs you end up losing the beginning of the next sentence. It can work extremely badly in conjunction with echo cancellation depending on the software -- to the point where the audio can be very difficult to understand.

It's probably my biggest gripe with Google Hangouts because there is no consistent way to shut of AGC on all platforms.

I've written VOIP clients before and it is possible to have incredibly good audio (much better than normal PSTN -- mainly because the codecs are better). But it's very tricky to get it to work on random hardware, with random network conditions and with users who don't want to configure their client optimally.

Got any link to good echo cancellation hardware? Not that I'd every buy it, I'm just interested in tech (and businesses) that's not for average consumers.

I work at a big telco in the voice group and we still have tons of DMS 100/200/250s. Anyways some interconnections we have require Ditech echo canceller (no idea what model but apparently when we need them, we use spare ones or have to buy them on the used market). We have been migrating traffic to newer equipment and the new gateways have echo cancel built into them (e.g. Genband G9s)

I'm going to agree with you and take this opportunity to plug http://cleanfeed.net/ We run a free 'VOIP' system that does zero processing on your audio.

It's actually for broadcasters who just want to get live audio from A to B, untouched. But as it's two way (and multi-way) we're finding users contacting us to say they're using it for regular 'conferencing'.

I suppose we are also doing it ourselves; we used Cleanfeed to communicate when developing Cleanfeed.

Elsewhere in the thread people mention jitter etc. issues. In my experience, 200-300ms of latency has much less negative impact on rapport than aggressive audio processing.

Thanks for the recommendation!

FAQ #1 should be "how does this remain free/why is this not going to disappear"?

Thanks, good suggestion. It's actually on the front page that our free offering is a spin-off of something commercial. But a FAQ to clarify this particular point would be sensible.

Not all uses of VoIP are done with software clients, mind you.

I manage SIP setups for myself at home as well as at my aunts shop, both of which have physical IP phones instead of softphones. While softphones are certainly useful, most people on a VOIP service of some kind are probably more likely to still use a physical handset, headset, or proper conference phone where echo cancellation is less of an issue since it's either handled by the hardware itself or non-existent in the case of users on a headset.

Jitter is one of the words you're looking for. 20 milliseconds does not sound like much, but when it's a delay in some (but not all) packets in a call getting to you, you can sure hear it!

Jitter is why you could have a 1 GB/s connection at home, and still have the same occurrence of poor call quality.

QoS (Quality of Service) can help, but only if it is applied at every router the traffic for a call goes through. QoS-assured bandwidth is more expensive, and cost-conscious VoIP providers don't seem to be willing to pay for it.

(edited to add) I'm a Skype for Business admin.

> QoS (Quality of Service) can help, but only if it is applied at every router the traffic for a call goes through.

You really only need it on routers where a bottleneck is. If a router is forwarding 100% of packets immediately there is nothing for QoS to do.

In practice what this means is that the most important place to have it is at the border where you go from high bandwidth LAN to lower bandwidth internet.

In particular if you control a router there, make sure it knows to limit traffic to what the next hop link speed actually is. In will often see the link as gigabit ethernet even if it's slower than that and then forward all traffic when you really want it to be the device that chooses which traffic gets dropped.

Unfortunately if your ISP supports burst bandwidth it's not always easy to get it right without breaking that.

QoS depends on tagging (DSCP), and if one of the routers along the way doesn't support it or isn't configured for it, those tags are often dropped - or at least that's what I've seen in our company's network.

Like so many other things in tech, "it depends".

Network gear can be configured to treat frames/packets in various ways -- leave the ToS/DSCP bits alone, strip them out, or remark them. At the edge of a network, it's common to mark them. In the core of a network, you'll usually leave them alone (but honor them). On connections to third-parties (such as ISPs), it's not unusual for them to strip your QoS markings -- unless you're paying extra for them not to.

For a random packet that leaves your home network and hits the Internet (even VoIP traffic), I'd be surprised if it ends up at its destination with those same QoS bits set.

Like zrm said, though, if there's no congestion on links, prioritizing traffic typically isn't going to do you much good. It's only when there's contention that moving packets to the front of the line helps out.

Except Internet routing is not smart, the routers cannot send you through a low bandwidth but uncontested and low latency pipe when your packets are unmarked because you didn't pay a premium for a peering agreement.

It actually can hurt the Internet performance as a whole, as the tiny VoIP packets prevent Jumbo throughput ones from working really well.

It takes just one bad hop to ruin latency.

Sure they can, especially on MPLS circuits sold for exactly this reason -- and with SLA commitments to deliver packets w/ specific markings within specific parameters (i.e. up to 5 Mbps of traffic marked ef will have maximum latency of 30 ms between A-Z).

Not on your home Internet connection, perhaps, but if you're a company and you want guaranteed performance levels (i.e. for VoIP traffic), you can certainly get it.

I think your complaint is about VoIP over the internet. And that's simple, the internet is best effort. Packets get dropped and that's expected. Also, packets can take a long time to reach their destination and there are no guarantees for jitter. This absolutely sucks for voice quality and there's nothing anyone can do about it.

Your landline is now VOIP too, it may be twisted pair to the central office, but then it's digitized and transmitted on the carriers owned networks.

Then how does Google Fi work? I don't know if it's VoIP but it is phone calls over the internet and I've never had a problem with call quality.

Both ends need to support wideband codecs, as does your phone (hard or soft phone). This includes the VOIP providers needing to be able to use wideband codecs between each other.

If you achieve a wideband setup the call quality is amazing. I've used it for years. It really sucks when you have to call someone on a legacy POTS line or using a narrow band codec (bad voip, cell phones, etc)

On that note it's unfortunate that cell phones which can do wideband codecs within the same carrier (HD Voice!) won't work if your call leaves their network.

> On that note it's unfortunate that cell phones which can do wideband codecs within the same carrier (HD Voice!) won't work if your call leaves their network.

This is quite annoying, but it's because the PSTN overall is stuck on G.711 and G.729 - my SIP provider doesn't support anything else because they would be forced to re-encode the signal anyway and lose quality doing so.

Ultimately, we need to be putting pressure on the telco's to enable support for G.722 - until that happens we're going to be stuck with 4KHz audio for a long, long time.

I've been in the VoIP industry for over a decade now, and in almost all cases of 'bad VoIP' I can trace the blame to a bad implementation of it. Let me give you an extreme examples...

- It's fairly common practice in countries with poorly enforced telco law for people to resell access to the PSTN. They will connect to the PSTN (either via a GSM gateway or a server with an E1 card) and resell that access to a 'wholesale VoIP business'. - That 'wholesale VoIP business' will have an infrastructure, cobbled together with open source technologies with no ability to scale, in the operational or technical sense. - The same 'wholesale VoIP business' will undercut the cost to access the countries PSTN (compared to accessing it directly via a tier 1 carrier) and sell the access to other tier 1 carriers. - The other tier 1 carriers lap up this cheap access, an embrace it because international calls fulfil a small portion of their multi-million revenues. - Therefore, when someone in another country tries to call that country with poorly enforced telco law, it will route via the resold route until it's shut down by the law or it breaks because it's poor infrastructure. - This is currently happening on a big scale in lots of African countries, some central american countries and in the middle east.

I'll leave the context of 'office to office' comms to the other answers as it's been covered in detail elsewhere. But I will say VoIP, all it's forms, is an old technologies, and problems relating to poor bandwidth.. voice quality.. backwards compatibility have been solved for many years.

First, as others pointed out, you will need a decent internet connection, with minimized packet loss and jitter. Most metro area data connections and quality WiFi access points will be sufficient. However, the wifi router that comes from your provider and stuffed away on the other side of the house might not work very well.

Secondly, good codecs make a world of difference. This is where I'll add the disclaimer that I'm in the collab side at Cisco. But I have the opposite experience using systems on my phone, my mac and with big telepresence units: the quality in all 3 settings far surpasses what's on PSTN, and if you're using good speakers or headsets, you can really appreciate the good quality.

In your case, it sounds like you might benefit from a better network access point at home.

Shameless plug #1: check out Cisco Spark. We still have some rough edges but I think the call quality is what you're looking for. I'm able to carry on a call and switch from wifi to LTE without dropping, or turn on VPN in the middle of a call and just get a few seconds pause before it recovers.

https://www.ciscospark.com/

Shameless plug #2: check out the new Sparkboard. It's got 12 microphones and does beamforming in order to filter our room noise away from whoever is speaking.

http://www.cisco.com/c/en/us/products/collateral/collaborati...

Does it automatically report your political and religious affiliation to chinese ministry of public security? Lol, zing.

It's end-to-end encryption. For large sites, customer owns the encryption keys on their hardware so we can't intercept/decrypt even if we're ordered to.

I've tried using spark but unfortunately video calls don't seem to work at all when I'm on my corporate VPN (which is Cisco, btw). I really want to use it because it otherwise works on Linux (compared to webex...) but I need to be able to use it on the VPN.

> Secondly, good codecs make a world of difference [...] quality in all 3 settings far surpasses what's on PSTN [...]

To be fair, this is probably true of any decent VoIP solution on the market. The problem is always that moment you have to connect to the PSTN and you're forced into a choice of G.711 or G.729.

VOIP calls are very sensitive to intermittent connectivity.

Sometimes wifi will work in batches. No packets will get through for a few hundreds of milliseconds (or seconds), then they will get sent all at once.

Sometimes UDP traffic is blocked completely and the call has to get relayed over TCP. Now no packets can get dropped and you get delays due to retransmits.

Sometimes 3g/4g connection disappears because someone entered a building an overpass or passed a corner in a city.

Sometimes the sides of the conversation are far and the audio delay is high.

If any side of the conversation encounters a problem described above both will get a bad quality.

Among other things, I'm a Skype for Business admin. We've found that having our users use Ethernet when working from home instead of WiFi eliminated a lot of audio quality complaints.

At home I use Ethernet, a Blue Yeti USB microphone and a solid pair of Philips USB headphones. Of course I don't know what my counterparts on the other end are using.

I guess my problem seems to be that we swapped a system that had a consistent call quality for one that is inconsistent, for a whole bunch of reasons. It seems to me, as if the general quality of telephony took a huge step backwards. And I don't understand, in an age of technical progress, why we should (or do) put up with that.

Are the 1 in 3 good calls pretty random, or do they tend to be with the same counterparts? Do you notice a difference between the quality of your calls into conferences (if on the same platform) and direct calls with colleagues?

WiFi can be a source of the problem even within the corporate network if the access points weren't designed for VoIP.

It seems random to me, but of course when I'm in User-mode I'm probably quite a poor observer.

I'm not a power user of VOIP, but these are my common scenarios:

1. At home, Skype and Google Hangouts. Hardware setup as above. 240 Mbps connection. Making calls to family abroad. This is the 1 in 3 scenario - we just keep trying till we get a connection that seems to work. Usually the call ends whenever the line finally "goes bad".

2. At work: Skype, Hangouts, Upwork. Mostly one to one calls with potential hires, colleagues, outsource people etc. WiFi. About a 20Mbps connection. Actually now I write it down Upwork is the best - which is weird because in general their systems suck. But Skype and Hangouts are both a consistent source of pain, dropped calls, bad quality, sudden total loss of audio etc...

3. At work: Company phones. Generally the quality is OK when calling out to landlines and mobiles (cell), but I have colleagues who I know are on VOIP at other end and these calls are like 1 in 2 where we just give up.

4. On mobile (4G): Viber, WhatsApp, Skype. Generally I'll call. Within a few seconds we'll be talking like people in a disco and I give up and call back on regular phone.

I also observe colleagues doing a lot of: "can you repeat that", "sorry repeat that" when they're using Skype at the office. Even hearing that contributes to my sense of VOIP-malaise.

Scenario one stands out to me. There will be a lot of hardware between you and the other party in your call and trouble can happen at any of them. The connection quality you get will depend on the worst performing hardware in that chain.

Sounds like your problem isn't with VOIP, it's with softphone VOIP clients and/or OTT carrier. The company phones are probably VOIP (do they just have ethernet connections?), but connected up to a telco which handles traffic nicely.

Most telcos are sending a huge amount of traffic round over IP. The difference is that they are controlling the bandwidth so stuff just-works, whereas when you use a purely OTT service, you're probably stuck with your packets getting the same treatment as any old web request. Most of the calls you think you can detect as POTS probably have VOIP legs in them somewhere if they're long distance.

Which of the services are the most expensive? My guess is the company phone. Are you paying for any of the others?

The only thing that could make VoIP voice quality as consistent as PSTN is a consistent application of QoS.

And that is why I have very mixed feelings about Net Neutrality. Sure, I don't think that Verizon should be allowed to penalize NetFlix or $randomVideoStartup by favoring its video traffic over theirs, but there is no technical way for voice to be consistently good without a consistent handling of its data traffic.

Network neutrality doesn't really have anything to do with QoS. Network neutrality says you can't prioritize packets from Comcast over packets from Netflix. It doesn't say you can't prioritize packets a customer has marked as more important over packets the same customer has marked as less important.

How much of the VoIP contention is on the user end vs the ISP end though?

I have a 1 Gbps connection at home and can still have a VoIP call across the world mess up since I might not have the best route to whoever I'm calling.

If you use Wifi, keep in mind that it's built on a very fragile physical layer. In fact, the MLME expects up to seven failed transmissions before calling a packet dropped (otherwise it would destroy TCP's performance due to TCP's default congestion control mechanisms). The delays each way from two people speaking over noisy Wifi networks can already add substantial lag.

Old fashioned PSTN lines would secure the full bandwidth needed end-to-end when the original call was established (which was really in efficient from a data utilization perspective).

With VoIP you are:

1. Beholden to the bandwidth/latency issues of the crappiest link in your end-to-end path.

2. Intentional use of poorer quality codecs (by your ISP and/or VoIP software to reduce aggregate bandwidth use). Ironically this can be inverted, there are codecs that sound much better than PSTN calls (but use more bandwidth).

> 2. Intentional use of poorer quality codecs (by your ISP and/or VoIP software to reduce aggregate bandwidth use). Ironically this can be inverted, there are codecs that sound much better than PSTN calls (but use more bandwidth).

I mean, people use G.729 for example because, yes, it is more efficient than G.711. But ultimately, if you want to communicate with the PSTN those are your only two choices and it's easier to just say "G.729 for everything" instead of wasting processing time on the server re-encoding everything that goes off the network.

You want better codecs? Bitch at the phone companies who have drug their feet on implementing G.722 while more and more IP phones and mobile devices continue to add support for wideband audio.

Single anecdotal counterpoint:

My corporate VOIP system is made of Polycom phones connecting via SIP to local Asterisk servers, and then those have interconnects to PSTN gateway services.

The quality on-campus is uniformly extremely high.

Some people have the Polycom phones at home (for working remotely) and calls to/from those are of similarly high quality, even in conferences.

Connections to cellphones range from mediocre to bad.

Connections to POTS land lines range from bad to pretty good.

Overall I would say that our VOIP system doesn't have a problem.

This is almost always a latency issue, and because nobody has figured out a way to milk extra cash for providing inter-network packet prioritization. In other words, this isn't an engineering problem, it's a money problem.

Skype used to be fantastic when you could establish a point-to-point call. Today, I think everything gets routed through Microsoft so they can provide access to law enforcement. Some days it's good, often it's pretty bad.

I believe FaceTime is still point-to-point.

I've heard good things about WeChat and it's support for low-bandwitdth networks but haven't worked up the courage to try it out.

Probably rather because the p2p algorithm was not included when they bought Skype, they only licences it...

I don't believe what you ask for can ever exist. "Clear, Distortion Free, Low-Latency. Pick any two" comes to mind.

When you're doing voip you're doing so over UDP because TCP has far too much latency to be agreeable, but this means a lost packet will be a digital distortion. The nature of the internet is that packets get lost or arrive out of sequence- that's why TCP exists in the first place.

So what you're saying is: VOIP will never match PSTN? Because that's the mood music I get from the sysadmins at work too.

Never is a long time. What if some day the technology exists to interpolate dropped packets (e.g., using ML) fast and cheaply enough to fill in the distortion? That doesn't seem too outlandish in the next 20, or even 10 years.

That sounds fine, except for the timeline. What I'd like is an affordable, ubiquitous solution next week :)

Uh, the modern codecs are already using linear prediction and correction. (essentially a rephrased more advanced perceptron) This is why SILK (Skype), Speex and Opus (SILK-derived part) can work even mildly well on bad links. They predict syllables and also long term speech characteristics. Going to full sentences or words is much harder and not likely to bring gain other than free subtitles. And would require really good voice recognition comparable to Siri at least. Good luck running that on an embedded device reliably.

Remember, the codec has to handle less than 20 ms latency. Running a sizable ANN (fast) or HMM in that time is not easy or even possible. A small one will be no good for various speakers.

Instead the codecs use various specialised error hiding tricks and adaptive strategies.

Heck, Codec2, one of the strongest voice codecs around could be used if all else fails. It is beginning to be used in SDR and amateur radio - optimised for availability and intelligibility not quality. It would be nice to have a variant integrated with CELT for higher quality.

VOIP is considerably better than PSTN if configured correctly. There's no comparison. Your sysadmins don't understand VOIP.

Their ignorance is always just slightly less than mine. Which is a sign of their intellectual "efficiency".

just want to point out: vast knowledge of telephony/VOIP is not something every sysadmin should be expected to know. If you don't have telephony expertise on staff I hesitate to suggest someone build out a VOIP infrastructure on their own. It can be extremely time consuming to learn and it is definitely painful to troubleshoot.

Shouldn't it be possible for me to get some IP phones in my office and tunnel through my router to a competent online voice provider? If so, can you recommend such a provider?

Yes, that is exactly what we do at work. We have ~20 IP phones in the office plus a few in remote locations. A few of us also use softphones (X-Lite or Zoiper) & USB headsets.

Our pbx is hosted by http://sipworxx.com/ and it works great.

http://voip.ms is pretty decent and offers a few nice features that can avoid building an on-site PBX.

I use this with good Polycom "HD voice" devices. I'm also rather picky about voice quality.

Eh, I'd argue this varies a lot. In some cases you are more likely to run into issues setting up multiple SIP clients behind a NAT trying to connect to a server out on the public internet than you are to install your own PBX on-premises.

Grandstream, Yealink, etc. all provide pretty easy to administer Asterisk-based PBX appliances that can be paired with a SIP trunk of your choice (shout out to Flowroute here, they're great).

If you're feeling more adventurous sipXecs from sipfoundry is pretty easy to configure relative to the additional complexity it brings over the more simplified alternatives. I went from having 0 real world experience with SIP to managing a small installation for my aunts shop as well as one at home in a couple days.

For proprietary offerings there's of course Skype for Business, etc. and I would argue they aren't terribly hard to configure or maintain either.

Most of the work is just in network configuration once you have multiple sites and you need to ensure QOS going over your MPLS or whatever to ensure your voice packets take priority - that can be more difficult but that's something a competent network admin can help you with.

I think that statement needs some qualification. If you're sending the VoIP traffic across networks you don't control all bets are off. VoIP across an enterprise's private network or through a leased network where SLA's can be mandated and QoS can be controlled is one thing. "Wild west" VoIP across the Internet, especially through sketchy networks, is another.

Edit: "canbe"

Its bandwidth efficiency, latency, and jitter, actually.

Just like in the old days of 9600 baud leased lines it was hard to get acceptable console performance, and now people have megabit connections and CLI performance is not unusually constrained over the megabit connections. Of course its only using a fraction of 1% of the capacity.

Also remember that end users might have megs of bandwidth to carry K of voice traffic, but somewhere along the line there's someone constrained. Just like work offices don't have to be dumps, but its hard to put a beancounter $ value on not working in a dump, leads to a lot of open office foolishness.

> When you're doing voip you're doing so over UDP because TCP has far too much latency to be agreeable,

Lots of wifis, corporate especially block all UDP traffic. Sometimes they even block all non http/https traffic. Apps like Skype or Biocoded will find a way to route the call, but that can sometimes mean it has to go over https!

It depends. I've noticed that sometimes my AT&T cell calls are of a higher quality, depending on who I'm talking to. I think this has something to do with their HD Voice product.

I like it, the quality reminds me of what I've experienced in the past with the Cisco UC platform on a well-provisioned network and UC platform.

The goal of Skype, Viber, etc. are to always give you some level of audio across all networks, so the codecs they are using have a different goal than Cisco UC.

All these codecs have low bandwidth requirements, much lower than what you have. If you have a residential ISP, QoS might be an issue with some of the higher sampling rate codecs, but you probably won't notice them with Skype.

Try running an open source VoIP MCU at home on your LAN to see how good it could be.

All quality problems would disappear if QoS would be implemented across the entire public internet. Your ears are so much more sensitive to small anomalies.

I've been deploying VoIP since 2003, both in the US and in Europe. In Europe it is easier to find cheap DSL with QoS applied, which allows laying in a separate circuit for VoIP only. In theses cases there are no complaints.

VoIP also works well on fiber. But as soon as one part of the network path travels the public internet all bets are off because there's no guarantee you packets aren't dropped because of traffic spikes.

Everybody who thinks the internet is stable in anyway is way off. Route flaps, packet loss, overloaded hops, last-mile problems happen all the time.

I could talk about this for hours.

And there's good news on the Wi-Fi front as well. Many of the same people who brought us fq_codel/cake have new algorithms for "Making Wi-Fi Fast".

Their paper tells how they've reduced bufferbloat as well as increasing "airtime fairness" in the Wi-Fi stack. The paper, "Ending the Anomaly: Achieving Low Latency and Airtime Fairness in WiFi", is now available as a preprint at: https://arxiv.org/abs/1703.00064 The conclusion of the paper states:

"We have developed a novel two-part solution to two large performance problems affecting WiFi: Bufferbloat and the 802.11 performance anomaly.

"The solution consists of a new integrated queueing scheme tailored specifically to eliminating bufferbloat in WiFi, which reduces latency under load by an order of magnitude. Leveraging the queueing structure, we have developed a deficit based airtime fairness scheduler that works at the access point with no client modifications, and achieves close to perfect fairness in all the evaluated scenarios, increasing total throughput by up to a factor of 5.

"Our solution reduces implementation complexity and increases accuracy compared to previous work, and has been accepted into the mainline Linux kernel, making it deployable on billions of Linux-based devices."

Fun reading! Oh, and it's available in LEDE today.

So VoIP can be better... at the expense of needing more bandwidth. During the session initialization the two VoIP endpoints handshake over which codecs will be used (kind of like the Accepts header in HTTP). Just like images, higher quality codecs tend to use more bandwidth (though like images its much more nuanced than that). Similarly like image formats there's different amounts of loss in the encoding/decoding of the voice data.

Voice has a major difference though. Its generally a real time protocol (its data packets are called RTP and sent over UDP generally). With images you can afford to wait longer for high quality images to load over a slow network. In a phone call that delay gets a bit annoying (to put it lightly).

Most people would rather trade quality for timeliness. You ISP would rather throttle your bandwidth and may not care about your quality (to a point).

Its quite a bit more complicated than that, but in a nutshell with VoIP, "higher quality" = "more bandwidth". If you don't have the bandwidth you'll have a very hard time getting better quality. If you have the bandwidth locally, then your provider or some network link along the way either doesn't have the aggregate capacity or may be actively throttling your rate to limit usage.

It really depends on which codec you're using. We had an implementation of G.711, which is not bad, but far from great. However, the license costs for implementing G.729 were so outrageous that my previous company simply couldn't afford it (or wouldn't). G.729 offers better call quality for the same bitrate, or similar call quality for much lower bitrate. Typically, G.711 would use ~64Kbps, whereas G.729 can achieve the same call quality for ~8Kbps (there's also a variable encoding that scales between 8 and 32Kbps). This being said, G.729 also requires a lot more CPU processing power.

If I'm not mistaken, each party of each leg of a call needs a license, which means that for a typical call centre with an IVR server between the agent and the customer, you need:

Summary:

- PSTN converter to IVR: 2 licenses / per customer

- IVR to agent: 2 licenses / per agent

- IVR to supervisor: 2 licenses / per supervisor.

It's pretty normal to see 300-agent call centres, and you typically want at least 300-900 calls waiting. Let's say there's between 10 and 30 supervisors.

300 * 2 + 900 * 2 + 30 * 2 = 2460 licenses. At $6-10 per license, that's more expensive than most hardware PBX solutions alone!

For IVR companies that handle multiple hundreds of thousands of calls per day, it's not really worth it.

Edit: Formatting

That licensing scheme is INSANE, especially since codecs like Opus exist, and do a much better job.

Once you touch the PSTN though, you're uLaw anyway. Why would you pay that much just for in-network voice quality?

Yes, codec licensing costs are one of the reasons people choose to not use the better codecs. Also, yes there is a CPU trade off as well.

The decision is more complicate with many trade offs. But in general, the OPs issues are either fundamental network ones or an intentional choice of one of his providers (typically for cost reasons - manifested in some manner).

Latency and speed are two very different things. For good audio quality, the first is way more important than the later.

Also, lossy codecs usually drop a good chunk of information in favour of compression, working with larger packets for better efficiency and adding even more latency. In that sense, G.711 or higher-quality PCM is the best if you want lowest latency possible.

I've worked for some time with telecom to know that each time you add a complex codec (G.72x) you may end up doing a few conversions on the way: in most gateways, if audio comes in as codec X, it gets converted to PCM internally and then re-converted to codec X again before going to the other end. Adding bypass logic adds little benefit as you have to scale the system for worst case scenario (all channels performing the costlier compression), so bypass only complicates your code.

So those "X ms of algorithmic latency" for complex codecs may end up being multiplied a few times: throw in echo cancelling, jitter buffers, etc and you get a call where you end up interrupting each other all the time due to audio delay.

You might try Google Fi. Not Google Fiber, Google Fi (https://fi.google.com/).

I felt the same way about the difference in quality between cell phones and old-fashioned landlines. "Telephone voice" used to mean "bad audio quality," but compared to cell phones, the old telephone sounds golden. The nicest thing about PSTN-to-PSTN calls is their timing. Cell phones have a delay that always trips me up and keeps us from getting into any kind of conversational rhythm. Again it's another thing that I didn't appreciate back in the '80s, when landlines were all there was.

Finally I got Fi. The difference in quality is refreshing, even when talking to people on cell phones with different carriers (which is all the time, because no one else seems to know about Google Fi). The latency is much less, and the sound is clearer.

Ready to kick it into high gear? I can make phone calls on my laptop. Plug in earbuds with a built-in mic, and phone calls never sounded so good.

No, I don't work for Google.

I find I agree.

- lossy vs lossless codecs (PSTN isn't analogue end-to-end any more, but is lossless as far as I'm aware. VoIP seems to have been about squeezing as much traffic down the pipe as possible, at the expense of quality.)

- circuit vs packet switched networks (Congestion, contention, jitter on shared packet switched networks. While I'm sure PSTN doesn't have "physical" circuit switching any more, I do get the impression that high-bandwidth bearers are chopped into fixed-width chunks, like 64k ISDN, and will reach capacity and stop accepting demand rather than degrade individual connections)

For me, wideband audio helps, even with the jitter and lossy codec. This is starting to become common in the UK on cellular calls between smartphones, even between different network providers. Audio bandwidth is comparable to Facetime Audio.

The lossy part of PSTN occurs at the front-end where only a 3KHz band of audio is passed through the network.

Data Scientist for Republic Wireless here. We operate a hybrid mobile phone service that leverages both VoIP and traditional circuit switched networks to let our customers make calls.

When asked to rate the quality of a call, our users consistently rate pure VoIP calls higher than their pure circuit switched counterparts. A significant portion of that is due to our "bonded calling" technology - if a VoIP call over Wifi starts to degrade, we can supplement the audio stream with additional packets sent over LTE until Wifi issues calm down.

When we first rolled that tech out, we saw a marked improvement in call quality measures, and a noticeable drop in customer support complaints about call quality.

In the end, I can't say this issue will ever be solved to OP's satisfaction, but we're making some incredible strides.

Have you tried Wire's audio? The creator of the SILK (Skype) and Opus codecs is on their engineering team.

http://opus-codec.org

http://app.wire.com

Wire's audio quality is the best in my experience.

Ive seen some enterprise level voip providers in the UK not only provide clients with VOIP systems either on site or a cloud PBX, but also a dedicated leased ADSL line. Usually 2mbps both up / down (They say this is good for about 20 concurrent users). They also connect the the line directly to their voip server, so it dosnt have to make multiple hops.

Never tried one in practice though. We run our voip over our main 200mbps down, 15 up fibre connection. Have QOS and VOIP prioritisation setup on the router, SIP ALG off etc. Seems pretty good, but as you said, if you know what to look for you can always tell your on a voip call.

Is that ADSL? If it's 2 up/2 down, is that not SDSL?

While A in ADSL stands for Asymmetrical, it is primarily name of the used DMT-based technology, S(H)DSL is completely different thing (in fact it is related to 1000-Base-T, but with significantly smaller symbol rate) that does not work across non-dedicated lines (and usually needs carefully selected pairs to work reliably).

That's counter-intuitive! TIL, thanks!

When I first tried JamKazam (a-play-in-a-band-over-the-Internet-thing) and talked to my friends over low-latency, high-quality gear, I was amazed but how much it felt like being in the same room.

That's when I realized that most conferencing solutions simply have too high latency. It's a matter of milliseconds - if there's too many of them people start interrupting each other by talking at the same time, and the feeling of having a natural conversation is lost.

They'd probably do better financially by selling their technology as a conference solution. ;-)

We have several VoIP lines on several locations. The quality is actually great. If you want guaranteed quality, use a dedicated voice VLAN or some sort of QoS.

Even in places where we don't have this in place, it seems to be working just fine for us. I can also say that I've used Skype to a PSTN in a separate continent over a slow mobile link and it also worked.

However: I have to mention that the quality is never quite as good as normal Skype (so called "HD" quality).

Your first paragraph could have been written by me so it was nice to discover I'm not alone. However, I find FaceTime audio calls to be of such higher quality than regular phone lines or most other VOIP services that it is my go-to for calls where I need to be able to concentrate. The only other service I've found to be comparable is Slack's calls.

Just as an aside, a potentially good alternative for international calls are callback services (depends on the service provider of course, could also be worse) as you will be able to skip your local networking hardware with them.

I don't have any specific recommendations though, haven't used one for years.

8kHz pcm ulaw or alaw isn't exactly stellar, and high bandwidth Skype or sip is much much better really.

Maybe you are conditioned to like the can-effect, your friends have really shitty headsets, or there is something wrong with the available bandwidth...

To clarify this a bit, SIP (session initiation protocol) is just a signalling protocol, meaning it is used to set up/tear down media streams between endpoints using various codecs. The sip standard includes a section of the message which is included in specific types of negotiation messages (invite, 183, 200 ok) called the SDP (session description protocol) to determine/negotiate which codec that gets used for the call (or video session).

In SDP you specify a list of the codecs you want to use in the media stream. The switch you are hosted by (either your Telephone Service Provider or Carrier) then determines which of these codecs match the other leg of the call it will be bridging you to, and chooses one of the codecs you provide to use for the media session. Sometimes it will be decide to transparently transcodes the call to another codec if required by the destination network. The payload in the media stream (called the RTP stream or Real-time Transport Protocol stream) will be encoded in the codec you specify, but may be transcoded on the way to the destination if required by the intermediary hop.

Among the codecs you could use, there are high definition codecs like G722 that sound really great and are becoming much more popular (this is the skype high definition audio codec IIRC), but if a call traverses the PSTN at all, that call will get transcoded down to G711U (USA) or G711A (everywhere else) and the quality will be much lower. If a call stays pure voip, there is a chance it will stay at G722 but typically that is the problem, you will encounter when calling to other phone systems that are still on the old circuit switched infrastructure.

So ultimately the quality of the call is dependent on these factors, how much bandwidth you have available, what codecs your VoIP provider supports, the quality of the path call takes and the codecs supported by the carriers the call traverses to get from your provider to the destination phone, and the type of endpoint you are calling IE: if you are calling mobile phone, a phone on another voip provider, or an old POTS line.

A POTS (plain old telephone service) line is (now sort of) dedicated from point to point. You might be surprised to learn that on the backend, most carriers use a lot of voip for internal network or handing off calls to other carriers. The old phone system reserved a 64k circuit for the entire path of the call which only contained the voice data for that specific call. This provides some quality guarantees are there that do not exist for packets traversing the public internet. VoIP on a private network with sufficient bandwidth and QoS for the media stream is always going to at least sound as good, if not better than the POTS network because the compression to squeeze the voice data down to a 64kbps was not very good when that stuff was invented. With voip you can use the new codecs, which nearly always results in a better call quality. This all depends on the provider you are using though, if they are hosting their services on AWS and using crappy carriers, your call quality will suffer.

@scandox: Here's some insight from an Enterprise VoIP perspective that you may find useful, from CiscoLive last year: https://youtu.be/zbjT3Fy4y1s

Could it be the upload speeds or QoS settings? I don't use voip, my wife skypes long distance and it seems pretty good for both video and audio and our internet is only 2ish Mb down and 1ish Mb up!

Some models of LG G5 have VoWiFi. You can make calls even when your cellular network is not available as long as you have an internet connection. Unfortunatly LG G5 H-860 doesn't support VoWiFi.

Where do you live where you can get 2 gigabit speeds at home?

It's more likely he meant 240Mbit/s

Yes sorry I did

240 Mbps. So AFAIK 240 megabits per second. Is my original notation ambiguous?

A capital B indicates "byte", and a lower-case b indicates "bit". 240 MBps is eight times faster than 240 Mbps.

When it comes to speeds in bits per second, M almost always means 1000000, and not 1048576.

Networking speeds are pretty much universally represented in bits (per second) -- "little b". Storage, on the other hand, uses bytes -- "big B".

So, yeah...

> I have 240MB/s at home ...

You're saying you have a 240 MBytes x 8 -- or 1920 Mbit/s -- connection.

Possibly OP was incorrectly confusing Mbps and MBps but in Japan Sony offers 2Gb/s for home connections.

I had their 1Gb/s plan for ~6000JPY/Month when I was living in Kyoto. Was great

How is a 2 Gb/s home connection presented? 10GbE?

Yes

Looks like they now offer 10 Gbps. And yeah, it shows 10 GbE ethernet card support https://www.nuro.jp/10g/

edit: here's a list of their supported end-user equipment, they all seem to only have 1 Gbps RJ-45 ports https://www.nuro.jp/device.html

edit2: for the new 10 Gbps service they have self-branded routers with one 10GBASE-T port (and 3 gigabit ports), dunno what hardware it's based on https://www.nuro.jp/news/150601/10g_150601.pdf

I always thought that cellphone calls were routed over ip networks and were using exactly the same low-quality codecs "real" voip is using.

Skype also has to deal with routing some of the calls trough their servers. Audio is probably crippled to save bw.

VoIP isn't getting better because vendors are not willing to delay output for better UX.

Little trolling: so you say you have 1.6Gbps connection at home?

Maybe your connection has some latency issues?

Yes sorry I'm a dufus about these things. The issue has been thoroughly gone into down below.

Mea culpa.

It's probably because most people have horrible quality microphones.

Sorry to down vote, but that's completely the wrong vector. Its a network issue (of some sort - bandwidth, latency, throttling).

Yeah, if you see the quality (or lack thereof) of microphones in old telephone handsets you'd be amazed.

How are games and voice software targeted at gaming managing to make things so much more clear?

If it's a network issue, it's likely unsalvageable because the network is just crap. We need to be using whatever those are using. It is usually crystal clear and people are easily understandable even during large gatherings (EVE Online battles).

I'm the same as the OP, and what's worse, people talk 20 feet from the cisco conference phones (because those are everywhere) and it makes everything sound completely terrible. I don't listen half the time. That is definitely a microphone issue.

If you're audio sounds great for a network game and crappy for a phone call (to the same person you are playing a game with), its because some one (your ISP, your VoIP software, the other parties ISP, or VoIP software, or any network hop in the middle), is intentionally choosing poor quality codecs for you to lower your bandwidth utilization.

I've heard directly of novice VoIP engineers unintentionally taking out corporate networks trying to "help" improve audio quality, by picking a higher quality codec. It worked great when they tested phone calls in the middle of the night maintenance window. It really sucked the next day when packets started colliding. Crappy sounding codecs sound a heck of a lot better than 50% packet loss.

Is it? Why do people with headsets sound 10x better over Google Hangout than people using the integrated laptop microphone?

Lots of great feedback here. I know from my own experience that the leading culprit of bad VoIP (and video) quality is a poorly performing ISP connection.

I'm one of the co-founders of Firebind. We use active traffic to continuously assess network quality of last mile ISP links, with a focus on sites using VoIP and video. Our software agent tests back to the cloud (combo of AWS and customer chosen destinations) every 5 minutes and makes 11 measurements by putting traffic on the wire. Two of those measurements are upload and download packet loss based on a synthetic G.711 VoIP call that runs for 25 seconds in each direction to one of our AWS hosted agent clusters.

By graphing the upload and download loss 24x7x365, we can see how good the network quality is on an ISP circuit far better than you can with a TCP Bandwidth test (which is effectively a destructive test) or anything ICMP based. And since we use active traffic, we have visibility even at 3am when no calls are being made.

I've reviewed hundreds of agent charts personally just in the last few months, many times with the customer. By far the number one issue is a bad quality ISP circuit, followed by an occasional bad or misconfigured network device at the customer site. Every once in a while the hosted VoIP provider might have some temporary glitch, but it has been pretty rare in my experience. Similarly the WAN path beyond the local ISP is usually not the problem.

Oversubscribed links are a very prevalent issue, especially with cable-modem based circuits where the upload bandwidth is a fraction of the download. Some of the things I've seen recently...

- Cable-based ISP circuit in Concord, CA whereby the upload packet loss increased in proportion to the daily high temperature - Customer site who accidentally configured their backup process to kick in at 3pm instead of 3am and hence flooded the upload path during working hours - Bad loss in both directions that was eventually traced to a bad coax line that had been nicked by a lawnmower - A cable modem with a memory leak that would gradually drop packets at a rate increasing up to almost 20% over the course of 2 weeks, where a power-cycle would "fix it" for a few days until the leak grew again and dropped more packets. - Incident with 2 Charter sites in 1 state and another in a different state whereby all 3 had transient 70+ percent download loss for three days back to our test site at AWS in US-East-1 (I sampled dozens of other agents around the U.S. and only the Charter sites saw this issue.) - And of course, the Netflix effect. Dozens of times I've seen download loss spike during prime-time TV hours, and that's the only time each day that remote site has download loss.

The best test you can do with open-source tools is to use iperf out to the cloud (to a $5/mo Digital Ocean droplet perhaps) on regular intervals and simulate the codec you care about. We built our own iperf-like solution that is highly concurrent but you can copy our settings and aim for 87 Kbps with 218 byte payloads over UDP. That should give you 50 pps of VoIP.

And of course for the shameless plug, if you want to check out Firebind, you can find a free trial link at our site. https://www.firebind.com.

I would be very interested in trying to see the results of the firebind stuff, with our stuff (fq_codel and cake based "sqm") in place on the link. Could you drop by the bufferbloat mailing list and chat with us?

Also: We use a tool that we consider much better than iperf alone, it's from "flent.org" and gives us the abilty to inject loads and measure the side-effects.

PSTN uses a digital backbone these days; it's just supersized carrier-grade digital backbone, not 'what hacky software can we get on the cheap'. Phone companies prioritise their voice traffic over their own backbone and have network admins monitoring it closely, whereas your skype calls are just 'more traffic from the proles' and get to compete with everyone else.

Reply on news.ycombinator.com