Major P2P issues in Israel and possibly other Middle East countries | Hacker News Reader

Major P2P issues in Israel and possibly other Middle East countries

github.com

283 pointsbabuskov135 comments

I think what we’re seeing here isnt Valve messing up but rather the middle east conflict expanded to cyberspace and spilling over to impact civilians. Look at the timing and affected countries. China isnt also exactly known for free internet.

WebRTC works as fallback. WebRTC is encrypted and cant be used for much else.

STUN in the otherhand is unencrypted and the protocol itself can be used for DDoS reflection/amplification. I would not be surprised if this is somehow weaponized and/or blocked/analyzed in real time that then breaks the connectivity.

I think you have that backwards, WebRTC doesn't work, and STUN does.

I think you have it sideways. STUN [1] is the NAT traversal / "NAT hole punching" process that allows peers to discover their public IP addresses and establish direct P2P bidirectional UDP communication. WebRTC depends on STUN to establish P2P communication. You may be thinking of TURN [2] which amounts to routing traffic through an intermediary node that is visible to the two peers.

[1] https://en.wikipedia.org/wiki/STUN

[2] https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_...

STUN/TURN is basically icanhazip for WebRTC. STUN gives you your public IP:port. TURN is the same, but the returned IP:port is the one that had been dynamically allocated to you at time of querying, rather than the actual ones.

WebRTC clients take that STUN/TURN response and send to peers through out-of-band, through e.g. a lobby server chat mechanism, to set up the connection. This allows NAT table entries to be created as if they are outbound connection at both ends.

You can't make P2P connection with STUN/TURN alone. STUN/TURN is just a tool required for WebRTC.

TURN is the last resort and isn't just signaling. It carries the traffic as well.

If you can make all the STUN servers fail from the perspective of the clients, you could hypothetically force them to use TURN servers that are more centralized and easier to spy on. STUN negotiates pipes n:n. TURN is closer to n:1.

> force them to use TURN servers that are more centralized and easier to spy on

Webrtc traffic is encrypted as it travels through the TURN servers, isn't it? Sure, you get some which-ip-contacted-which-using-what-service metadata, but any active middleman able to mess with STUN traffic already has that.

It could just be that someone's fucked up a setting somewhere. I mean, the reason WebRTC has loads of options for 'interactive connectivity establishment' is because it's common to see users behind NAT, users whose NAT cant be traversed with STUN, IPv6 being broken, UDP getting blocked, TCP ports other than port 443 getting blocked, etc etc.

If a country's ISPs use CGNAT to avoid giving users precious IPv4 addresses, and world events made the ISPs turn the security settings up to 11, STUN just stops working.

The traffic is encrypted, but this makes it a lot easier to acquire if you have some way to break it.

And metadata plus encrypted traffic fingerprinting is enough to provide huge signal to an intelligence agency.

> TURN is the same, but the returned IP:port is the one that had been dynamically allocated to you at time of querying, rather than the actual ones.

I don't know you mean by this, but I think you're confused. I have implemented STUN, so I know how it works. AFAIK, TURN doesn't reveal an address/port any different from that revealed by STUN, and cannot, because its discovery feature is STUN. (Also, a typical home user has only one internet-facing address, not a dynamic one plus another one.)

Rather, TURN provides a STUN address/port discovery service and a data relay service. The relay is for cases where two peers wishing to connect are both behind difficult NAT, meaning there is no quick and reliable way for them to directly connect even when they have their STUN results. So instead of connecting directly, they communicate through the relay.

I admit that I only have rudimentary understanding, but: my understanding was that TURN uses a modified STUN format that returns the address/port on the peer facing side of the TURN server, a la address of a hotel room or PO box, of querying user. My point is that STUN/TURN(especially STUN) are not encapsulation protocols for WebRTC, but just means to determine(or get assigned, so TURN sort of is encapsulating, by being a transparent proxy) client's own public IP/port because there is no way to do so than by asking someone with a global IP.

IPv6 and minimal assembly-written network code going without niche and complex features.

We do P2P in our networking software and this is why we do it all in band instead of using STUN, TURN, or other common methods. Those get blocked and they’re also often insecure.

STUN has mitigations now against being weaponized but it’s still a shit protocol. The fact that neither STUN nor TURN contain any way whatsoever to accomplish any kind of rendezvous without yet another signaling path boggles my mind given how easy it would have been.

> The fact that neither STUN nor TURN contain any way whatsoever to accomplish any kind of rendezvous without yet another signaling path boggles my mind

Interesting. Can you expound on this a bit? How does ZeroTier do it?

ZeroTier has "roots," which are nodes that relay packets and also tell you what your IP info is. Everyone in the world connects to a pool of these.

Other than relaying and STUN-like IP info reflection, they're dumb and do very little. They can't see your traffic or other information or even what virtual networks you're on.

Once both sides learn their external info, they communicate via the root to arrange P2P rendezvous. If both have IPv6 they use that, but still do a hole punch due to stateful firewalls. But with V6 it works almost 100% of the time. If one or both have V4, they do more cumbersome V4 hole punch maneuvers.

Our next-gen product, which is still in pre-release and has been shown only to some enterprise customers, is called ZeroTier Quantum. It's called that cause it's built on PQC (pqNoise to be exact) but it's also a full-scale reengineering of the whole system. But it still uses very similar techniques. Everything is in-band. No STUN, TURN, or even DNS dependencies.

I developed WireGuard-like P2P VPN on WebRTC and performance was great 300Mbps+

I think we can align on WebRTC for realtime P2P gaming and enterprise networking and more, instead of IPs base solution as end-users will not need to figure out firewall issues and IPv4/IPv6 differents

I know I'm just preaching to the choir here but my favourite thing about open source/published source libraries/applications is discussions on bug reports/pr's like this.

It's just something so heartwarming of multiple people coming together to describe their symptoms, workarounds and theories of what could be causing it.

GitHub discussions used to be so much higher quality though when the platform was for professionals. Now, I see so many discussions that devolve into practically being reddit/4chan threads. Another reason to leave.

I feel like it's gotten more professional. 10+ years ago people were dropping the hard R in pull request reviews, now everyone is acting like LinkedIn-speak and Stars will get them their next job

...What? Is this the Linus Sebastian misconception of what the hard R is?

Yeeeah I'm pretty sure I've never seen a hard R on a PR.

I'm guessing they are referring to a certain synonym for idiot/moron/imbecile/cretin/dolt/etc. which fell off the euphemistic treadmill

Yeah, that's the Linus "hard R" (he thought "hard R" referred to "retard"), but it's just wrong. "Hard R" is "nigger", in opposition to soft r ("nigga"). I don't think there's even a question, that's how hard/soft has always been used. Anything else is just confusion, I think.

That would be the “Linus Sebastian misconception”

Ok, but I can't possibly be the only one who has no idea who that is, let alone what misconceptions they have.

There is a thing called Google, "Linus Sebastian Hard R" is full of information.

https://www.youtube.com/watch?v=MFDiuBomSuY

Only on those posted to social media including Hacker News. There is no devolving into memes for niche discussions only interested parties know about.

Don’t blame Github for getting spammed whenever an issue reaches the front page.

Not only. I see it across all of GitHub. Spam, +1 comments, feature begging are all particularly common.

Feature begging on GH has been a thing since forever, I remember plenty of it 10 years ago.

Hell, I remember feature begging on developer mailing lists myself 20+ years ago. (To be fair I was 13 at the time)

To be fair, a lot of the users spamming and feature begging on github today are 13 right now

I wish HN would ban posting links to issue trackers with comment sections, like lobsters has done. Although the spam volume from HN and reddit is pretty small compared to that from youtube reaction video influencers

    > when the platform was for professionals

When was that?

Eternal September.

Title does not match GitHub issue: "Major P2P issues in Israel and possibly other middle east countries"

Wild hypothesising here on HN but if you read to the end of the GH issue users have been reporting that STUN has been failing (i.e. no P2P link establishment, fallback to high-latency relay servers.) Multiple users have been able to work around the issue by manually substituting older Valve WebRTC dlls. I'd love to read a postmortem from the Valve devs.

Valve fascinates me because the devs there occasionally seem to be simply the best on earth in a given field, but despite that, bizarre bugs will persist for a long time. My favorite was how steam in home streaming from a PC to a steam deck wouldn't work if the steam deck had an Ethernet and wifi connection - one of the connections had to be disabled or the stream would always crash.

Maybe they need a few average devs there to spend time sweeping up behind the paragons that are pushing the envelope into these features existing at all.

Valve famously has a very flat org structure so it's possible that that problem just isn't sexy enough for someone to pick it up on their own, without being told by a higher-up.

I wish they offered remote; I'd happily work there doing those sorts of unglamorous bug fixes. High-reliability engineering is my jam.

They say they have a flat structure. People who have worked there, despite some axe-grinding, indicate otherwise.

grug tribal animal, tribe always there even when chief say is not

People keep blaming the flat org, as if conventionally-organised companies never had any bugs or never focused on very visible and marketable features rather than bug fix.

In fact, the flat org allows a random person to work on a niche bug management doesn’t seem to care about, which wouldn’t be possible if you had a boss breathing down your neck.

My favorite bug family, that somehow to sneak in every time, is how their react frontend (or whatever the store runs) manages to semi-crash and the controller inputs are no longer recognized.

I kind of hope at least they'll fix such issues permanently before the steam machine release.

That is the bane of my existence. Steam's UI is so slow to react due to its web roots, that I feel like people must be insane to think that Steam is somehow this great app. It's terrible.

I shop on GOG.

Steam was rewritten in React relatively recently. I think most people formed their opinion of Steam back when it was mostly developed in VGUI, the same in-house native UI framework Valve used in games for stuff like the Half-Life 2 title screen and the TF2 server browser.

> Steam's UI is so slow to react due to its web roots

I actually wouldn't blame the web roots. Battle.net is also a CEF based launcher and it feels so much more snappy compared to Steam. For some reason Steam just feels really slow.

The company is very small, and they're doing a lot with what they have. Steam alone is full of arcane features that I keep discovering. There's a lot of backend stuff. They're making games and hardware.

Perhaps some of this is contracted, similar to the Linux compat and drivers, but it's still impressive to me, compared to the orgs like Spotify, order of magnitude larger with barely any features at all. (I understand there's legal, huge backend, and I didn't see many bugs over time, but still)

The company makes $50,000,000 for every employee each year. It can afford more employees.

But would it actually help. More employees means more communication and overhead. Lean organisations can move much quicker. Part of why valve can do what valve does is how lean it runs.

To summarize this conversation:

"Steam is bad because it has few employees."

"Steam can afford more employees."

"Adding more employees would make Steam worse."

Good talk.

No doubt they can, maybe they should. My point is that I'm more impressed with the results than many other companies that I know.

The number of developers needs to grow log(n) to the number of users to handle all error reports. Valve is way under the log(n) of user.

> in Israel and possibly other middle east countries

Why did you leave this part of title out? For clicks?

You've been here long enough to understand that would exceed the title character limit.

I just tested it. Copied&pasted the original title into submit form.

Nope. Right within the limit.

It's so funny when people come up with these arguments so confidently and then seeing them getting disproved so quick. Bro was never more glad there's anonymity on the internet

Your comment reads like a tweet. But I agree with you!

Or maybe because if there is one thing the world doesn't need, it's yet another thread devolving into flamewars about the Israel/Palestine conflict?

Then don't make this thread. One can't discuss an issue about Israel/middle east's internet connection while pretending the war doesn't exist. Technical issues don't float in a perfect vacuum sphere.

> Technical issues don't float in a perfect vacuum sphere.

I agree. But if there is a chance to not immediately draw in the wrong crowd... I prefer if people take it.

What an absolute dud of a submission, I can't believe this got so many upvotes. I guess people saw "Valve" in the title and figured it must be important, even though the content of the issue doesn't even line up with the title.

The rabbit hole started as a major P2P issue in Israel and possibly other middle east countries and further investigations revealed it seems to be a worldwide problem.

worldwide means israel russia and china so far. all countries that dont exactly like internet freedom and have a long history of spying and censorship. this might be a side effect of some government policy against p2p networks designed to make it harder to bypass censoring isps.

Mmm im in China and played a third party game through steams Spacewar dev game (enabling steam p2p i think) like 3 weeks ago and it worked fine.

this looks like not Valve issue. the problems noted seem to indicate only countries which very aggressivly scan and filter connections. P2p is very sensitive to this.

SDR is a relay network, and encrypted, so like onionrouting etc.

its well known malicious actors can abuse it by publishing a p2p game and running coms over SDR via that game...

you can imagine that people want to inspect this traffic in these regions..

The title make it seems like it's broken everywhere...

This Steam problem has affected all countries worldwide. There are confirmations on twitter from european, japanese, and other residents of "free" countries. The fact that only israelis, chinese and russians are complaining doesnt mean its related to their governments. Its just that in "free" countries Steam servers (SDR) are located throughout the country, while in "non-free" countries these servers may not exist at all. For example, to play with a neighbor both russian players need to connect to Sweden. I hope you know the geography and can predict the ping depending on how far a russian player might be from Sweden.

interesting, people speculated that Street Fighter6 went from P2P to relay a few months ago on one of the updates. never wouldve thought it would be actually a valve issue

The age of people all across the world being able to just connect to each other other the internet is coming to an end. I wish the internet was still a business backend and hobbyist playground but I'm not sure it ever was just that.

Is this a bug on Valve? Or is it simply a case of "My ISP is fucking with my internet traffic and they won't admit it please help me"

Reading the github thread points to a case of: "My country's governemt mandated it's ISPs fuck with my internet traffic, but steam P2P stuff used to not be affected but now is" across mutiple countries. People have found it works again if they roll back some of steam's dlls so Valve can probably fix ir.

@dang, title should be updated:

`Major P2P issues in Israel and possibly other middle east countries`

Hm, I have always wanted to use this to play couch co-op remotely but is this even the same "service" that provides that?

Looks like they tracked it to a steam update in March, and there's a workaround for at lest 3 games that involves all players copying steamwebrtc.dll to the game's ./binaries folder.

Paging Fletcher Dunn

Whats he going to do? Stop the Israel conflict?

Nah. Fix the bug in their NAT code and/or route around bad network performance by deploying more relays in the Middle East with different hosting providers.

https://partner.steamgames.com/doc/features/multiplayer/stea...

I blame Bricks and Minifigs

My unpopular opinion: Valve is basically a parasite or a landlord. They've been so successful it's hard to imagine a world without them, and they say "you gotta give the parasite its due" and we believe them and comply.

It's been kept around because they treat their customers ok, but they absolutely exsanguinate their developers.

And their engineering culture is... odd. They hire senior people and then let them all fuck sound aimlessly. Their APIs are terrible, their infrastructure is all over the place, they still have patch Tuesdays. But because they are the landlord that owns every house in town, what are you going to do, not pay rent?

Gabe is out there cruising the world in a billion dollar yacht, eating thousand dollar meals. All that came off the backs of developers who actually make the games.

> It's been kept around because they treat their customers ok, but they absolutely exsanguinate their developers.

This is true, but "treat their customers ok" goes a long way. When everybody else severely abuses their customers, the one company that doesn't generates a lot of goodwill.

I'd question the idea that they treat developers poorly. Epic Games Store exists and Famously beats Steam (and others) over the head by charging only a 12% fee

Hell, they even buy timed exclusive access to certain games

And yet. Steam persists

I lack an informed opinion on the matter but I have to wonder what you think the one thing has to do with the other? Developers have very little choice but to go where the customers are.

Why aren't the customers going to Epic Game Store? It's the PC, after all. It's explicitly not a walled garden

Who knows? Presumably because Steam hasn't done anything to drive them off, they've generally been satisfied with the service, and the titles they want are available. At least that would be my guess based on my personal experience but I assume Valve has a much better grasp of their audience than I do.

The Epic Game Store is just kind of mid. The app feels spammy, the game selection is less, and it doesn't really offer anything over the existing options beyond the monthly free game gimmick. If they want customers to head there it needs to be better, not just good enough.

EGS has yet to reach the feaure parity of Steam from 15 years ago.

As a user of both EGS and Steam, the one thing that EGS doesn't have that Steam does is multi-threaded downloading. Steam can saturate any connection I've thrown at it, whereas EGS gets a few-hundred mbit/s and saturates a single CPU while doing it.

Perhaps game devs get a whole bunch of "gee whiz" features from the Steam Platform that Epic Games doesn't provide, but I -personally- couldn't care less about those.

You don't need multiple threads to saturate a gigabit connection. Even many embedded SoCs can do it.

That said, Steam has a rather absurd CDN.

> You don't need multiple threads to saturate a gigabit connection. Even many embedded SoCs can do it.

What I've observed is

* when Steam downloads are in progress, between four and nine logical CPUs worth of processing power on my 32-way Threadripper are being used and zero logical CPUs are running at 100%

* when EGS downloads are in progress, exactly one logical CPU on that Threadripper is pegged at 100%

It's true that you can do gigabit downloads without having a multithreaded downloader. [0] But it seems to be true that the two biggest PC-game-store clients absolutely cannot... for whatever reason. Given the prevalence of gaming machines that have CPUs with four or more logical CPUs, I expect it's not really worth the effort to make whatever Steam is doing single-threaded, or whatever single-threaded thing EGS is doing fast enough to saturate a 1gbit+ download.

[0] One widely-deployed example would be SSH/SCP.

I go to EGS once a week or so just to see which free game is on offer, and the experience is only barely tolerable.

If you're giving away free games and can barely manage to attract people to your storefront, you might be doing something wrong.

In their defense I suppose, most other competitors weren't much better. I don't think anyone misses Origin, and you'd have to pay me to spend any amount of time on Ubisoft's storefront. Only GOG comes close, and they earn a lot of good will in other ways.

The epic games launcher that famously takes 46 seconds to launch. It’s cost them 100s of millions and they refuse to fix it.

The Epic store is horrendously slow though. I bought a few games there but in practice the client is just so slow that I avoid it if I can.

Having worked in the games industry for long time, everyone is constantly trying in vain to escape the 30% tax.

It’s fascinating how (mostly western liberal) game developers argue in favor of 99% taxes for general population “for maintaining infrastructure” and yet cannot fathom paying a fee for maintaining actual infrastructure that is necessary for their business to function.

I'm not familiar with whatever strawman you're invoking here.

By "everyone" I mean game studio owners. They're desperate to not pay 30% to Valve / Sony / Apple / whatever.

The vast majority of people that work at game studios don't really care about that, they see a shrinking fraction of the profits of their employers and worsening conditions.

It makes a lot of sense for businesses to seek to reduce their costs wherever they can. But, from what I've heard about brick and mortar distribution, you used to pay quite a bit more and get a lot less than what Steam gives you.

From what I can tell, that 30% cut gets you -for the rest of forever-

* distribution for both the current version of the game and some number of older versions you choose to make available [0]

* a place in their searchable games index [1]

* "cloud" storage for your players' savegames

* basic forum and blog hosting for discussion of and news about your game

From what I could tell as someone who used to buy games in retail stores, in a bricks and mortar distribution unless you were -like- the Starcraft/Diablo/Warcraft boxed set, you got like maybe a half year of time on the shelf. I've heard folks say that you had to pay a 50->80% cut for that.

[0] Valve will even distribute games that don't work anymore. This is both good and bad, but Steam's no-hassles refund policy combined the existence of unofficial patches that make games work on current versions of Windows make me generally fine with charging for and distributing games that no longer work as-is.

[1] ...at least until the wrong horde of pearl-clutching busybodies demand that credit card companies require your game be erased from the commercial world because it is art that discusses those busybodies' bugbear du jour

> Epic Games Store exists and Famously beats Steam (and others) over the head by charging only a 12% fee

https://www.fortnite.com/news/fortnite-developers-will-soon-...

Unless you're inside Fortnite, where Epic takes a 63% cut of any 'in game item' you sell, and you don't have a choice of storefront inside the game.

Rules for me, but not for thee, so sayeth Timmy Tencent as he collects his next ten cents of revenue from a twelve year old.

Totally agreed. I'm building a Steam competitor, that's web-based (WebGPU/WASM) as well as cross-platform. Light on games atm, but the goal is to replicate over time virtually every feature Steam has to offer, as well as more. You can check out a preview of the portal here:

https://gameselect-knvxf8av.manus.space/

What lets you host Monkeyball like that. Are you going to port Xonotic to WASM?

Starting a sustainable steam competitor with piracy sure seems like a great idea!

The background music is ripped from the PS5 home menu too.

It's how Spotify started. And CrunchyRoll. And slightly metaphorically Uber. And in a way Netflix. Turns out you can't scale if you never offer anything of value to customers and therefore never get customers.

He already did lol, just didn't update the main website. Doesn't work very well for me, but it's something.

https://www.reddit.com/r/xonotic/comments/1tyqx5w/i_ported_x...

Eh, Steam is kind of like the liberal democratic US empire. It may be evil in a lot of ways but it could actually be a LOT worse. We may actually historically be very lucky to have had a non-shittificationmaxxing games platform for a couple decades, just like we were lowkey lucky that the world was briefly ruled by a somewhat democratic country.. Enjoy both while they last, may not be around long.

> the liberal democratic US empire. It may be evil in a lot of ways

May be? It's absolutely evil in a lot of ways. It's an active participant in multiple genocides at the moment. And has been for a long time.

I guess it could be worse, but being stuck laboring under Saruman's orcs and pointing at Mordor and going "At least we aren't over there" isn't exactly a defense of the situation.

Developers, like gamers, are the most oppressed people.

Reply on news.ycombinator.com