Hacker News Reader

Comment by zeinhajjali | Hacker News Reader

This reminds me of a project I worked on for a grad school data science course here in Canada. We tried to map this "digital divide" using public data.

Turns out, it's really tough to do accurately. The main reason is that the public datasets are a mess. For example, the internet availability data is in neat hexagons, while the census demographic data is in weird, irregular shapes that don't line up. Trying to merge them is a nightmare and you lose a ton of detail.

So our main takeaway, rather than just being a pretty map, was that our public data is too broken to even see the problem clearly.

I wrote up our experience here if anyone's curious: https://zeinh.ca/projects/mapping-digital-divide/

View on HN

Really interesting perspective, thanks for sharing.

I think in so many fields the datasets are by far the highest impact thing someone can work on, even if it seems a bit mundane and boring. Basically every field I've worked in struggles for need of reliable, well maintained and open access data, and when they do get it, it usually sets off a massive amount of related work (Seen this happen in genetics, ML of course once we got ImageNet and also started getting social media text instead of just old newspaper corpuses).

That would definitely be advice I'd give to many people searching for a project in a field -- high quality data is the bedrock infrastructure for basically all projects in academic and corporate research, so if you provide the data, you will have a major impact, pretty much guaranteed.

I'm in the USA with nominally a 1.25 Gb/s down, 50 Mb/s connection from my cable ISP. And you'd think that it would be fast, low latency, and reliable. Well that would be true except my ISP is Xfinity (Comcast). At least 4 times per week, I experience frequent packet loss that works with older web servers but makes most newer TCP based technology just fail. And the connection will randomly fail for 10 minutes to 2 days at a time and sure they give me a credit for it.

So anyways, I bring this up with my local government in Chicago and they recommend that I switch to AT&T Fiber because it's listed as available at my address in the FCC's database. Well, I would love to do that except that

1. The FCC's database was wrong and rejected my corrections multiple times before AT&T finally ran fiber to my building this year (only 7 years after they claimed that it was available in the database despite refusing to connect to the building whenever we tried).

2. Now that it is in the building, their Fiber ISP service can't figure out that my address exists and has existing copper telephone lines run to it by AT&T themselves so their system cannot sell me the service. I've been arguing with them for 3 months on this and have even sent them pictures of their own demarc and the existing copper lines to my unit.

3. Even if they fixed the 1st issue, they coded my address as being on a different street than its mailing address and can't figure out how to sell me a consumer internet plan with this mismatch. They could sell me a business internet plan at 5x the price though.

And that's just my personal issues. And I haven't even touched on how not every cell phone is equally reliable, how the switch to 5G has made many cell phones less reliable compared to 3G and 4G networks, how some people live next to live event venues where they can have great mobile connections 70% of the time but the other 30% of the time it becomes borderline unusable, etc.

I have AT&T fiber 1Gb/s symmetric in the bay area. It is much better than the XFinity coax/cable from before. I still have random slow down and packet loss though. I suspect the shared fiber is oversubscribed one too many times.

Oddly fitting (or perhaps that's double irony) that your "mapping the digital divide" project was derailed by the literal digital mapping division boundaries.

Here you go:

https://medium.com/spin-vt/impact-of-unlicensed-fixed-wirele...

Reply on news.ycombinator.com