Why the Internet in Sub-Saharan Africa Sucks
facebook.com
facebook.com
Kenya is not even the worst of all three, Uganda is really a disaster in terms of connectivity. In Uganda it's even difficult to get actual phone signal in most of the country. And you never have reliable Internet anyway, it works like half of the time even if you are in the few right places in the country. Building applications for this kind of environment has been quite challenging.
In a few shops in Uganda, the only way to get data back is by USB keys, so you have an actual truck going there with data to send to the shop and data to get back from the shop. We are building connectivity out of trucks and USB keys.
I routinely see 270-300ms pings to facebook. Anything on Amazon's CDN is at 150ms that as is CloudFlare. Akamai does have local nodes, as does Google but the lack of competition on the cable makes it slow.
Guess I should move to Kenya?
The Singapore and Japan zones help, but most still goes to the US.
I really hope that something like ppsp / swarm / ipfs gets integrated into browsers. I'm pretty sure almost every non-user-content file I download every day has been downloaded somewhere in my town. I shouldn't have to go all the way to the US to get it.
Still won't shake the pain that is a single connection to the rest of the world (the AU links are only used for local traffic).
10ms to hacker-news… 130ms to facebook. But only 12ms to a facebook CDN: scontent-lhr3-1.xx.fbcdn.net 10ms to twitter. 13ms to BBC.
We are already live in Johannesburg and there will be many more.
Your company's motives are not pure and altruistic. Don't pretend they are.
Huh?
Let's be clear: CloudFlare is a for-profit company and hence wants to take in more money than it spends. We do that by getting people to pay us. Our revenue is from web site owners paying us. It's pretty clear what our business model is: pay is a little for bandwidth and hardware is possible, get web site owners to pay us.
The problem comes later, as it does when any company gets big enough. Few years down the line, the board of directors has a look over the company and see's a big opportunity: all that data you're flowing around can be turned into money with some very clever thinking at 'no real loss to anyone'.
People don't like to leave money on the table if they can just find a clever enough way to put it in their pockets, and more than enough very smart people saying "hey, pay me a little bit and I'll get all that nice money off the table for you - nobody will mind, privacy is dead anyway!"
When they pay for our service they are trusting us with their DNS and HTTP traffic. If CloudFlare were to start messing with their data it breaks that trust and we lose our customers. We'd end up biting the hand that feeds us.
Most companies that see a lot of traffic, could decide to engage in shady side activities to monetize every bit of data they see. But no leadership in their right mind would want to take that sort of gamble.
It's interesting that the "evil eye of sauron" argument always comes up for discussion involving CloudFlare, but is never even mentioned for companies like Akamai. My guess is because of the freemium approach which "has to have a catch, right?". But CloudFlare is a proven business model with substantial revenues coming from people who like and trust the service. There is no catch.
ps. former employee here who joined on the idea, and still firmly believes of offering a quality service for a very affordable cost, without being deceitful.
That being said the speed boost is very obvious :)
When it's with HN it shows something like the twitter feed of @HNStatus pointing out the CloudFlare server as Johannesburg with a request id (I think).
When it's with reddit it shows the reddit mascot crushed up with his orange beady eyes over a pile of orange stuff. Scary stuff.
Next time it happens, will screenshot and send to cloudflare.
I saw a cloudflare gateway timeout on HN within the last hour, and the twitter feed had a recent post excusing some technical issue.
At first, I wondered why the HN Feed never explained the downtime. Then it made sense, it must be cloudflare.
Given that CloudFlare is everywhere & on everyone's websites, it's a pain when it behaves this way affecting users all over a continent.
Uptime websites also report the site as up.
I have been wondering if it could be something with limits of the number of multiple connections, or even requests, affecting the AJAX requests in the background, or some kind of unfortunate timeout limit in the javascript layer.
Not sure, but with this info, I'm of course thinking more about the lag.
Would be interesting with more details on how it affects typical AJAX apps!
JHB means usually going to Cape Town first, then up the west coast of the continent, wheras Mombassa uses SeaCom, going around the east of the continent. That saves a lot of time from a pure distance perspective alone.
Indeed. I was responding to a comment that was discussing JHB though.
SA traffic is split - some of it goes round east (Seacom), some west (SAT3). Ends up being roughly 200ms east or west.
Source: Got my interwebs over said cables for years
There is some slight chance that Amazon or Google might introduce compute cloud nodes in Africa... as soon as that happens, we'd love to extend our presence there. And if that's not forthcoming in the near future, we'll definitely explore other hosting options (as we've done in India for now).
I lived in Tanzania for a year, and while the internet connection there was not perfect by any stretch of immagination, I had way less ping than this (100 ms to London, where my company's servers are, 120-150 to Boston).
For reference, that is: aerial to my ISP two kilometers away, then fiber to Dar Es Salaam, then Seacom to Mombassa, then Seacom again to Marseille, then terrestrial fiber to London via the channel).
This smells a lot like contention on the seacom link (the big fiber optics carrier that goes up the east coast of Africa through the Suez canal and accross the mediterranean to Marseille). I would be interested to see a traceroute.
Most of the times ISPs in west Africa don't buy enough upstream traffic on undersea cables, creating contention. I can't blame them, it's expensive, but it's mostly an economic argument, not because the servers are too far away (also, most ISPs run pretty agressive caches of everything under the sun).
the eassy link covers much of the same route: http://www.submarinecablemap.com/#/submarine-cable/eastern-a...
They are "working" to get it into Apache Traffic Server.
For simpler applications you can drop parts of that, for example just RTP over UDP if you don't need retransmissions or encryption.
I will probably benchmark everything later and share it with everyone.
Akamai could help, depending on the use case. If the content isn't cacheable, there's not much you can do. They do have a service that keeps open TCP sessions from certain POPs back to your origin, thus cutting down on TCP 3-way handshake time. Clients connect quickly, locally and then send the request because the origin connection is already established. That service is pretty expensive compared to CDN and they don't push it very hard. I used it when I ran an ad network where latency mattered (and could pay for this feature in an explicit increase in conversions and impressions).
https://blog.cloudflare.com/keep-alive/
https://support.cloudflare.com/hc/en-us/articles/200168346-B...
https://aws.amazon.com/blogs/aws/amazon-cloudfront-support-f...
while I certainly love the excitement this has generated, it's not as easy as "we'll PoP Kenya! We'll PoP Tanzania! We'll PoP all of Africa! 50ms for everyone!"
you would think a place like Djibout with five submarine cables and terrestrial routes into Ethiopia, Sudan, and others, would have a vibrant peering fabric or CDN nodes.
Level3 just put a CDN node in there, and Telecom Italia Sparkle has been there for a while. AFAIK the CDN node is completely under-utilized and Sparkle provides high cost MPLS services for private and governmental outfits; not as much general IP transit.
More work needs to be put in getting carriers to peer at existing points of presence, and costs have to be lowered any way they can.
Plus you have TCP preconnect as a lightweight way to avoid that initial delay.
At least that's my experience with anything cellular-network based.
There are startups whose plan is to create local network coverin city for high bandwith content, without relaying to the outside connection as this is mostly bandwith limited. So called alternative network of catche servers and wifi, as often even watching youtube is pain depending on the time and location.
From what I've seen running an HE tunnel through London, Youtube and Facebook make up the bulk of IPv6 traffic (as expected). At least those networks are the only thing routinely dedicating 10%+ of the total inbound traffic to IPv6 according to my home network flow analysis (I think it's about 18% IPv6 usage on a monthly basis).
Huh? I thought they're Apple, Inc.
If I had a couple billion dollars to blow, I'd be spending it on that. Though, maybe that's why I don't have a couple billion to blow =)
This glosses over the fact that your service even can geo-scale well over high latency and high jitter networks.
The profits from serving ads for consumer goods will be pretty slim though.
Like any business, we'll get the low hanging fruit first, then move on to the higher, harder to reach branches. We'll improve Internet service in Africa, but not before the time is right.
Check out the prices from vendors. Also, check out the profits. Also the margin! from the ISPs. Take into account the Capex too. Everything will seem odd, except the volume of data consumed per user [which is related, or better said limited to the maximum actual bandwidth]
Safaricom which is well known for its MPESA platform is the most profitable firm in sub saharan africa [not south africa or nearby] and it's all because of data.
Datacenters are difficult:
- Higher cost of electricity
- Unreliable electricity (blackouts and under-rated voltage)
- Costs of acquiring equipment/servers
- Heating/Cooling is on the harder side
- Lack of competition means pricing is farther up on the higher end
- Back down to cost of data, which is really expensive per GB or to lease the fiber required.
Another thing is there is a large latency chunk added as the vast majority of consumers use 3G which adds another chunk of latency in. This is slightly disincentivising to add in local datacenters.
It may be worth noting AWS is actually run in great part from Cape Town. Everyone uses their data centers in Virginia yet does anyone wonder why AWS doesn't offer anything in South Africa when it's being built here?
Given that each one of these has the vigorous majority of their users outside the US and most have substantial non-US infrastructure build, it gets a little strange to label them "US-based".
But I think it's fair to point out that if performance lags to the extreme it may open market opportunities for local competitors to differentiate themselves with speed. Local hosting options are limited, however, making things more challenging for indigenous entrepreneurs. I was amused and appalled to find that many "Kampala web hosting" companies are actually themselves hosted in Los Angeles.
So: not much is hosted in-country. Very little traffic terminates domestically outside caches.
The recruiter who showed me this was very proud of google's accomplishment and indeed it was impressive to see the spikes erupting from the east and west coast of the US, Europe, Japan and India.
But I watched quietly, sadly and solemnly South America and Africa. There were modest spikes in south africa and brazil. Not many anywhere else.
Both continents are quite populous.
Among the reasons for the choices I've made in my life and my career the last few years is that I have come to regard the vast majority of the code I've written in my long career as a developer as being of no real lasting value to anyone.
Instead I wander around downtown and oldtown portland seeking out crazy people then finding some way to lift their spirits somehow. It can be a little frightening as I specialize on the particularly violent sort. Those are also the most cruelly isolated:
"LEAVE ME ALONE! LEAVE ME ALONE! WHY DON'T YOU LEAVE ME THE FUCK ALONE?"
The gent he was accusing had no clue.
"Hi! My name is Mike. What's yours?"
"Jerry! Pleased to meet you."
Does it have to be so hard?
My point is that those google search spikes would be erupting all over South America and Africa were someone to take responsibility to enable them.
I intend to contribute somehow, someday, in part by setting up some nameservers in cybercafes. There are actually lots of them, cybercafes are quite popular in the developing world.
The issues which prevent those spikes you'd like to see on the maps at Google are education (primarily literacy, especially English literacy), access to devices, and culture. Access to devices is being solved by cheap smartphones becoming obtainable to some degree and constant improvements to data networks in African countries, but the other two have no easy solution.
EDIT: To maybe make it a bit easier - consider someone in a village (with a cellphone and access to a data network) who wants to find out why something happened to the carrot he was growing at home. All of us would pop open Google and search about some information about carrots. A guy in a village would head up the road to talk to his friend who's been farming for 50 years and knows about carrots. If you tried to tell him "hey, you could ask Google that question" the only reply you'd get would be "I don't really know about that".
I've had someone come to me to ask me to look up bus fares on the internet for them. They'd hand me a piece of paper with the URL for the bus companies tariff page. Not too interesting until that same person later pulled out his Windows Phone smartphone to make a phone call, and he had full internet access and a good mobile browser on that phone.
One of the things that hurts progress in my country isn't really a lack of education as much as it is ignorance in the strict sense. It's funny, because a couple of years ago, a trans-Pacific cable was laid close to the island, but the decision was made to not invest in tapping into it due to the cost--even nowadays, people are interested in improving connections, but it isn't a priority. They really haven't experienced the internet, at least not in it's full capacity, so they don't know what they are missing. They know about it from what they see on T.V. ("viral" review shows targeted at baby-boomers who still watch T.V., I suppose) and social networking, so simply, they don't know what they're missing. I recall when I finish high school and moved back here to the US for college--I basically missed the dotcom-boom days of the late 90's and early to mid 2000's, the internet was not just a convenience, it completely changed my living habits.
But the so-called "other three billion" really can't know this because they haven't experienced this, so not only do you face real obstacles, but you face lethargy as well. At least for Palau, I feel like that is one of the main reasons internet access is pretty subpar there today.
[*] I do admit that this might not be relevant in parts of Africa. For one, Palau is a pretty small place (no more than twenty-thousand people), so attitudes are probably much more homogeneous (with that attitude towards the urgency of improving internet infrastructure) compared to hosts of nations with millions of people in sub-Saharan Africa.
There's the electricity issue (we call the rolling blackouts "load-shedding") which affects the cellular base-stations and telco exchanges as much as anything, but also another more obvious issue for content delivery: the local loop.
For the past 15+ years (as long as I've had Internet access), the incumbent telco (TelkomSA) has not been forthcoming with upgrading or maintaining the ADSL infrastructure. Finally, there seems to be traction with FTTx providers, but this seems limited to affluent pockets of a few thousand people in the country. All the new fibre offerings serve the same locations.
While it's hopeful this will improve the situation in the medium-term, the backhaul from ISP to eyeball is oversubscribed at least 30:1 for ADSL, with equipment being replaced by (IMO) inferior network equipment. Additionally, there are reports of automated line-conditioning software which should improve local loop connectivity, but I've seen DSL sync issues only since the software has been implemented.
The (ex-public, privatised under Telkom) Exchanges have been neglected and have recently been divided into three groups: Earners, Maintainers and "Lost Causes". The latter two will not receive upgrades from ADSL1 and the Earners will likely migrate to newer technologies. This is largely due to copper-theft and vandalism of exchanges/cellular base-stations (which are oversubscribed enough that wireless technologies routinely sustain 3000-20000ms latencies). The oligopoly running cellular networks have no incentive to improve this, since the Regulatory Authorities don't really seem to push competition as well as we'd like.
TL;DR: There are issues, but rose-tinted glasses make me hopeful in the medium term. Claims are that wireless will fix everything seem bleak, since base stations become oversubscribed faster than they can be built/upgraded (and the backhaul from them is measured in megabits/s, rather than gigabits/s charged at ~US$0.15/MB).
Our mobile networks are in pretty good shape:
* Anyone can buy a prepaid SIM from any supermarket and access the Internet within 10 or 20 minutes. Last time I visited France, it took me 4 days to achieve a similar result.
* Mobile data is cheap when buying in bulk. I pay USD2 per gigabyte.
* LTE coverage is growing fast and RTT is good. For example my ping time to google.com is 30ms.
Conclusion: Private ownership and free markets lead to good service and low cost. ESKOM proves that state ownership of companies and monopolies lead to bad service at high cost.
Even India cannot provide reliable electricity to its cities. Rural India has scheduled blackouts for 6-7 hours a day.
It's never really made sense to me that we even have a nationwide grid. Why not local grids with local power plants? That's what islands use. While there is economy of scale in a big power plant then you need a bigger grid to consume its power.
There are many ways to generate electrical power. I expect many aren't used because power companies can't earn money from them.
I expect you'd have fewer scheduled blackouts if the rural areas had their own local generators.
I am not an expert on this topic but I am Indian so I know that corruption and apathy play a big role too.
The solutions are largely straightforward, for example "inclusionary zoning" would require that new housing developments include a certain proportion of affordable housing. But inclusionary zoning is forbidden by Oregon State Law.
In what way does that law serves the needs of the people? Well that was not considered by the legislators, no the developers lobbied them, no doubt set them up with hookers and blow so as to maximize shareholder value while throwing lots of young people out on the street.
The country has around 40GW of installed capacity, dating from over-investment before 1994, but the state opted to both not build new stations and not carry out any maintenance on the existing stations for years so about 10GW is now offline and unusable. Hence the frequent power outages.
Speaking as a US native, the reason we'd look for carrot information on the internet is that none of us know anybody with carrot experience. If I knew someone who knew about carrots, they'd see all of my carrot questions before I ever asked the internet.
So if carrot info is all you ever need, great -- but otherwise, you are going to need nearby buddies who are experts in many, many other things. In fact, you're going to need far more buddies than you can ever actually have, due to the limits of the number of social connections any of us can have (not to mention the limits that time itself imposes on the conversations we'd have to have in order to maintain those relationships even if we could keep them all straight in our heads).
Google's primary utility isn't in making you a perfect expert on anything in particular -- it's that it allows you to become sufficiently expert at anything at all in a really short period of time. Even an army of friends, with the most varied experience you can imagine, cannot come close to matching the breadth of expertise available via the net.
What's going on here is that Africa isn't really reaping the benefits of specialization of labor in the way the civilized world is. The guy in the example isn't even growing multiple carrots! When I get a problem with my roof, I'm not going to try to fix it myself after a crash course in Google-based roof repair. I'm going to pay someone else to do it for me. I do the same when I want to eat fish or goat, or fix my car. The whole point of a developed economy is that you don't know how to produce the stuff you need. Instead, you produce what you're good at producing, and everyone reaps the gains from trade.
The title of this piece strikes me as a little over-specific. What in sub-Saharan Africa doesn't suck?
I'm surprised you didn't notice Russia and China not spiking as much either.
As for Africa, it is probably a case of not having gained a critical momentum for the kind of infrastructure the article talks about. This doesn't mean it won't happen - while we don't build CDNs at the place I work, we do build basic infrastructure for access control in wireless and mobile networks, and we are definitely seeing more activity and business from African regions compared to how it was earlier.
Most of the code I've ever written, if it will build, won't run on today's computers. While it could be made to by porting a few years after that again it won't work.
Consider Homer's Iliad and Odyssey; you can find them in any library and most good bookstores despite having been composed almost 3,000 years ago.
Huh? South America has good overall Internet penetration[1] and Chile is going to 70% right now.
[1] https://en.wikipedia.org/wiki/List_of_countries_by_number_of...
So South America would always show up as a few big coastal cities with vast emptiness inside, even with huge search penetration on Google's cool visualization. It really doesn't tell you much either way without raw data.
My take on what to do about that is to set up servers in both north and south that have website mirrors, as well as a geographically-sensitive nameserver. I don't have a clue how to do the GeoDNS but surely it is a solved problem as big sites require it for load balancing.