The Curse of Dialup World
allenpike.com
allenpike.com
Now that time I hadn't verified that our backups were working and lost all of our data, that was a little less good.
[1] not a typo, it still exists, though I doubt anyone would use it over MySQL, postgres, or sqlite. https://en.m.wikipedia.org/wiki/MSQL
Used Windows NT for web hosting and Slackware Linux with qmail for mail hosting. Sold out at what I thought was the top - which was a year or two too early but that's OK.
When 56k came around you had to do ISDN, but for many rural ISPs the tariffs for ISDN was cost prohibitive so POTS was the only option.
I worked for a few ISPs and negotiated telcos to bring in redundant OC12s to support the need, allowing them to use our data centers for the local switching. We never needed that capacity ourselves, but as it was before they started using multiple colors per fiber that was the smallest they would run.
I also started out with stacks of external modems and NT RAS.
I remember that the plastic on stacked modems would blacken due to heat in about 6 months.
I dealt with sendmail, bind, etc and didn't do the account provisioning but it was written in Delphi.
As I had written an entire 7 digit double entry accounting system in college in two weeks backed by dbase, this stories excel and paper based system seems like an anomaly for the time.
NT we hosting was a pain, but front page did make it a popular option you had to support.
While I did introduce Linux, we were mostly on DEC Alphastations because you could support a lot more load for less money than the sun boxes at the time.
It was a fun time. I remember hopping on #nanog on irc because I couldn't get uunet support to pull routs from us and had them pulling from us in less than an hour.
I don't remember the details, but at the micro-ISP I worked at, I got frontpage extensions working on Apache for customers.
We had two T1s, one for upstream, one for modems (which I never got to play with), we also contracted out nationwide dialup through megapath? or someone... We ran a radius auth server and they did the rest. Some java accounting package; I did an integration for ach billing, and made a module so we could sell domains (via Tucows OpenSRS). At some point, the modem T1 stopped working and we couldn't get the telco to fix it, so we had to move all the customers to the outsourced dialup provider, which wasn't great for the business... I left because of scheduling/poor performance at school and they got purchased by a customer.
They never charged us for it. Roughly a year later, we got a letter from their big four accounting firm saying that they were going bankrupt due to billing irregularities and our service would be terminated soon, but it was easy to get service then.
We never paid a dime for it.
Wife also worked for att and dsl was cheap for them except the billing system swore my apt was in another county and would send the installers there constantly. Took 8 months and was only fixed when wife went on a tour of a facility and mentioned it to an engineer and their VP.
It really sucked not to get anything faster than dial-up. It also explained why my dial-up couldn't connect reliably at 28k and I'd often be stuck with a 19k connection.
The experience did teach me a lot about Squid as a caching proxy and using wget and screen to do overnight downloads (I didn't learn about whey's background flag until a few years later).
One small business I have seen just has tonnes of word documents sitting in a single folder, with a mix of customer details, invoices, etc. To write a new invoice they copy an old one, then change the details, save it with a new name with a "system" (that changed over the years).
Up until semi-recently I saw a small-medium sized business (hired quite a few people) that did all of their accounting _by hand_, for hundreds to thousands of transactions. They had an old lady that came in to manually run their accounts on a calculator and it took several days.
A small business I once dealt with had a small black book (A6?) that was all written in note form, scattered, not dated, random important scribbles everywhere, mix of cash and card (not documented) - it was a nightmare. Somehow they had a good reputation in the local area, but behind the scenes it was pure chaos.
[Edit] Also, most CPAs I have talked with know and can integrate with Wave.
It was mind-boggling. They had a team of 20 people updating spreadsheets. Hopefully they've gotten their stuff together by now - but I doubt it. We tried to get them to buy a product data management suite (they were bad at building software in-house)...they finally agreed...then the recession hit, and they reversed course.
They had a separate spreadsheet for every state (all 50 of them). Some states just had a small number of customers in them (e.g. Wyoming). Others had large numbers of customers (e.g. California).
Their support team could only update one spreadsheet at a time. So adding a customer in Texas required locking that spreadsheet until the transaction was complete. Doing reports required compiling data from all the relevant spreadsheets. Queries that could be done on a relational database in under a second, took many minutes (or hours).
I tried to show them how my system could easily load all their data and make everything work much more smoothly (and saving them hundreds of man-hours); but they just couldn't see the value of spending a few hundred dollars a year for a license for my software.
They were bought out by an even bigger company a few years back, but they were managing to chug along until the mid 2010's at least.
The problem with pivoting to broadband was, most could not. The only option was reselling DSL, becoming a CLEC or WISP, and taking the dirt nap. The company I worked for tried everything and eventually went out of business after the dialup was sold in a rollup.
I was at a 123.net colo a few years ago, and off in a rack was half consumed by Livingston/Lucent and US Robotics Total Control dial-up T1 endpoints, running. The USR actually showed signs of use.
It broke my heart to be taking a call from someone willingly asking to pay for 12 months of dialup in advance. But they did, and it was like $130. Ow.
National Rural Telecommunications Cooperative members were slowly but surely lighting up fiber, at least.
As far as I can tell it's this: Their user management consisted of an excel sheet, but their billing was based on physical piles of filled out signup forms. A user cancelling their contract was implemented by (only) removing their physical from, but this was not communicated to the acquiring company, so they started billing customers who had cancelled.
But then how did they stop cancelled customers from using the service? How did they implement a user authentication mechanism in the first place? There must have been a software representation of user records somewhere in their system, which was kept up to date with cancellations. And the acquirer never learned about that, which would have just been a dub mistake, not exactly delightful or poetic.
Unless... they somehow managed to have the dialup system query the excel sheet directly and never stopped cancelled customers from using the service because they assumed those had switched to broadband anway and wouldn't want to use free dialup even if they could.
These would have been a much more interesting WTF than the mere existence of the Excel sheets and piles of physical forms, and I would expect them to be mentioned explicitly.
So... what gives?
> one of the aggrieved chose to show their displeasure by posting the list of all Dialup World user accounts online – along with every password in clear text.
- When a customer submitted their signup form to Dialup World, they would add a
row for that new customer to their 20,000-row Excel spreadsheet. Then, they
would put that signup form in a pile with all the other signup forms of
customers who had signed up on that day of the month.
- Every day, they would find which one of the 31 piles of literal signup forms
corresponded to that day of the month.
- Then, they’d charge the customers in that pile, one by one, for a month of
service.
I had more incoming phone lines than the rest of the street sometimes (this business was run from home), at several addresses around London as we moved. And we broke each telco's billing system in a new and interesting way...
I wish everyone had this privilege.
Unfortunately EnterAct was eventually sold to some larger company and DSL became more popular.
(I left about year before the 21st Century Cable acquisition).
1: Did the billing system (and its problems) show up in due diligence?
2: Did they ever figure out who leaked all the passwords and sue them? (Honestly, given that this generally soured the acquisition, the perpetrator could have been on the hook for a lot of money.)
(Now, before you jump to the leaker's defense, remember that in tech, things can change fast, and we all will end up working for an obsolete or acquired company at some time. Part of the game isn't pissing on the seat on the way out.)
Eventually became a Jr. Sysadmin, cut my teeth on managing Apache HTTPD, MySQL, qmail, built my first Linux server compiling Gentoo from Stage 1 on a dual 700 MHz Pentium 3 box. Learned so much from an awesome old school Linux sysadmin who, like me, got his start as a young kid at this very ISP. Oh, and I had access to 8 bonded T1s, which when the fastest consumer bandwidth I could ever dream of was a consumer cable modem with 3 Mbit down, and 256kbit up, being able to surf day in and day out on 12 Mbit of bandwidth was blazing fast.
However, like every other dialup ISP, much like the story goes in the OP, this little ISP was far too entrenched in their Dialup install. We did offer DSL service, but we were effectively a reseller to the larger ISPs in the area, so for customers that did switch over, we either offered poorer service, or they were paying more for the same service they could get direct from the bigger ISP. We were suffering from consumer attrition as the great ISP consolidation was happening, and the money was getting tight. A startup looked to be our savior - they wanted to deploy WiMAX (this was before Sprint bought up all the WiMAX spectrum for their form of 4G) to the region. What was originally pitched as a full acquisition became an acquisition of our customer list and some of our services like our shared hosting. Spoiler alert, those jokers didn't know what they were doing, people started leaving the company left and right, eventually equipment started failing and the people that knew how to fix those things took their knowledge with them, and eventually, my paychecks started bouncing, so I left.
Personally one of the most rewarding jobs I ever had. I learned so much, got paid quite well for very little "work", and it really set a course for the rest of my professional career. But hoo, the death spiral story that OP showed gave me a stark reminder of the very bad end times of that time in my life.
Also, I wonder if OP is talking about GlobalPOPS as the company that bought all the dialup ISPs that they worked for.
Helped them shuck and rack the USR courier modems for a couple closet expansions. they'd got a local carpenter to make them a standard 16 wide modem board carrier that was very slick. Still, punching down lines and getting all that stuff woven together was tedious.
The outfit I worked for started in 1990 and had actual POPs with aggregated 3Com, Livingston and Micom serial aggregators that went to a central location over leased lines. They had 1500 lines between retail SLIP/PPP and a couple bulletin boards. That was quite the monster.
What else to call it ? browsing the web ?
(Not even a joke.)
"Surfing across the world with multimedia, surfing day and night on the data Autobahn"
And the verses are so much worse.
A Toll Booth called a "Paywall."
Lazin' in the Shade (of the Information Superhighway)
Level Zero is "we have no idea what our process is". You get a cookie and promotion to level 1 just for writing down your batshit process, which some engineers find particularly cathartic and thus are all too eager to contribute. Pithy things like "and then we play phone tag and finger point for a week until someone gets fed up and volunteers to get it done," can end up in the first draft.