Global Network Navigator was the first nation-wide ISP in the US, and was a partnership between O'Reilly and AOL. But then it got too big for O'Reilly to run, and AOL wasn't ready to take it over, so we made ANS take it -- at the time, we had recently acquired ANS as our Tier-1 network provider, and they knew a lot more about DEC Alpha hardware than we did. But then it got too big for them to deal with, and AOL finally had to take it over. The mail system part was given to me and my co-worker. We learned a lot about DEC Alpha hardware from GNN, and we loved it.
That system worked great, up until the time that DEC sent out a Field Engineer to upgrade the firmware on the drive arrays. Then we got weird errors when we tried to come back up. So, we switched to the backup systems that were connected to the same drives. And it was also hosed. Fortunately, I had started taking user mailbox backups to our Auspex NFS file server that week, so we had backups. We soon realized that over 50% of the POP3/leave-on-server mailboxes will change radically over the span of an day. And some of our mailbox backups were over eighteen hours old. So, most of those backups were actually useless.
We did finally get back up and running by mounting the backup mailboxes via NFS, but we lost pretty much all user confidence in the system by that time.
The drive arrays had been set up to mirror at the lowest level, then stripe across the mirrors in hardware. We then striped again across those volumes in software, which was a high performance disk striping technique we called "plaiding". We ended up breaking each of the lowest level mirrors and sending the whole bunch to the guy who wrote the AdvFS filesystem for DEC, but he never got better than 80% recovery of the data, which meant that on average every fifth character was wrong. So, pretty much none of those recovered mailboxes were useful.
That was the death knell for GNN. If you can't trust the mail system, then what else can you trust?