How Linux Saved A Fast Food Giant.
therealedwin.com
therealedwin.com
If they'd tested the McAfee definitions on even one of the POS terminals before rolling them out to all the restaurants everything would have been fine. They could have even automated having one restaurant get the definitions one or two days before all the others so only a single location was affected. Or better, set up staggered updates within each restaurant, so they all stay up and at worst lose a single POS terminal.
I doubt it though.
Deploying McAfee in the first place to all those terminals is a painful process by itself, yet many people do it because of the "extra" security it provides. When you can't trust something like McAfee for your production, then you're doing something wrong, and that's not the lack of testing upgrades to McAfee ;)
IMHO, the WTF here is that they've used Windows in the first place for what are in essence dumb terminals that help them manage drive-through orders ... for that you can always deploy a highly customized and minimal Linux OS, with most of the ports blocked and with a minimal number of services running, with reduced capabilities and enhanced with Linux SE if you're a security freak ... making the area of attack for any potential threat a lot smaller.
Then you wouldn't need a piece of shit like McAfee to handle your security ... all anti-viruses lately are pieces of shit IMHO.
> The real issue here has nothing to do with Linux or Windows
I think it does. On that terminal they could've had a small Linux image providing a ssh server, whose purpose would be to boot into it in cases of emergency. And you can control a Linux instance a lot more efficiently remotely. Try doing that with Windows.
It's awesome that Linux saved the day and that the author was resourceful enough to pull it off, but the whole event still could easily have been avoided (and without having to change much at all).
As nice as it is to blame McAfee, only the end user can control the extent to which third parties can affect them.
At Loopt, we stagger both Windows and Ubuntu updates for this very reason. We do the same with our own code updates. It costs very little and has saved us on numerous occasions.
In most of those settings (read: no need for remote access) it is per se not worse than deploying Linux unless you have them connected to an open LAN/WLAN and/or the Internet.
It sure made sense for a lot of people during the last 15 years for various reasons, occasionally one can even spot a DOS system still being used (now you know why they still sell floppy disks in 2010).
Big companies like McDonalds clearly had the resources long ago to go for a different approach but only during the last 5 years or so this has become a more viable option for many small-medium shops.
Even though Ubuntu, Google, Wikipedia & Co have "happened" during those Windows POS years we aren't there yet.
I suspect mainstream small-medium business will skip this step anyhow:
For example I myself am working on a distributed POS solution using a certain set of 2010 state of the art technology (the no-brainer variation: "cloud" back end and administration, newest generation of touch devices as "terminals").
It allows for low on site IT infrastructure and maintenance plus a strong emphasize on UI/UX which thanks to Apple is finally becoming ubiquitously expected.
About six months after I started, they replaced all of the POS terminals that had been there since I began with new shiny ones.
New shiny ones that were dumb terminals to virtual Windows XP instances running out of the back office. Unfortunately, there were approximately 20 POS terminals in the place.
And every single one of them was on the same collision domain. All the Cat5 was ratnested under the counters; some of them were held in to the plethora of cheap 5-port hubs by duct tape. In the networking cupboard behind concessions, the wires were coated in old "butter flavored topping." Occasionally, all the POS terminals in the box office would cut out - usually because someone in the back office accidentally knocked over the five-port hub back there, and one of the connections fell out.
Needless to say, management was super excited to have these shiny new POS terminals. Everyone who had to use them? Well.. we all did our jobs a little bit slower: it's hard to ring up a ticket sale when it takes 3-4 seconds for a touchscreen press to register. The actual hardware of the POS terminals was pretty (needlessly) impressive. A lot of times the people running a place really have no idea what any of "that computer stuff" does - they rely their POS salesman and their crack team of installation specialists to get everything running. And from what I gather, a lot of those companies are pretty much used car salesmen...
(wonder if that's a potential business op? It's a hard market, to be sure, but most store owners I know that use a POS hate the system like they hate fleas, mosquitos, and AIDs)
Looks like someone else has developed it: http://www.crunchgear.com/2010/04/27/the-ipad-pos-system-com....
If your deployment environment doesn't offer NFS/SMB/CIFS you can fetch your images via HTTP from your PXE booted kernel too.
Regarding data centers - it can even boot over serial in connection with HP iLO and IBM RSA/AMM out of the box.
http://wiki.grml.org/doku.php?id=rescue
http://grml.org/changelogs/README-grml-small-2010.04/
PLoP seems to be a great tool for similar cases, interestingly both are Austrian distributions:
http://www.plop.at/en/ploplinux.html#pxel
I've got a grml usb stick with me at all times (now even supports multiboot 32/64bit!). I just recently recovered a relatives' WinXP after a seemingly failed automatic update and did a dll rollback in a couple of minutes.
It's a great tool plus it shows you how to use the mighty zsh properly!
http://wiki.tinycorelinux.com/tiki-index.php?page=Netbooting
It's well documented and easily customized:
> If I had not flirted with Linux during college, where would we be right now?
I wonder how much in support just relies on good luck. Someone happily trying out stuff several years back on his own. Curiosity saved the system.
Corporate support is based on knowledge bases built up by customers "trying stuff out" and reporting in when it doesn't work.
I admire his sense of humour. Easter eggs in the face of calamity indeed.
Part 5.1 used to include this note:
Note: Systems commonly affected by viruses typically do not include UNIX based operating systems or mainframes.
Consumer grade MacAfee that auto updates is still probably a very poor choice for this application as they should have a test bed in the lab that they beta all changes on before they roll out to thousands of machines that likely need a truck roll if something goes wrong.
Enterprise grade McAfee still sucks, it's just a more manageable suck. Despite that, it still blew up with the bad DAT file - those are the same between product lines.
Even when the POS is not running Windows? Or does the PCI-DSS assume all computers are running Windows?
I once had a car that had a bad bulb socket in the "cyclops" brake light so it wouldn't pass inspection. Did I fix the socket to get past? Hell no. I removed the whole light. If it wasn't there, it didn't have to work. Those were the rules.
Neither used AV, but they were tightly locked down. My impression was that most customers don't care about the OS. Some prefer Windows, they already have the tech personnel in place. On the other hand, the Linux ports were done because a very big customer said that they had to have Linux... It certainly cost them more than licensing Windows. :)
From a dev point of view, Windows was easier, as it is generally more friendly towards C++ development. The OSes were about equally capable, I didn't see any Linux technical advantages in practice.
You might even say it's... a trojan.
groans
I guess point of sale software is considered 'enterprisey' enough to warrant vendors wanting to run Windows and SQL Server for 'support' reasons.
There is a Java implementation of the standard (called, unsurprisingly, JavaPOS) as well as a .NET implementation (called POS for .NET). I've run a Linux based point of sale that used JavaPOS to interface with peripherals without issue.
Several hours later, after hours of phone calls, hundreds of orders written on paper and totted up by calculator, a confused-looking technician managed to get it working properly again.
What a mess that was.
It should simply be a matter of phoning the manager and tell him :
"Don't worry, just insert the disk that says "RESCUE DISK" on it, reboot and it will roll back the machine to last week".
Then the fix would have been a matter of simply swapping the dhcp entry for the pos devices on the server with one that had the pxe directive. The pos would have caught that during its next reboot and restored itself.