Hacker News Reader

Comment by wronglebowski | Hacker News Reader

As someone who worked in stores for years, this isn’t entirely accurate. The machine isn’t just being an asshole to drive up service calls.

The machines are filled with a dairy product which makes the “ice cream”. This is boiled nightly to keep the mixture free of bacteria. If the nightly heat cycle fails to complete for one of a dozen reasons the machine locks out. Those unaware of the process usually end up calling for service even if it’s a operations error(not enough product to boil successfully)

Every two weeks the machine must be torn down and cleaned. It’s slightly complex but all well documented. Once the 14 day marker has been passed the machine is inoperable. It must be cleaned for food safety reasons. If your “cleaner” is out or you just have poor operations you’ll end up placing service calls out of confusion.

If the heat phase fails or the 14 day timer has expired the machine "locks out" and becomes inoperable. This is where the vast majority of service calls come from.

I never had an issue with a machine that was less than 20 years old. Taylor makes good equipments you just need the most basic level of operations proficiency.

View on HN

> If the nightly heat cycle fails to complete for one of a dozen reasons the machine locks out. Those unaware of the process usually end up calling for service even if it’s a operations error(not enough product to boil successfully)

This example is covered in the video. ;) The thing is, at least according to the reporting, even these operations errors are not clearly defined by the messages given by the machine. So, according to franchisees they talked to, all roads lead to the service technician, even if the problem doesn’t need one. That obviously leads to more downtime.

It’s part of the problem because Taylor is incentivized to keep people calling in technicians, which means they deliberately update the machines with error codes even more difficult to understand. (Examples covered in the video too)

Additionally, the video covers that many other chains, such as Wendy’s, use machines from Taylor with no problems. Supposedly, this is something specifically wrong with the model given to McDonald’s, because franchises don’t have a choice to use other models.

When the video came out, about 14% of machines in the US were “broken”, which is pretty terrible for an industry application regardless of the cause!

I always had access to a Taylor manual and a Taylor technician via phone at no cost. Even with machines that were far out of warranty. Troubleshooting was never an issue for me.

I never worked anywhere else with a Taylor machine. Perhaps other companies have better deals or easier to use equipment. I would like to know if Wendy's and others machines operate using the same food safety standards.

"Food safety standards" would not result in downtime rising from <1% to over 10%.

> I always had access to a Taylor manual and a Taylor technician via phone at no cost.

Was that technician for free? Or did your franchise owner pay for it, out of (your) sight? The claim isn't that they do a poor job with technicians, just that they are perversely incentivized to profit from it.

The video also explains there was a (very limited) owner manual, while all the secrets were in the service manual. Which of those did you have access to?

I find it potentially a bit of a low blow to question Wendy's and others' food safety standards.

Did you see the video?

Payment type would also matter. If it’s a flat monthly fee then they’re incentivized to avoid service calls. I imagine franchises and Taylor itself would also vastly prefer such a scheme, as it would reduce cost variation.

I understand that companies like Taylor benefit from income streams being predictable. But as per the video it seems that their revenue model was predicated on having as many incidents as possible, each billed separately.

If you have many incidents, they become statistically predictable again.

I had access to both. We did not pay for any of it. In fact the Taylor facility in our local market offered free “training” days once a year specifically for McDonald’s employees.

McDonald’s is much more political than people think. The franchise I worked for was very well connected and well represented. It’s possible that this access was something they pushed for or is not well known.

Was it at McDonald's? The article specifically says that it's McDonald's machines that require (edit:excessive) maintenance and not the ones used by other companies.

From this comment tree I wonder if the problem is some evil company being greedy, or is it a lack of incentives in McD and franchise owner to update SOP to properly train and assign a worker to properly maintain the machine to keep the machine running to offer ice creams consistently.

Why do they keep an item on the menu if they are not being able to make it reliably?

It's a minor but important part of the profitability picture. While this is much less true today, one of the core concepts was always that McDonald's could execute things at scale no one else could.

Now there's a ton of competition and specialty shops, but in the past this wasn't always the case. A small town might only have one other location where you can get "fresh" ice cream. If you can execute on this machine properly and you have the market to support it they can be huge profit makers.

>>>I always had access to a Taylor manual

You expect people to RTFM.... come one this is 2021 no one reads manuals any more...

//wish that as sarcasm but the number of times I have seen a solution to a problem that plagued people for weeks be found clearly laid out in the manual for a given product is far to high

> It’s part of the problem because Taylor is incentivized to keep people calling in technicians, which means they deliberately update the machines with error codes even more difficult to understand. (Examples covered in the video too)

I do believe this happens but I'm also "skeptical" of the ability of a high attrition place with underqualified workers working under pressure to get themselves up to date in maintenance practices and manuals.

Edit: and yes, the franchisees are the ones who get the short stick. Franchises are thorny and usually not in the favour of the franchisee.

It is very common for engineers to hypothesize (often with limited evidence) that the problem is “the users” rather than the technology and its support infrastructure. I think this can be true, but two points:

1. The high-attrition low-skill workforce at fast food restaurants is a fact of the industry. (I am not endorsing it either.) So if a solution is failing widely when faced with the reality of its users, then the system is flawed not the users.

Of course, you might argue that it’s impossible to build a system that works with this type of users, which leads to:

2. The video argues that failure rates at competing fast food restaurants are vastly lower. While I can’t verify this myself, if we assume it’s true then it’s strong evidence that the problem here is unique to McDonalds and therefore doesn’t represent some systematic impossibility.

I don’t care much about ice cream machines, but in the broader technological sense I dislike “users are the problem” arguments. The whole point of tech is to make users’ lives easier. When other people are doing that and you’re not, there is a problem.

While I understand this viewpoint and I don't necessarily disagree with it, this one piece of equipment is unique and that's why I think it is so often "broken".

No other piece of equipment has the mix of complexity and niche impact on the restaurant. Even the brand new at the time espresso machines are mostly self cleaning. There's no other object that requires this(minor IMO) skill level to operate. So it's just left undone. It's widely regarded that while it's a high profit machine most of the time you can run a highly successful business without it.

I agree with you, and I was not blaming the users/the workforce. As you said it is a fact of the industry.

McD is known for making their processes simple and repeatable and it seems the machine is failing that paradigm.

(And oh well, my comment was speculative and made before I saw the video, the users really can't do much when the company hides information from them - though we don't know exactly how the procedures are laid down in the manual and how they can be improved. it's one thing if the manual says "fill this up and press the button" another if it says "fill this up to this line and check A, B and C before pressing the button")

> McD is known for making their processes simple and repeatable and it seems the machine is failing that paradigm.

Which, coincidentally, is based on Frederick Taylor's theory of Scientific Management. Pretty sure Frederick Taylor and the Taylor making the ice cream machines aren't connected in any way though.

There is another bit in the vid I watched there was a dongle they could buy for awhile. It was telling them exactly what they did wrong and how to fix it. Basically taking the obscure error code and turning it into human with pictures. The franchise dudes were extremely excited for it.

That lends a decent amount of credence to a 'process' issue. If that many people are messing it up. Then something is wrong either in the physical or training realm or both. I usually have to have this conversation every few years when someone says 'oh we will just train the users'. Then I go 'dude I know how this should work and I am totally lost, do you expect someone who does not give much of a crap about it to pick it up?'.

One thing the vid misses though is the lockout rate could be coming from mcdonalds itself in a different way. They do not want to end up on the wrong side of a lawsuit and bad publicity. Remember the lady who spilt coffee on herself, and got 2nd/3rd degree burns. That cost them a lot of time, money and bad press. So they may have set the machines to be oversensitive.

I can imagine that it’s hard to keep up to date with maintenance practices and manuals when the manuals are explicitly unavailable, as is the case for McDonalds.

This isn't a high attrition work place!

Sure in general McDonald's is. However there are always a few people who have been around for years who know a lot more than the average worker. They are in management or maintenance. They have more training, and better wages. (not to be confused with the person who has been there forever but never advanced to such a position)

Anecdotally, my local McDonald's never has trouble with their milkshake machine-- it's my local Burger King that frequently does.

So a franchisee-specific training problem would make a lot of sense.

My sister worked there as a kid and labor on demand had certain chaotic results.

To give an IT analogy if an IT department staffed based on corporate's seemingly random rolls of the dice it could be painful to make sure weekly backups get done weekly, for example. Especially if the only feedback was when the restore process occasionally broke down. Whereas if some dude is always at work every Friday morning at 8am he can get in the habit of being the guy who always runs the backups. Also imagine if the only way to run backups was to flip the power switch for a half hour and hope you know how to complete the backup in less than a half hour... nobody wants to be that guy who shut off the DB server even if maint is required...

On the other hand... if you're swapping people in and out of a roll, it forces you to have an SOP and training.

If it's the same person for 5+ years, I can guarantee there's nothing accurate written down (even in regulated industries).

Given the two problems, I've seen more dumpster fires as a result of the latter than the former.

The reporting is based on conversations with franchisees, who are a part of the equation here too.

McDonalds franchisees are always in a tug of war with the company over various issues, especially loss leader products like ice cream and the dollar menu. Thus, it mysteriously “breaks”.

In some markets, a cone is $0.29 — it’s a money loser that the franchisee doesn’t like and probably doesn’t drive traffic in many locations.

I’d look at the human element before declaring a major manufacturer of a simple product incompetent. They wouldn’t be consistently failing for their biggest customer over many years.

This seems to be an issue with only McD. According to the video other vendors like Wendy's don't have this problem.

Remember that McD the company has nothing to loose from failing ice cream machines. That loss is taken by the franchise owners

>According to the video other vendors like Wendy's don't have this problem.

because their machines don't self-sanitize.

The video covers this. All franchises for all companies typically do this overnight.

They all have to be cleaned nightly, but they're not all self-sanitizing. The woman at the wendy's drive-through said they "wash" the machines nightly, which suggests they take it a part and wash it rather than running a self-sanitizing cycle.

Out of curiosity, is there a difference, work load aside, in the end result of the two processes?

Not necessarily. McDonalds owns many restaurants themselves as well.

This is funny to me as the milkshakes are relatively expensive here in Germany yet the machines in some locations still seem to be perpetually broken. I'm not sure if the milkshake and ice cream machines are the same thing but if they're not I guess some franchisees just don't want to deal with the hassle of keeping them operable regardless of whether they're loss leaders or not.

Wendy's has 2 soft serves (vanilla and chocolate). McDonalds has like 8 ice cream/shake drinks, so it shouldn't be a surprise that McDonalds would have a more expensive and complicated machine to get 4x as many varieties.

I worked at McDonald's in the 1980s. The machines weren't so fancy then. They didn't have a self-santizing heat cycle. We broke them down and cleaned and sanitized them every night after close. The next morning they were put back together and turned on. Once a week we discarded any remaining dairy product to break the cycle of any bacteria that might be present. Same with the milkshake machines.

In several years of working there I cannot recall any breakdowns of these machines.

I worked at the concessions stand at Muir Woods in the early 90's. Every night the machine was emptied, and the removable parts were put in the dishwasher.

Every morning the goo was put back in, and there was never a complaint, or a down day. I rang up a lot of $1.00 Softies.

(Loved my coworkers. Still think about them all the time. The greatest group of people I had ever met. Hated the private sector corporation who did everything they could to lower the percent they had to give back to the government.)

Same here. But I also recall that when the health department would do spot checks of bacteria and coliform counts on the machines, they were always in or near the danger zone.

All you need is one kid that didn't wipe their ass correctly before assembling the beaters and you've got a health problem in the making. The self-sterilizing machines were designed to prevent this.

I worked at McDonald's for 6 months in the late 90s and it was the same process. I actually quit because someone started the machine breakdown early, incorrectly, and without telling anyone because they wanted to leave at close.

> Every two weeks the machine must be torn down and cleaned. It’s slightly complex but all well documented. Once the 14 day marker has been passed the machine is inoperable. It must be cleaned for food safety reasons. If your “cleaner” is out or you just have poor operations you’ll end up placing service calls out of confusion.

This seems entirely reasonable to me. I've heard horror stories about bacteria & mold growth in commercial ice machines from infrequent cleanings.

But...how high is the barrier to entry to becoming a "certified" ice cream machine cleaner? How much training is required, how much does it cost, etc?

In other words:

> Once the 14 day marker has been passed the machine is inoperable.

Who has "admin rights" to reset this "most recently cleaned" timestamp?

Is there any reason why I, if I'm a McDonald's manager, couldn't take the training myself, and be the person who does the every-two-weeks maintenance?

> Is there any reason why I, if I'm a McDonald's manager, couldn't take the training myself, and be the person who does the every-two-weeks maintenance?

I'd go out on a limb and say that it's McDonalds corporate that doesn't trust the franchise owner with the machine, not the manufacturer trying to swindle franchise owners.

Sure. But replace "franchise owner" with "shift manager" or whatever other job title you want.

The question remains - what's the barrier to entry to becoming "certified" as an ice cream machine cleaner?

Is it legitimately a difficult cleaning process that requires an extremely specialized skill-set that only the High Priests of Ice Cream Machinery are able to perform?

Or, as seems likely to me, cleaning the machine is relatively simple, but the ability to reset the "time since last cleaning" setting may be artificially restricted?

If I were the manufacturer of these Taylor ice cream machines, setting them up with a "one of our certified technicians needs to touch them at least once every 14 days, or they stop working" seems like an excellent way to seek rent / generate passive income.

I think the barrier of entry isn't training, it's trust. McDonalds corporate signed a contract with Taylor with the caveat that only Taylor maintain the machines because they trust Taylor to do it correctly and, because the contract with McDonald's is probably enormous, Taylor has a lot of incentive to make sure nobody gets listeria from McDonalds.

It's "artificially restricted" in the same way that I could give anyone a key to my house (duplication is trivial) very easily but I don't.

That’s an unusual usage of the phrase passive income if a tech has to go out and lay hands on the machine.

As I've tried to express elsewhere, it's really not difficult. Anyone can do it given a few hours training.

Suggesting the franchise owner do it or even the store manager is far off from how these operations work in my experience.

Franchisees (Owners) have so much to do. They never "have" to work a single shift after they complete onboarding. While this can take around 9-12 months once most are done they never look back. Owners are tasked with running and growing the business, focusing on their local community and the people element. Any equipment work is not considered a good use of their time.

It has been 20 years, and surely the machines have changed. But it wasn't difficult to learn to tear the machine down. Considering this is a company that often has picture instructions for employees and tends to make things easy to understand, I can't imagine it has gotten more difficult. The store I worked at did this every night after running disinfectant through the machine: The parts we took off were cleaned every night. There were trays to keep the parts organized. These were organized in a way that made them easy to grab for proper assembly in the morning. Occasionally the morning staff had some problems, but those were usually fixed in time for there to be ice cream at lunch.

In other words: Store employees can do all of the basic stuff necessary with the machine.

Again, though, the machines have likely changed (Though they probably work similarly). And if the current machines need cleaned and sanitized less often, employees probably don't do it as often. I'm gonna guess instructions can fix this, though.

There’s no real certification. Usually you’d watch the training materials, then shadow someone once, then do it live with someone who trained watching. It’s not hard you just need to have done it a few times.

There’s no admin reset or anything like that. There is a maintenance menu and an admin menu but even in the higher level “secret” menus changing this was never an option. I’ve seen people try and short the main board with a screwdriver to get it to “reset” the date.

It’s a very intelligent machine and when it’s cleaned you need to go through the cleaning “process” in the menu. Wash, rinse, sanitize etc. it monitors the process with various sensors and it being flushed so many times. Faking it isn’t an option, I’ve seen too many people try.

The video says the Taylor machines at other fast food places (Burger King, Wendy's) rarely break down, or at least much less than McDonald's. So you're saying this is untrue? If not, why the discrepancy?

the other machines don't self-sanitize?

mcdonald's has much higher customer volume, and possibly much more intense burst traffic ... lunch & breakfast, for example.

the higher frequency of machine breakdown probably should first be linked to frequency of use, rather than trying to find something nefarious going on.

The Wired article explains this more clearly than the video. These machines do more than the ones at the other places.

"But what makes the machine special is that it has two hoppers and two barrels, each working independently with precise settings, to produce both milkshakes and soft serve simultaneously."[1]

1: https://www.wired.com/story/they-hacked-mcdonalds-ice-cream-...

Sounds like having two machines would cost them some space and increase the reliability.

Do the franchise owners want that increased cost and reliability?

If it's mostly operator errors, than why not make the information clear to the operators on what the error actually is? Your post only confirms how bad Taylor machines at McDonalds are. It looks they are deliberately making the machines obscure to the operators

Yes, this seems to be exactly what they're doing – I posted a link on another thread to a Wired article that sheds some light on why these machines break down, and how McDonalds have waged legal war on a startup trying to bring some transparency to the internal operations of the Taylor machine.

Here it is again, to save searching through the comments: https://www.wired.com/story/they-hacked-mcdonalds-ice-cream-...

[edited to add link]

Seems to be paywalled? Even if this was the best idea executed at the highest level it wouldn't matter. This is all politics.

Any time the franchisees try and circumvent the system of corporate it brings massive amounts of force to the table. Mcdonald's 100% intolerant of any kind of franchisee revolt or action against their direction. It's got nothing to do with the mission and everything to do with it being "outside the system".

As a note, a lot of that wired piece is also in the video, including the startup. :)

This doesn't appear to contradict the GP at all.

You have pointed out that when the machine locks out, there is an actual reason behind it. I don't think anyone claimed that it's simply triggered by an RNG or a timer.

However, why are the machines this opaque and difficult to keep running? I think the point has been clearly made, that an incentive structure optimises for low reliability. Nothing you said contradicts this.

> worked in stores for years

Specifically McDonalds? The video states these problem aren't experienced at the same magnitude at other franchises, and alleges that McDonalds franchisees are locked into purchasing a downgraded model with indecipherable error messages.

Do you have any explanation why this is purely a McDonald's problem, and other fast food places who use essentially the same ice cream maker have much, much less down time?

mcdonald's has more customers.

That would explain why the ice cream machines a never broken at some location, but frequently at others. It depends on how the franchise is managed and run.

'boiled' is AFAIK a misnomer - I believe it is pasteurised ... at 151F - not actual boiling

I was crew for a few years in the early to mid 90s. Back then, the machine simply needed to be cleaned nightly (water cycled through, all the gaskets cleaned and relubed). Not hard, but definitely required some training and most weren’t taught. It was rarely broken but we often started cleaning early so....

I guess it’s just way different now.

It's a restaurant - 99% of the machinery in there has to operate within certain boundaries for food safety. Somehow every other machine manages to be fine, execpt this one that has a 15% failure rate.

Also, other restaurants service dairy - also with a much less than 15% failure rate.

There are simple solutions for everything you just said. Taylor makes garbage equipment and is ripping off franchise owners. McDonalds corporate likely owns stock in the company and uses it as a side hustle of revenue from franchise owners.

Do you have any insight into why the machines are broken so often then?

They aren't "broken". It's operations failures. The basic fact of

1. Keeping the machine full of the right amount of product to complete it's nightly heat cycle

2. Have someone trained to fully clean the machine every 14 days

Is a burden too great for many franchisees to bear. The turnover is too high and investment in people too low to keep them operational. Nothing about them is "broken"

Poor UI is also a bug. And why do other fast food chains not have this problem then? I appreciate that McDonalds tries to safe money on training of new staff, but I just can't imagine that they do so much of a worse job at it than the competition.

Did you see the video?

Why are they using such a complicated ice cream machine? I worked in food service in the 2010s and ours was pretty simple. One switch at the back for on and off and that was it. It took one person like 10 minutes to break it down and have it cleaned at the end of the night. The hardest part was wiping the exterior down. You basically just put a bucket underneath, ran the ice cream out, then ran hot water through it (forgot if we used some sort of sanitizer here), and hand washed a couple pieces. Other commercial icecream machines I've seen work like this. This mcdonalds machine is blowing my mind. I wonder why they opted not to use the bog standard ice cream machine every other fast food place uses.

It sounds like the idea is to only have to spend time manually cleaning it every fortnight instead of daily, and perhaps to limit waste of the product? But perhaps that gets offset by the fact that the increased complexity seems to make it so difficult to operate that it leads to constant interruptions and expensive service calls.

Most likely, the real answer was given straight away: a worker from other company says they clean their machines daily. I'd expect that in common language that means they disassemble and wash them the old way without using any automatic maintenance programs. But then you wouldn't have a vaguely sensational video on a pop cultural topic with a million of views.

I guess it's a typical big company nonsense. Someone probably had an idea that daily cleaning is too time-consuming and expensive, and the task to solve that problem was introduced. Some lab made tests, and defined The Only True Way of keeping the product edible. (Side note: re-boiling a mix of fats and sugars each day for two weeks? Who would eat the result of that? Who thought it was a reliable idea?) The cleaning period could only be stretched to the maximum, for obvious business reasons. That resulted in tiny error margins (which maker can't voluntary change without spending money on new research). So presentations on clear financial gains were made, and the new option was proudly implemented. Some employee has probably been keeping eye on changes in cream composition, regulations, and related research, and issuing new checks for the machines. However, in reality this doesn't work that well (even indifferent part-time workers who don't give a damn after a shift are part of reality you can't simply rule out). But the goal of reverting or altering the process is not really important to McDonalds, as making it work is now pain in the store owner's ass.

You have probably noticed similar cases in computing, when some high-tech solution are introduced, and their gains are shared with lots of pathos, but the resulting system looks way too complex and unstable even from a distance.

It just seems like some vendor got the contract before a cost benefit analysis could be done. I mean 15 minutes out of a shift worker's time costs mcDonalds maybe $5. $5 for a day of ice cream sales seems like a cheap service contract to me.

The video is not a complete story.

I suggest you read the Wired article that came out a few days ago, as it addresses a lot of the questions being asked here.

https://www.wired.com/story/they-hacked-mcdonalds-ice-cream-...

It should be worth noting that the story in this video and the Wired article appears to have been initiated by the creators of the Kytch.

And note that the very existence of the Kytch says there's a big problem with the UI of the machines.

It seems like you didn't bother to watch the video, but used the OP synopsis as an accurate distillation of the investigation... which it isn't. McDonalds (purportedly) have to use an inferior machine. This machine is used by many other franchises, including Wendy's, but the UX has been specifically throttled in the McDonalds mandated model such that the error messages are opaque. So much so, that when a newer version was rolled out, one of the techs posted a video wondering if anyone else noticed the error messages were getting worse (and the video was promptly taken down). Thus training becomes much more difficult when error messages are in Eldritch screeches instead of plain English, like every other model.

I'm not sure if you've worked at McDonalds (you've never specifically stated in any comment) and you've never specified when. Is it possible things are different now to when (or whichever franchise) you worked?

I guess you didn't read the article or watch the video.

Doesn't most of this apply to milkshakes as well?

Reply on news.ycombinator.com