Edge Computing at Chick-fil-A
medium.com
medium.com
These got installed, but adoption was low due to cost, they were tied into the POS.
Not meant to be a rant, just wanted to make the point.
In some sense, the automation of demand prediction takes some of the fun out of gaining that intuition yourself. On the other hand, it is nice not to be the queue blocker with orders piling up behind you because you weren't prepared for a sudden burst of traffic...
The human brain is definitely wired for this sort of thing.
In hindsight, “sod” wasn’t the right term to use (that’s what you get for posting at midnight) but the point I wanted to make was the removal of meaning, autonomy and human interaction when not only are you doing a repetitive job, but how you’re doing it is dictated by a machine. The movie Brazil came to mind, although I’m sure there are better parallels to make.
Wendy's used to do this around 91/92 (clipboard, not iPad)
I think they ditched it with the smoking sections.
I don't know what it would be like at NYC since it's not a drive thru.
https://www.youtube.com/watch?v=TIQWyYQ0EYg
FWIW, also an edge approach, with Prometheus etc.
Spoiler: It's a stack of Intel NUCs.
[1] It's the same software that runs my home automation system, so an inevitable feature is my car and home units interacting, I wrote half of a piece of navigation software before I stopped, and the entire interface is intended for voice control so I need to throw that in there at some point too. I intend to work on a CAN interface to connect to the car, but I've had a few roadblocks to getting started on that project.
The code is here: https://github.com/ocdtrekkie/HAController but I don't know if I'd really recommend others use it. The main perk to me is that it's designed around what I want and use (I tried a lot of alternative options before rolling my own), so unless you also really like Visual Basic code and have a brain ordered bizarrely similarly to mine, you may want to start somewhere else.
I'd love to chat about what I've learned along the way or what ideas you have (because I might borrow them!), if you want to chat elsewhere hit me up at inbox (at) jacobweisz (dot) com
I once left the reading lights on; car was supposed to have turned it off when the doors locked, but somehow didn't. I found myself calling AAA in the morning to get a jumpstart.
There is an external battery solution for the NUC, and I bought it! http://www.mini-box.com/NUC-UPS But I haven't switched to it yet. The big upside is that it will gracefully shutdown my NUC after I power off the car. But if I use the cigarette lighter for power, it will lose the ability to turn itself on when power is present, because the batteries are always present. NUC UPS supports using a different power connector which can turn on the PC on ignition, but I haven't had it installed in my car as of yet as that's a little bit more work, and I'd need some help from someone who knows more about the car's electrical system.
As far as fast power-on, it's a pretty high end i5 NUC with an NVMe SSD. It boots pretty darn fast, and my software takes less than a second to load once the OS is up. The slowest part of boot is that I don't want my location history easily steal-able, so it's encrypted, and I have to key in the code to unlock the machine. (I'm looking at a security key or similar to replace this step in the future.)
As a note, I'm more or less specced out what's involved in a solar power setup on the roof of my car to independently power my computer all/most of the time when the car is off and otherwise charge a secondary battery off the car's inverter, but there's no reasonable or sane reason to do it. ...But I thought about it.
Probably the only real concern I'm afraid of is condensation when heating up the car on a cold day, but it hasn't been an issue so far, perhaps because the computer is not near a window.
The NUC-like hardware that goes into tanks have chassis consisting of a mesh cage to allow maximum airflow since the cabin temperature alone can exceed 100F.
I've been planning to make some custom-cut USB cables and a 3D printed part just to make the setup of my display a lot cleaner... and I haven't done either of those and it's been a year or so since I planned to.
They do actually make a dual NIC NUC, but it's a little overkill for most uses: https://www.amazon.com/Intel-Machine-NUC8i7HVK-Radeon-Graphi...
On the other hand, my NUC has plenty of USB 3 ports, and an Ethernet dongle will run you about $7.
Part of the problem is that even if you only write 1GB per day the card is highly likely to reuse the same sectors over and over.
One would have expected this to be an area of focus for RasPi developers? Surely some driver could be updated to avoid this?
This talk about SD cards is really great and explains some of this:
And even then I'd be surprised if you could game physical sector offsets by playing with the virtual offsets you have access to.
I did a bunch of research on this a while back and the conclusion I found was: yes, but buy name-brand better rated cards to avoid cheapo cards that do not do wear leveling or do it very badly.
It's nowhere near as good as SSD drives, but it's better than just a naked flash chip... unless I'm wrong.
Even the brand name cards aren't worth shit; you need to buy "industrial" cards.
Source: tried to get a root filesystem to work on name brand SD cards for a high availability product.
How much abuse was your high availability product delivering to these cards in terms of writes? Was it something like a video recorder, database, cryptocurrency, or some other application that did large amounts of write I/O?
I ask because we're about to ship something that uses SD cards, but the I/O is very low. It's a network appliance and doesn't do anything locally that is high write throughput.
We even saw SD cards screwing up in RO mode in fact, from major brands. Trust me, you want to swap out for an industrial card in your BoM.
It's definitely getting better though
As an end user, I run a personal authoritative DNS server that has small RAM requirements. The RPi (or other SBC) boots to a mfs mounted root, then mounts all directories as tmpfs. Then I remove the SD card.1 As such, the logs for this server, which are automatically rotated and do not exceed 5M in total, are written to RAM.
1 I only use the SD card to boot. The only files on the card are a bootloader, a bootloader config and two kernels, each with an embedded filesystem. If updates are necessary, I make them to one of the kernels at a time. The other is the backup. The bootloader and bootloader config lets me specify which kernel to boot.
Funny how the world has changed.
I got my first taste of real computing from a guy in the next town who would buy dead PDP's from local McDonald's restaurants, fix them up, and sell them on.
He always said the hardest part wasn't replacing broken parts, it was that invariably the machines would always have soda spilled in them, no matter where they were stored. Cleaning the insides was his most time-consuming task.
One of the advantages of an Intel NUC is that it probably has a lot more horsepower than an old PDP/11. It's probably also a smaller target for wayward beverages.
It wasn't like having an account to crunch all the numbers on staff full time was cheap. Especially when you'd have to have 3-4 accountants just to have 24 hour service, regardless of the work load
Also is Highlander open source? I don't see a link to it in the article.
We use a VIP that the NUCs share... ie; one of the three will always have a VIP, and if it dies another NUC grabs it. This is a poor man’s load balancer in that sense, because we only have the NUC hardware onsite;
https://github.com/kubernetes/contrib/tree/master/keepalived...
We are also looking at metallb
It would've been much simpler to have a plate with multiple pressure sensors and a multi-color LED per spot. When you set the fries down on the sensor it activates and the LED turns green. After X amount of time the LED can turn yellow, meaning that it's becoming stale. Finally, after X+N time has passed and it's no longer fresh it can turn red. Removing the fries turns it off. Aside from being way cheaper, I'd conjecture that this would be much more reliable. I'm pretty sure I could get a prototype of this up and running over the course of a weekend or two.
If you wanted to get really fancy I guess you could also track the room temperature and moisture levels and use that to get a better guess of how long a group of fries will remain fresh. Although I don't know if environmental factors like these have enough impact on fry freshness to be worth taking into consideration.
Anyway, it looks like they were just doing this for fun and learning, so I guess it doesn't really matter.
At the same time, they harbor beliefs which are incredibly damaging to both me and those I love, so I keep a healthy distance while understanding that they mean well.
Don’t mistake their kind acts for open-mindedness or acceptance, though.
Anti gay marriage
A person's open-mindedness is independent of whether or not somebody else agrees with them. Otherwise, by definition nobody - including you - is open-minded, since there are people that disagree with you. It's completely nonsensical.
So to recap, the CEO of Chick-fil-a is part of a group of people ("they") who are actually unable - as in, incapable - of honestly considering certain viewpoints? How did you reach this conclusion, and how did you come to be so certain that this CEO is in that group?
Is your understanding that he gave no serious thought to his opinion (i.e. it was just imposed on him), or that he thought about it lightly but lacks critical thinking skills, or something else?
Is there any possibility that he simply has a different opinion than you, and that's all there is to it? Like, he educated himself like you did, thought about it like you did, etc. but just reached a different conclusion than you? Or is the only explanation that he is flawed/limited in some way?
Apologies in advance if anything I've said comes across as demeaning, I'm genuinely trying to understand. Thanks!
I am assuming that his Christian perspective is similar to the one I had. My experiences with Christians since have reinforced this expectation.
Does that help?
FWIW it's virtually impossible to determine the open-mindedness of someone from afar, and to make assumptions about their ability to even consider other ideas seems like a way to insulate oneself from difficult ideas - by declaring that someone is incapable of an honest evaluation of your perspective, it prevents your perspective from being scrutinized (which, ironically, could lead to being close-minded if you're not careful).
A parting thought: most people (a) can be reasoned with if not attacked - you can find common ground and have a good discussion with nearly anybody (b) on every "interesting" social/political issue there are intelligent and thoughtful and just really great people with nuanced and reasonable perspectives all across the spectrum - the extremist nutjobs on the fringes are wildly outnumbered by pretty normal people
So as a random dude on the internet, just a friendly warning that if the above strikes you as untrue (e.g. if you believe large chunks of people are intolerant or incapable of being reasoned with, or if you see a difficult social issue as pretty much black and white and can't understand why so many people don't get it), then there's a good chance that you've been duped and have been sucked into a form of modern tribalism.
Have a great day!
(And really interesting problems can be really boring to work on, and really interesting companies can have a lot of cruft and technical debt under the hood.)
Their GitHub page (https://github.com/Nordstrom) doesn't have much in the way of internal projects, but it has lots of forks that will give you an idea of some of the tools they used.
https://serverless.com/blog/rob-gruhl-serverless-event-sourc...
Here's a recent talk from KubeCon/CloudNativeCon: https://www.youtube.com/watch?time_continue=5&v=xZO9nx6GBu0
Being forward thinking in tech has helped them a lot with regards to being able to move pretty fast on new things. At the same time, a website as old as theirs has a LOT of legacy.
Unfortunately, as this was before the OSS boom that came later, we probably will never see these solutions out in the open.
Seeing down thread they were big DevOps adopters changes my mind of how I saw them as stodgy. Too bad I dislike pretty much all they sell!
This is prime deep fried learning pun territory...
They talk about providing a service for developers to deploy stuff easily, but I'm wondering how that works in practise.
I've very little experience with Kubernetes, the author mentions they run 'mini' kubernetes clusters at each restaurant. Does that mean they have to deploy software/container updates to each restaurant one by one? Or is it abstracted above that level, where they can see ALL of the restaurants as one "big" cluster?
Each individual restaurant gets it's own cluster (we aren't fully deployed yet). There are too many network latency challenges and too much immaturity around federated clusters to take the route (unfortunately).
We currently use gitops, which houses all the configs per cluster in a single repo per restaurant (CRAZY amount of corgis!)... we call that Atlas (the repos) and we made a little pod called Vessel that polls and applies configs in those repos.
We're almost done building something called Fleet that will generated and manage those repos (Atlas) at scale. Ie; a UI where we can say "send this version to 10% of restaurants" and it will regenerate and deploy those configs to all the appropriate Atlas repos, which will then get pulled down by Vessel.
We tried doing all this with Helm but failed miserably. Maybe it was us? But templating vs gitops... the choice seemed obvious.
Other issues included; tiller would sometimes become unstable... version mismatch issues between helm local and roller... lack of a clear, outage free canary deployment... we even found cases where helm would not cleanup after itself during a deployment and retain previous config settings within k8s.
Kidding aside, do you guys do maintenance on Sundays or is that forbidden by company policy?
That being said, it’s really REALLY nice to mute pagerduty for a whole day every week.
Par excellence.
Since we have a highly available Kubernetes
cluster with data replicated across all
nodes ..
Curious what the kubernetes implementation / configuration looks like for the setup they've described. Particularly in terms of HA-cluster k8s configuration, networking solutions integrated, and replicated storage.
Considering there is no AWS / GCE / Azure block-storage primitive to build on, it'd be cool to understand the choices and tradeoffs made in this scenario.
Is all data being stored on something like HDFS or CockroachDB? Or perhaps the k8s minion slave nodes launch containers with Ceph volumes.
I wonder how they do it or why other companies can't or don't do it. CFA stores are so much busier than even an average McDonald's, maybe the efficiency gains are just worth more.
No idea how they handled the chip card shift.
Perhaps CFA can do it even better since they could possibly keep a database of bad cards at the edge. Remember those books that cashiers used to thumb through to verify if a card was stolen?
I have noticed that some chipotle cashiers print the receipt before the card goes through, the CFA cashiers could be doing something similar.
I used to be able to pay for my order with the order taker on his/her iPad using my Apple Watch.
For the last six months or so, I have to go all the way to the pick-up window, and they have to get a manager to unplug a card swipe terminal from the front counter, plug it in near the window, log in to the terminal, and then hang the actual terminal out of the window so I can tap my watch.
Very strange. Hopefully it's only a problem at the one CFA in my geographic area.
Most places have convoluted the whole card process. It's slower than cash now...
We call local clouds "fogs."
Because more than one person on the team has a background in meteorology, and to them a localized cloud event is "fog." I think it started as a joke, but it just kind of stuck.
It helps that our events that require localized cloud infrastructure are sporadic, just the way real fog is. And just like real fog, we don't always know where it will be. Once or twice a year we have to deploy about 150 devices to a different location in the desert. Then as quickly as we arrived, we pack up and move on.
On a side note, it's interesting that Kubernetes is the word from which cybernetics is derived, meaning "governance".
That's just something people think so that they don't have to feel empathy for people who lose their jobs to automation. It's kind of an internal excuse.
The reality is that not everyone is re-trainable. And if someone could easily get a job in customer service, or some other marginally better situation, they already wouldn't be working at a fast food joint.
In addition, many fast-food jobs are temporary/seasonal. A high school kid can get a six-month stint at MacDowell's to pay for car insurance over the summer. A call center isn't going to take him in and train him knowing that he's going to bolt come September.
If a system can pinpoint a prospect is close to onboarding, and customer service rep is a great closer, the software could pass the prospect off to the "closer". The same could be said for keeping a customer who is on the fence from leaving. All of the sudden, a highly skilled closer has data and metrics attributable to the company's bottom line, and in theory, someone like this should become a high priced asset.
In some instances, could automation actually greatly increase the salary dispersion in certain low wage fields?
Right, they can compete with all the other displaced workers for the #1 position most likely to be outsourced.
Hell, many Targets don't even have a customer service desk anymore. They demolish them and assign the workload of both sales and returns to a random cashier.
A lot of people are not qualified for skilled work and will never transcend entry-level fast-food jobs. If you take away those tasks that keep them productively occupied, it might be a good time to start buying stock in industries related to prison management.
Low skill, low knowledge, repetitive work is key target for automation.
Fast food workers wouldn't have large downstream effects. Automated truck driving would though. There are likely entire towns right off of highways that depend on humans passing through daily - restaurants, repair shops, etc. If automated trucks don't stop anymore, a town could literally be a ghost town.
Partially true, many of them went to Mexico. Import of partially completed goods, as well as automation of assembly, continue to drive overall manufacturing output in the USA. But many jobs have been exported. A recent high-profile example: http://www.nydailynews.com/news/national/carrier-plant-movin... and a study discussing overall impact to jobs: https://www.epi.org/publication/heading_south_u-s-mexico_tra...
I can see the automation and analytics helping staff appropriately and offer more options for efficiency and waste reduction.
I hate talking to machines, I hate talking to automated voice menus instead of getting actual assistance from a human being. I hate ordering stuff online and dealing with slight one off cases. The google Ai assistant had me cringing for weeks. I especially have a deep hatred for self checkout lines. They can all burn in hell.
It's like trying to un-discover nuclear weapons, once it's out there you can't go back. Things are going to get more efficient, more automated, and those jobs will be lost. It's not a matter of "if", but "when".
What we need to do is to figure out ways to cope with the change. Find tasks that these workers can do that allows them to earn a living, experiment with universal income to maybe one day not need everyone to work, fund education programs so these people can learn to do something different, provide some safety nets so that these people have time to cope with the changes.
We need to adapt as society, not pretend that this technology doesn't exist.
yes to this.
I guess if you aren't in one of those areas, and you have a business that hires a lot of low income people, times will be good, but like you I don't think the rest of the market can take up the slack; there are only so many people who are willing to pay for laundering.
A key feature of the times we live in is that knowledge is ubiquitous. If an idea becomes known and understood to be feasible, and it serves a useful purpose, you can't hide it from existence. If your restaurant doesn't capitalize on these tools to improve efficiency, another one likely will. This might mean that your restaurant gets be out-competed by a better product elsewhere AND your workers will be out of a job.
Trying to freeze a system that works at a given time is becoming more and more difficult as we advance our knowledge and technology. This paradigm of rapid evolution has been playing out since the industrial revolution, and seems to be accelerating with time.
Maybe it ends badly for us in the macro sense, we'll see. I feel for the folks that can't evolve and get left behind, but I'm not sure how to help them. Resisting the change probably won't work.
However when I see that retail stores are using dialup to do batch uploads of inventory and transactions I wonder if their central IT systems would be remotely ready for something like this.
https://hornet.com/stories/chick-fil-a-anti-lgbtq-donations-...
Imagine being so dead inside you feel perfectly fine, even proud of working for Chic-fil-A, lol
Do you really think Chik-Fil-A is directly causing more suffering by donating to a few charities in amounts this small than by raising in captivity and then killing billions of chickens? I see no way to reasonably argue that.
If I thought Chick-Fil-A shouldn't be supported for either their treatment of animals or for their anti-LGBTQ corporate giving, and wanted other people to stop supporting the business, I would focus on the latter in my messaging, because I suspect it would be more effective on the audience. Attempting to draw conclusions about which of their behaviors I personally think cause more suffering from this is not a good use of your time.
But any effective altruist should have their goals set higher than this, at actually reducing the insane amount of suffering that's created. It does very little good if everyone stops supporting Chik-Fil-A only to flock to another chicken company, which will be the same in almost every way, possibly minus some donations, possibly one which treats animals even worse. Effective altruism is a very different beast than simply appealing to or signaling the crowds' most popular views on ethics in order to persuade.
Chick-Fil-A isn't really in the meat business, they're in the restaurant business, and given that they aren't in a very tiny niche, their menus feature meat. They sell plenty of soft drinks and fried potatoes which reduce the amount of meat as a percent of the total calories.
Finally if you're going to get people excited about animal rights, it's probably better to mention cows than chickens. To me, Chick-Fil-A's ads suggest eating chicken as a lesser of two evils, without ever saying that eating chicken is evil. We also don't have a separate word for chicken meat like we do for cow or pig meat, which I think serve as euphemisms.
I posted this article to our company #random channel and that was the first reply- better chicken than beef!
We do have a word with a French root for chicken, by the way: poultry. As to why the separate name didn't catch on, I can only speculate that it's because chicken are cheap to keep and so the English may have eaten them themselves.
Not all languages have separate words for meat - German perhaps being the most famous. It's slightly disconcerting to walk around a supermarket and see schweinefleisch being advertised.
Chik-fil-a is only a useful example in so far as how well the ethical throughlines match things we've seen many times before, the media has taught us to care about, and how much time and effort political interest groups have spent highlighting those to the public. Ethical thinking on this topic will not be engaged, people will take the opinion their political affiliation and chosen memes have predestined them to take, and they'll act like it's serious thinking.
Parent poster said "Imagine being so dead inside you feel perfectly fine, even proud of working for Chic-fil-A, lol." Substitute in Facebook, Google, Apple, Microsoft or Twitter, and you might be approaching an interesting statement.
Where the ethics gets engaging are the day-to-day business model at any currently big or hot software company. Only a small subset of people have the background to comprehend the issues. And the people saying "let's stop and think about the consequences of this" gets steamrolled by the status quo that has money to make. That's where all the good examples live.
But it's a bit harder to come to terms with when the focus hits close to home. So we have whipping boys like Chik-fil-a. "At least I don't work THERE"
And to quote Chick-fil-a's CEO: "Jesus had a lot of things to say about people who work and live in the business community … Our work should be an act of worship. Our work should be our mission field." So he sees everything they do as political as well.
The Heidelberg Catechism sums it up well:
> Q. What is your only comfort in life and in death?
> A. That I am not my own, but belong — body and soul, in life and in death — to my faithful Savior, Jesus Christ.
HN won't let me reply to your next comment, but regarding:
> a reflection of their religious principles or their political ones
This is actually a common misunderstanding and what I'm trying to explain. "Religious principles OR political principles" means there are actually (at least) TWO categories of ideas a person holds, with specific items (homosexuality, death penalty, etc.) falling inside one of those categories.
That's a secular worldview, and it isn't how Christians think. We have one category: God. God is sovereign, he owns everything including me. As Thomas Aquinas said "all truth is God's truth". When we're being consistent (key caveat), we approach all issues as "what does God want me to do?". Political activities follow from there. We often disagree about those secondary conclusions (what God wants from us regarding issue X), but not the core starting point (ultimate allegiance is to God alone).
> That's a secular worldview, and it isn't how Christians think
And yet literally the previous comment you say
> Not political. Religious.
So which is it?
> So he sees everything they do as political as well.
Obviously we have political views, but they're a only subset of religious views. By analogy, geometry exists and is a legitimate field of study. But if you said "all mathematicians think about is geometry" that would be false because it's too narrow in scope. If he's consistent in his Christian worldview "he sees everything they do as religious as well" would have been true.
Nonsense. A certain subset of our society insists on making everything political.
Most of us don’t like it because it ruins otherwise interesting conversations.
What do you think politics is? Like an actual definition.
What political consequences are there to most conversations?
[0] https://www.huffingtonpost.com/shane-l-windmeyer/dan-cathy-c...
Nothing changes with attitudes like that.
I don't see how that follows.
I do not see how one can claim a criticism of their engineering follows from his statement.
Surely castigating the owner of Chick-fil-A for advocating his personal beliefs is the truest example of bigotry here.
Just because some publications use the word in that way does not mean the definition of the word should be shifted.
As to pedantry, you're the one who started arguing this clearly lousy definition is not lousy, brought up the OED, decided the OED is no good, finally acknowledged the existence of a broader definition but then somehow decided that it's, in fact, as good as identical to the narrow bad definition. I'll be the first to acknowledge I have a habit of getting involved in some deeply silly threads but I'm left utterly flummoxed by this Cirque-du Soleil-level display of mental gymnastics.
I just hope the authors of that lousy definition aren't holding your cat hostage.
He suffers nothing by permitting LGBT people to live as they wish.
We'd call that what it is: bigotry.
I'm fine with a bit of pedantry but ultimately in the real world we need to show a little more intellectual maturity.
Case-in-point: Twitter CEO Jack Dorsey being forced to apologise for eating at Chick-fil-A during LGBT Month. What kind of bizarre intolerant nonsense is that?
Another case-in-point: Boston and Chicago mayors claim "Chick-fil-A values are not [our] values" and refuse to let them open branches in those cities.
Absurd.
Come again?
It seems like you're the one who's missing the point, anyway. You didn't address anything I wrote in my comment. Par for the course, I guess. Just imagine I wrote the exact same comment again, and reply to that if you feel like it.
I'm inclined to divide intolerances into two types: intolerances based on what somebody thinks, and intolerance based on what somebody is. I'm further inclined to model intolerances as a directed graph where nodes represent a group of one of those two types, e.g. conservatives and men might be represented by individual nodes of those two types respectively. Edges represent an intolerance, e.g. there would be an edge pointing from Klan members to black people. The former node would be of the first type, the latter of the second.
Where there's a cycle in this graph we are seeing a mutual intolerance of people with different ideas. An example might be people who think the Earth is flat and people who think it is round. Who is right? It's impossible to say without diving into the issues themselves.
flat earthers <=> round earthers
Note that I'm talking about liberalism in the general definitional sense, not in the modern U.S. politics sense (i.e. "the left").
So, how about this graph taken from real life (the <=> represents two edges, one pointing in each direction):
anti-racists <=> white racists -> non-whites
However, deletion of 'anti-racists' leaves the racism intact. I'd argue that this form of graph appears in many places where there is an illiberal group intolerant of a type two group and a liberal group locked in mutual intolerance with the illiberal group. The inverse setup doesn't really appear because, as I noted before, edges pointing to type two nodes represent a sort of intolerance that is anathema to liberals.
I'd also argue that there is a strong implication that these sorts of edges are the most important to remove—because, as demonstrated, doing so causes the rest of the graph to collapse. They are the "root cause", if you will, of many webs of interconnected intolerances.
It should be obvious that this model denies the assertion made by many modern American 'conservatives' that intolerances are all equivalent and relative, "the tolerant left", etcetera. I would say the water is most muddied by the fact that our 'left' and 'right' do not map cleanly onto liberal and illiberal. It's easy and sexy—but invalid—to apply the same political label to someone who thinks gay couples should be able to enjoy the same legal benefits as heterosexual married couples, and someone who thinks all white people should be killed.
P.S. Damore is a moron and his pseudoscientific screed was legitimately offensive to his co-workers. That's fine justification for him to be fired from his job at a private company. As for the rest of it, we live in a culture of growing illiberalism and zeal for generating outrage. It's a bummer.
A lot of people made a lot of politicized fuss about the Damore thing, but the simple fact is that his document contained offensive generalizations about women and if I was a woman I would not want to work with him. He's free to say whatever he wants; as I said before, though, this is a fine justification for termination.
A lot of people from all sides of the spectrum made it a political issue, because of course they did. The left said Google didn't act quickly or harshly enough and the right acted like Damore's right to speak freely was being taken from him. I'm not one of those people and I'm not interested in discussing the issue in terms of 'ideologies'. Ultimately, as I've said before, what he did does not have to be viewed through a political lens for it to be a legitimate justification for termination.
edit: I said "if I was a woman I would not want to work with him." Actually I would not want to work with him regardless.
I may have. It sounded like you were highlighting the paradox itself, while I was intending to highlight the way the paradox lends itself to abuse. Perhaps ironically, your comments about Damore and the memo are exemplary--the memo contains zero offensive generalizations; it's been picked over and no one has produced an 'offensive generalization'--only phrases that allegedly contain offensive hidden messages. And because of those hidden messages, the mob pats itself on the back for helping to suppress Damore's "intolerance" (by the way, the same mob was rationalizing the New York Times hiring Jeong despite her four-year montage of overtly racist tweets). Again, I don't want to focus too much on any one example, because there are so many others and I don't want to steal attention from the broader phenomenon.
My intent was to illustrate that not all intolerances are only relative degrees of the same thing (as claimed in the popular 'conservative' talking point); some are categorically distinct, and in a web of interconnected intolerances the two types that I identified interact in asymmetrical ways.
I mostly just wanted to write down an idea that occurred to me on my walk home from work. I don't expect something that long to play well in an internet forum. It's meant as an interesting abstraction, not an airtight position.
> Perhaps ironically, your comments about Damore and the memo are exemplary--the memo contains zero offensive generalizations; it's been picked over and no one has produced an 'offensive generalization'--only phrases that allegedly contain offensive hidden messages. And because of those hidden messages, the mob pats itself on the back for helping to suppress Damore's "intolerance" (by the way, the same mob was rationalizing the New York Times hiring Jeong despite her four-year montage of overtly racist tweets). Again, I don't want to focus too much on any one example, because there are so many others and I don't want to steal attention from the broader phenomenon.
Have you read this: https://blog.ycombinator.com/ask-a-female-engineer-thoughts-...
It's quite literally the only piece of coverage I've seen of the Damore saga that I mostly agree with. Otherwise I think I've said what I have to say about him and his document. If you want to project politics onto me, I can't stop you, but I don't think it's productive. My opinions on this issue aren't political.
Ah, I see. I agree, though I wasn't espousing the "popular conservative talking point" (which I've never actually heard before), so I'm not quite sure why the conversation took that particular turn.
> Have you read this
I have, but I don't really want to dissect that here and now.
> If you want to project politics onto me
I don't want to, and I don't think I did. :) Sincere apologies if I offended.
(I will also observe that, where the article notes: "Popper asserted that to allow freedom of speech to those who would use it to eliminate the very principle upon which they rely is paradoxical.", it clearly applies to those who wish to (quoting from your post) "...call out intolerant views and actions and work towards eliminating them".)
Totally bizarre intolerant nonsense.
To make the point more directly for you, I think and I’m guessing the poster thinks we ought not to normalize certain kinds of discriminatory corporate behavior by just looking away and cheerfully admiring their oh-so-efficient tech stack.
The basis was the observation that such articles are,in essence, a "work-for-us" ads, and in that vein, informing the readers of other, broader actions of a corporation is relevant, especially when it pertains to ethics.
[1]https://www.snopes.com/fact-check/chick-fil-a-gay-marriage/
But then again it's highly likely that what they do is not bandwidth intensive and will work fine on a 5Mbps x 1.25 Mbps DSL connection.
You could easily do that on dial-up.
At this point, I don't think it's a problem for CFA.
CFA doesn't try to go into every little town in the nation like McDonald's. It's expansion is very deliberate and only seems to go into very high value locations, where internet presumably isn't a problem.
Moreover, landlords fall all over themselves to land a Chick-fil-a. Almost as much as they do to get an Apple Store. A landlord will make business-grade internet happen, if it means getting a CFA.
The article did mention if a failed node is detected, a next-day restore 'disk' is delivered to the store, so obviously that is faster/more reliable than random ISP.
Also, McDonald's corporate owns all their real estate, franchisees, too.
True I didn't consider the cable investment either. Normally the landlord pays for this but I imagine chickfila probably pays for part of the bill. That I dont know though
When a McDonald’s is located on a pad in a Target parking lot, the McDonald’s franchisee doesn’t own that land. It belongs to the owner of the strip mall.
Ditto for the tens of thousands of McDonald’s restaurants located in office buildings, malls, airports, and inside other stores like Wal-marts.
It’s not the 70’s anymore. The standalone McDonald’s is far from a majority of the outlets these days.
It’s why when architecture firms render new commercial buildings the fake logos often look like those brands. It looks much better in financing presentations and when you’re trying to attract other tenants.
Some agents even Photoshop Apple and CFA logos onto buildings in their private (and sometimes public) presentations to make the projects look more attractive.
Just because you’ve never worked in this space doesn’t make it conjecture.
If it's the latter, would you class the big tech companies as equally non-diverse, since Christians are frequently treated with derision there?
I get the feeling this is a completely unnecessary use of technology.
If you had to go across the internet to the cloud at such a high rate from each restaurant, this would probably come with pretty high failure rates and levels of flakiness. It makes sense to me to process the local data firehose in-house and keep the rest of the common high-level stuff in the cloud.
There just isn't that much data there unless they're attempting to do real-time control of equipment (in which case, just use a regular control system).
I can imagine a requirement for logging potentially, but thats not a low-latency requirement and the use-case for edge-computing is exactly that, low-latency.
Still not seeing anything here but a dramatic overuse of technology I'm afraid.
Requirement for low-latency implies a use of data that is time-sensitive.
What time-sensitive data is there in a single chicken restaurant??
- Credit-card transactions don't have low-latency requirements, its 'nice' if they're quick, same as everything else. The bottleneck here is entirely dependent on your ISP tho. No 'edge-computing' here.
- Mobile orders. This will necessarily go thru to a central server. This is traditional client-server stuff. Again, no 'edge-compute'.
- Order receiving. Simple local data entry. Again, no 'edge-compute'.
- IoT devices. Hopefully these have local control systems without the server being in the loop. Control systems are not 'edge-compute' either.
'Edge-compute' is generation of knowledge at the edge rather than shipping raw data to a central server. This reduces required bandwidth.
What in your system takes a high rate of data and generates a low-rate of data for transfer to a server for further use. I see no analysis of raw data into a more processed form, this is simply traditional data entry and CRUD activites.
They did drop a few hints, though. They are just hints, and going by what's in the blog post, a bit of imagination is required to fill in some of the gaps. However, if you do that, I can envision a few possible needs for low-latency edge computing that are interesting and forward-looking:
- Kitchen automation. Sensors that monitor food being cooked, and help implement a pipeline to ensure quality is consistent. (example here is that neural net monitoring the staleness of fries).
- Inventory automation. Monitor and automatically restock items. Track purchases and ingredient usage in real time.
- Data analytics. Collect detailed time-series data about food quality, facilities maintenance, foot traffic, noise levels, security, etc.
These aren't standard restaurant problems. However, it's an interesting approach to scaling data analysis and automation in the physical world. All of these use cases assume the need for "real-time" data, and they assume that such a thing has business value (which is where I think your critique is coming from).
I'm not affiliated with Chick Fil-A, so I don't know how correct any of my speculation is, but that's my takeaway from reading the article.
Internet connectivity for restaurants stink.
So we put our compute at the edge.
Trust me... as an SRE I would much prefer we shove it in the cloud and call it a day, but alas... no such luck.
The driving need for edge computing is low-latency. There just isn't the need for it in a restaurant.
I do hope they're not doing real-time control of kitchen equipment via something in a container, containers in Linux are nowhere near deterministic enough.
This strikes me as someone who is familiar with only web technologies but unfamiliar with control systems attempting to create a control system.... completely inappropriate tools for the job.
Examples of things we are controlling are timers for food, cameras that can recognize and track food items, drying machines that will automatically trigger and fry fries.
For example I'm guessing it doesn't matter if one of your timers are a second late due to Linux deciding to swap a process out or prioritise some upgrade check but imagine if your time constraints were a lot tighter.
Similarly if you're attempting to track food on a conveyor belt and Linux decides to prioritise the file indexer on your filesystem... oops, a lettuce has been thrown on the floor.
Real-time operating systems exist for a reason.
So far we’ve been satisfied with Linux (Ubuntu 18.04 to be exact) and it’s overall capabilities.
Also, don’t forget that cost at scale is a big factor... doing things perfectly but expensively is not profitable at 2k locations.
Sure, you may be able to tolerate a 1 second 'blip'... how about ten seconds? a minute? do you know for sure how long Linux may put your process on hold for? Linux provides no guarantees at all.
It seems to me that what you are really creating is a traditional control system with web technologies and using the hype-of-the-day (i.e. edge computing) to justify it... with the corresponding increase in overhead and cycles.
This does not seem like a good solution to me, the only novelty here is the inappropriate use of technologies. i.e. you've managed to dig a tunnel with a spoon.
If they can get "good enough" results using Linux and running with technologies that the average developer is more likely to know, is it worth the investment in running a RTOS? Especially given running a RTOS will make any higher level app integration more difficult?
The benefit of running with Linux outweighs the risk of "lettuce on the floor", imo. Yes, there's probably overhead in terms of clock cycles, but there's also probably dramatically less overhead in terms of developer resources.. and when you're running off commodity hardware you can guess which overhead will cost more!
Correctness of the solution is important, otherwise we're just playing at being engineers.
> At some point, you will realise that your job is not to create an idol to the software-engineering gods but rather to have a working product.
> Does this product work? if so, its doing its job.
Something like RTLinux however, would. (https://en.wikipedia.org/wiki/RTLinux)
They could be using VeloCloud and many of those internet connectivity problems would go away. Use multiple internet links with cellular along with dynamic multipath packet optimization
Luckily there's a whole article on that topic. I try to avoid "Did you read the article?" comments, but the first three sentences in the article tell you that this is the question they're answering.
No business case for avoiding more traditional architecture.
What are you computing? This is what everyone is wondering.
Riding the hype-wave methinks....
I’m not a fan of hype... k8s is one of the few hyped technologies that has delivered on its promises.
1) MQTT -- we have a lot of cases where we want to share data between physical or "software" devices in-restaurant, without internet connections. MQTT is our primary channel for sharing messages between things, and for collecting data in general to be exfiltrated to the cloud.
2) "Brain" apps -- these are "smart" applications that collect data from MQTT topics and make decisions about what should be done for a given restaurant process. Today, the reality is we do not automatically cook anything. We do, however, have some restaurants that have more intelligent screens to help them know what to cook at any given moment based off of forecasting models that use the data we collect from many different things. These generally run at the edge to preserve their function in cases where there is high latency or bad WAN connections.
3) In the past we had a POS Server in the restaurant. It was a single point of failure for things like drive thru order taking with iPads and mobile orders. The move to K8s lets us run a microservice that can intake mobile orders on a more resilient infrastructure. To be clear, we have not made this shift just yet, but its something we are working on that will ultimately be transparent to customers but that will make their ordering experience more reliable. We have many cases like this.
Overall, our workloads aren't "heavy" which is why we did an array of three moderately sized commodity compute devices. We wanted a smart, sensible ROI on the hardware. We do have enough business-important apps that need to run in a low-latency high-availability environment to push us out of just cloud (internet dependent) and to the Edge. In the future, maybe this can change and we can use cloud only. It seems Edge is an industry trend though, with many workloads moving towards their consumers (a lot of gaming, but a lot of retail as well).
Great question and I hope this helped -- if not just let me know.
So the current workload doesn't do any of the fancy stuff but is just typical POS stuff with a 'smart' display for orders driven by a process monitoring a few MQTT sources.
You've just put in enough beef to allow for future expansion.
I can honestly say that answer provides more information than the whole original article.
Thank you.
>As a simple example, image a forecasting model that attempts to predict how many Waffle Fries (or replace with your favorite Chick-fil-A product) should be cooked over every minute of the day. The forecast is created by an analytics process running in the cloud that uses transaction-level sales data from many restaurants. This forecast can most certainly be produced with a little work. Unfortunately, it is not accurate enough to actually drive food production. Sales in Chick-fil-A restaurants are prone to many traffic spikes and are significantly affected by local events (traffic, sports, weather, etc.).
>However, if we were to collect data from our point-of-sale system’s keystrokes in real-time to understand current demand, add data from the fryers about work in progress inventory, and then micro-adjust the initial forecast in-restaurant, we would be able to get a much more accurate picture of what we should cook at any given moment. This data can then be used to give a much more intelligent display to a restaurant team member that is responsible for cooking fries (for example), or perhaps to drive cooking automation in the future.
Can you help me understand what you mean by "more traditional architecture"?
I'd be happy to share more about why we chose k8s vs alternatives.
I've enjoyed reading (all) your comments here. Just wanted to chime in and say I have a great appreciation for what CFA does and have had great experiences with the app and in general the entire ordering process at CFA. I've been, in fact, quite impressed. Nice to read about the stuff in the background that helps make that happen.
Doesn't hurt that every employee I've interacted with at CFA has been courteous, kind, and respectful.
Then a couple years ago, one showed up near my work, and I heard they had good chicken. The first thing I noticed about CFA was how their employees acted. I was impressed! Then, I downloaded and installed the app and had a fantastic experience. I was impressed! And their food is great!
Certainly, I agree with the ideology of their founders. However, that is independent of the fact that that I've been impressed! When I go to CFA, I wonder HOW they get me through the line so fast. I appreciate how they've trained their employees in good-old-fashioned southern hospitality. As far as I've seen, at this location, they hire friendly people who care about taking care of the customer.
So, back on topic... how DO they get me through the line so fast?
This was basically a printed out excel sheet with predicted amounts of food to keep ready to serve.
i.e. between 5pm-6pm have 12 chicken breasts already cooked. People follow these charts to know how much stock to pre-cook before the hour basically.
These predictions are usually based on previous years sales for that same day/time + growth + whatever other inputs to the sheet.
It sounds like what they're trying to do is run ML models and make really fast predictions in real time to react to surges of customers. To me this sounds a lot smarter than the approach I worked with.
From my experience when the numbers were off you either generate a lot of waste product or you make customers wait really long.
I imagine the inputs to the ML model are probably something like real time sales data + whatever else.
I think you'd want edge compute here not just for latency but for availability.
You'd want to both run these models fast and make sure even if the crappy internet connection of the restaurant dies you can keep providing these numbers.
You don't have low-latency training of a model and use that same model in real-time.
And in your example (admittedly small scale) 12 items of data an hour is not exactly high-data-rate, not enough to justify racks of machines.
So how often do you distribute a new model for controlling fries?
You cannot seriously tell me that creating a predictive model for cooking french fries requires a lot of computing power.
And the results would have been no different.
This is just hype-driven development (HDD).
It's really exciting to see a company like this just want to own everything they do, and not reach out for 3rd party tools and SaaS and such.
"I have some Raspberry Pi's in my basement and at my vacation home"
VS
"I'm keeping bare metal close to the action with our uniquely scalable Edge Computing implementation"
I hope it isn't Bluetooth or WiFi (or worse yet, GSM/LTE) metadata from their customers.
https://zwischenzugs.com/2016/04/12/hitler-uses-docker-annot...
Don't forget that the people who run Chic-Fil-A support a toxic version of Christianity that, among other things, holds that God himself is going to punish the United States because I'm allowed to get married.
(Also, in case you're a few years ahead of the current western ethical consensus, they torture and kill a massive, genocidal number of chickens every year in order to better monetize related public health crises.)
Maybe don't work for them if you value modern society.