I had to build a web scraper to buy groceries | Hacker News Reader

I had to build a web scraper to buy groceries

utkuufuk.com

244 pointslord-bazooka167 comments

Of all the grocery companies in my area, only Walmart has it right: Choose an available pick up time, then start online shopping.

All the others let you fill your cart, then at checkout tell you there are no slots available until the end of time. Waste of time, energy, and hope. F U, Target, Safeway, Albertsons, and Kroger.

Walmart is also tied for my top spot for in-person shopping during this. Walmart gets that spot because of their mobile app which does a pretty good job of telling me what is in stock at my local store.

My other top choice is a small (three store) chain in Western Washington called Central Market [1]. (If the name seems familiar, it may be from season 10 of "Top Chef" which was set in Seattle. Top Chef normally sends the contestants to Whole Foods for most shopping, but this was before there were any Whole Foods in the Seattle Area. They used the Shoreline Central Market instead).

They don't have a mobile app or online way to check stock that I know, but they seem to be doing the most of any grocery store I've seen to deal with the virus when it comes to in-store shopping.

1. They've got separate cart return and cart pickup areas in front of the store. Every cart returned gets disinfected before being put into the pickup.

2. They have installed plexiglass shields in the checkout lanes between you and the cashier.

3. They have prominent marks on the floor for you to stand at while inline that provide good spacing.

4. After each customer checks out, they disinfect the conveyor belt area and the credit card terminal area.

They are also nowhere near as crowded as Walmart, and the people that shop there seem to be paying more attention to social distancing.

[1] https://central-market.com/

New Seasons Markets in Oregon and Washington are doing all these steps as well.

Our Whole Foods is also limiting the total amount of people in the store, so you have to wait in a line for people to leave before you can enter

PCC as well. Actually the ones not taking these steps are in the minority. Safeway and QFC are doing a few of these

Don't have any personal experience, but in the UK I think most supermarkets are taking similar steps, at least when it comes to plexiglass shields and limiting capacity in shops. I was surprised at the extent of the distancing measures in some cases -- looks like Tesco defines a route around (some of? I got this from an ad [1], so, pinch of salt) their shops and has 2m intervals marked throughout.

[1] https://twitter.com/Tesco/status/1243670478524383232

> ...this was before there were any Whole Foods in the Seattle Area.

I'm not sure where this story originated. The first Whole Foods in the Seattle area* predates the filming of that season by about a decade and is closer to downtown Seattle than any of the Central Markets.

(*the Roosevelt Square store, still at the original location)

FWIW, Save-On-Foods does this as well.

Save-On-Foods has another - different issue. They normally let you modify orders, which is great. However I've noticed that modifying when there are no additional available time slots for your post code does not work. Your time slot stays reserved however upon trying to check out you will get told that they do not deliver to your area. Eventually new slots opened up and I could update my order but a frustrating bug all the same.

Did you let them know?

They appear to be aware based on their social media responses.

I believe Kroger recently fixed this. Over the past two weeks I’ve had to deal with the insanity of putting some apples in my cart and start the checkout process to see if there was a slot, and if so, grab it and then come back and modify my order with all the items I needed.

But just today using the app I opened it and it asked me to confirm a time slot for my next pickup. No items in my cart - just choose a time, lock it in, and add my items later.

In my area at least, online + curbside pickup was not very common in the past, so it maybe was previously a non-issue because a slot was generally available. Glad to see they’ve made some quick adjustments.

Except when you reserve your cart they have to put a clear timer on the cart reservation, and you can DOS them by reserving all carts and holding them with bots.

During the Corvid pandemic the programmers are the only ones who’re gonna get fed XD

That's insane. All the online grocery shops here in the UK work that way (pick slot first then pick items).

There are still no slots at all available anywhere for either delivery or collection though.

Not quite. Ocado[1] (who I use almost exclusively), lets you fill the basket full of food, and then when you pick a slot, it tells you what's out of stock for that delivery slot, letting you pick alternates if available.

However, the 'correct' way is to pick the slot first, as that way you can see what's out of stock as you shop - this does put you on a time limit though, as the slot can only be held for 1 hour (I think).

So it's not enforced, but it is preferable.

---

[1] pre-corvid-crisis

I’m not sure how crows are related to this but it made me laugh ;)

The click-and-collect as well as delivery shops are all like this around here. Terrible experience, even without COVID.

It was early, not sure I was awake enough to type tbh. XD

Care to give some examples?

I live in the UK and most of the online stores let you pick the delivery slot after filling the basket, which is an infuriatingly poor UX, given the current circumstances.

A trick I picked up while working on an online grocery delivery store was to book a slot, put a multiple of a single item in to get over the minimum fee, then check out. You’ve then got the slot booked and reserved for you, and can go back and edit the order as the week goes on. Just don’t forget to take the 40 pints of milk out of your basket before the deadline for editing.

No, it's quite sensible to do the basket first. If your slot is not available just wait a day or two and try again with the same basket. The vast majority of people have more than enough for a week in their cupboards and do not need to book right now.

Ocado has slots for existing customers. They took a while to get it right but they now have limits on orders per week and show you slots for the next few days. If they are empty just try again later and your basket is still there. I had a delivery a couple of days ago.

I hope not too many people try to subvert systems like this with a script to grab slots, because they're just contributing to the problem.

Kudos to the developers at Walmart Labs. They clearly know their stuff.

If only they'd revisit some of the internal apps. Some of them have been revisited and are great, and some... Well, to be fair, perhaps they lost the code base back in the early 00s.

I find this is is equally a pain in the arse though. In normal times, I add stuff to my cart throughout the week then at the weekend top it off and book a delivery.

Ocado in the UK has the best model in my experience - allowing you to reserve a delivery slot before shopping if you want, but not enforcing this. Unfortunately they have also been the worst-affected from a surge in customers from a technical standpoint and were bordering on unusable for a week.

With ASDA you can do an order and book a slot and checkout. Then you can make changes up until about 11pm the night before delivery.

Just to get this right: It's called online shopping, but actually it is only that one chooses online and then needs to go there to pick it up (basically, no delivery service)? Or are the pick up times, when you can pick it up at your own door?

It gets worse. In my area, five out of six supermarkets I tried asked me to register and took my details to allow me to order _before_ telling me there are no delivery slots. Those five are: Waitrose, Sainsbury's, Tesco's, Asda and Morrison's. The other one was the Co-Op that charges for registration.

Additionally, Asda had an unticked, opt-out tickbox for its spam email ("news and special offers" etc). That's a dark pattern and illegal under the GDPR.

Fuck those people. Unfortunately, everybody else I tried to order from online is out of stock or closed, or it's Amazon where you can buy a can of beans for the price of a brand new Mazeratti (I exaggerate).

The safeway website lets me pick a time slot first, though I don't know if it reserves it. I've never had it disappear out from under me after I selected it but right now there are no slots open at all for me to test.

Why is grocery delivery considered a ‘safer’ option than going to the store yourself. I actually prefer to do it myself. I’d rather be the one to lay hands on all my own products.

Why would you trust an unknown group of people to take all the proper precautions? Who picked the item by hand? Who is this bringing it to your door? Are you sure they washed hands? Did they accidentally sneeze on anything? Is the vehicle sanitary? Also, I would imagine people doing these jobs are probably at higher risk of being exposed, just by nature of traveling & being in public all the time.

I would rather trust my own skills to be responsible and protect myself, rather than other unknown entities to touch all my stuff. All the supermarkets I’ve been to have someone at the door sanitizing carts as the hand them too you. I am hyper aware of avoiding touch anything unnecessary. I can keep as much distance from people as I want. I know my hands are clean when I start, and then avoid touching my face until I sanitize after.

How do you trust the employee who restocked the shelves with all the groceries or filled the stands with fruits?

The point is you’re eliminating a completely unnecessary vector for getting infected. The risk of getting infected from within the grocery is assumed in both scenarios.

I think the flip is more the reason to do delivery/curbside. Symptoms aren't always present so you going into the store could be exposing many people to infection.

Delivery isolates. Yes your driver and handler could be infected, but so could you.

That’s true, but that can’t be eliminated.

It's a necessary step either way. The point is that you're going to have to clean whatever you end up purchasing anyway.

All good points, but these cautions apply before you encounter and touch your purchases as well. In either case, you're cleaning and disinfecting your potentially contaminated purchases inside your home, whether or not someone touched them before or after you purchased them.

I don't think you're eliminating a vector by going out yourself. Not significantly, if at all.

Indeed, someone had to touch it to stock the shelves. I guess if you wipe everything down before you put it away it is a moot point.

Maybe for me it’s just a good reason to escape the prison of my house :)

> I’d rather be the one to lay hands on all my own products.

Someone had to stock the shelves that morning.

I've been keeping my groceries in the bag, sticking the entire thing in the refrigerator, and then not touching it for at least 24 hours.

The only thing delivery/curbside pickup changes is reducing the time spent breathing in close proximity to other people.

> I've been keeping my groceries in the bag, sticking the entire thing in the refrigerator, and then not touching it for at least 24 hours.

Unfortunately, this won't do anything to kill/remove the virus, and may potentially make it last longer. You should be washing your items with soap or spraying them with isopropyl alcohol.

Yes it will, the virus doesn't survive long outside the body, it may not be the best way as it can survive more than 24 hours in some surfaces, but it is way better than nothing. Note that spraying them with alcohol is also recommended.

From the research I've read, it seems that it can survive 2-3 days on plastic, and potentially a lot less on other surfaces. The cold air in the refrigerator probably does extend its lifespan a bit. I normally wash produce anyway, so it's more of an additional precautionary measure than a single line of defense.

Other corona viruses are said to survive for two years in the freezer.

>Why would you trust an unknown group of people to take all the proper precautions? Who picked the item by hand? Who is this bringing it to your door? Are you sure they washed hands? Did they accidentally sneeze on anything? Is the vehicle sanitary?

All of these risks are already present from the stockers and checkout at the grocery store, which is why you should wash/cook everything they touch. Even if most of your items are contaminated, you can still keep yourself safe.

>I can keep as much distance from people as I want.

They can also get as close to you as they want, and one unlucky interaction can make you sick. Limiting your contact to one driver who will do what you say is safer.

Fair enough. Sanitizing every item once at your house works too. Maybe I just have an aversion to a bunch of unknown people touching all my stuff, pandemic or not.

Either way it’s a good reason to get me out of the house, which a hard to come by now days :) It’s good for my mental health. I can’t let myself just sit around being afraid of the world. This thing isn’t voodoo magic. I’m confident I can safely handle myself in public.

> Maybe I just have an aversion to a bunch of unknown people touching all my stuff, pandemic or not.

I don't know if this will help or hurt, but by the time you pick it up off the shelf it's been handled by plenty of people.

> Why is grocery delivery considered a ‘safer’ option than going to the store yourself.

If we both go to the store, you can infect me and the people working in the store. If the store delivers to you, it is easier to make sure you (the customer) do not infect anyone (they can leave the goods outside, no need for close interaction).

As others mentioned, a lot of people will interact with the goods before you - but it is much more feasible for a store to train and equip a handful of people to take precautions, than to expect the public at large to do so - just a simple thing like not being able to change in a proper wardrobe before entering the store, means customers are a lot more "dangerous" than staff.

The primary mode of transmission is through droplets from coughing and sneezing. Minimizing the amount of time you spend around other people is by far the most effective way to avoid getting the virus. If you put away the delivered items, wash your hands, and then don't return to the kitchen for a few hours, then it doesn't matter if the delivery person infected your items. The virus can't last all that long outside the human body.

However, from a public health perspective you should only use delivery services if you're in an at-risk group. Otherwise you're just transferring your risk onto another person and also doubling the number of people who could be infected by this trip to the store (assuming you live alone).

Fair points. Some folks are saying just wipe all your items down before storing, seems like a fine solution. I don’t know if I’d trust how long the virus survives. We might not really know yet, I’ve seen varying reports on that.

I've seen many reports of the virus lasting many days on plastic/hard surfaces. I imagine that's under lab conditions but still worth being aware of.

>I’d rather be the one to lay hands on all my own products.

I've seen this, and similar, opinions about grocery stores that just illustrates how detached the public is from modern food supply chains.

Do you have any idea how many hands touch your food before it gets on your plate? From the fields (where works probably urinate while on the job) to the shelves, it passes through probably 10s if not 100s of hands.

We just sanitize everything that comes into the house. Anything in a package gets wiped down with Clorox wipes d anything else gets washed with soapy water. Bags to into the garbage the kitchen gets Wiped down with Clorox wipes once it’s all put away. It slows things down but it’s the only safe way I can think of to make sure you aren’t bringing anything home from the store. That goes for delivery or self shopping; you have no idea who or how many people touched the products you bought and are bringing home, so better be safe then leave it up to guess work.

Young healthy people shouldn’t be doing this, the slots should go to the ill and elderly, people who are in immediate danger from catching the illness.

To add to this, if the sick can’t get grocery delivery, they will go to the grocery stores.

38% of the people hospitalized with COVID-19 are between 20 and 54. [1] Everyone can catch the illness, everyone can wind up in the hospital, and depending on the state of the local hospitals, everyone is in immediate danger.

[1] https://www.msn.com/en-ca/health/medical/close-to-40percent-...

But not everyone can get grocery delivery, so who should have priority?

>38% of the people hospitalized with COVID-19 are between 20 and 54. [1]

This is odd statistic to give, considering it's not normalized by how many 20-54 year olds are there. So I clicked through to the original study[1] and found a more helpful figure: people between the ages of 20-44 have a hospitalization rate of 14.3%-20.8%, and an ICU admission rate of 2.0%-4.2%.

[1] https://www.cdc.gov/mmwr/volumes/69/wr/mm6912e2.htm?s_cid=mm...

All the more reason to dedicate the slots for the disproportionately-represented older cohort.

Or reorganize to have a central delivery service that goes to all the houses in an area.

That 38% figure is not relevent in this context, because it does not control for the demographics of the population in general. 45% of admisions are over 65, despite the fact that >65 makes up only 16% of the total population [0].

Further, according the the underlying report, of the 22-44 year olds admitted to the hospital, only 2-4% went to the ICU. The 75–84 bracket had 11%–31% admitance to the ICU.

Of the deaths reported, 80% were over 65, while the other 20% was 20–64.

Yes, this virus can kill anyone, but statiscally speeking, it is far more likely to kill you when you are old. If there is a limited supply of a protective messure, we should prioritize giving it to the far more vulnerable.

[0] https://en.wikipedia.org/wiki/Demographics_of_the_United_Sta...

So you get to decide that risking death for a 70 year old is worse than risking permanent lung damage for a 30 year old?

Agree with your point (that raw % of hospitalizations is misleading), but it's 2-4% of total positive cases, not 2-4% of those admitted to hospital.

...which means 62% are people 55 or older. I'm not sure you're proving your point as well as you think you are.

you missed the "healthy" part.

obese, diabetic, or cardiovascular condition means you're in a risk group, regardless of age.

pretty sure the relatively high numbers cited in the us are partly a statistical fluke resulting from the way the sample was generated and partly due to higher rates of metabolic disease etc in the us compared to european or asian countries

How not-young must one be to be legitimate in taking steps like this to avoid contracting this potentially fatal illness?

I’m not yet 40. I have bought all of my food online and had it delivered for years now. Should I now begin exposing myself to the infection risks posed by going inside a grocery store?

This may sound like snark but it’s not. I’m curious about the opinions of people who feel this way.

Are you otherwise healthy? If so and if you find almost every food delivery slot taken then I would say yes, you should be doing your grocery shopping in person. It's not a huge burden to change your behaviours for other people's benefit especially if others also do the same.

And if the slot gets taken by a 20 year old what has been achieved? It's really down to governments to set up food delivery to affected groups. Altruism can't solve this problem.

I'm assuming that if I leave a slot available, there is a higher chance that someone who is sick or vulnerable will get the slot they need. Maybe you're suggesting that all the slots will be taken by 20 programmers who write bots. Even in that case, I can't see how my behaviour doesn't help the chances of that vulnerable person.

> It's not a huge burden to change your behaviours for other people's benefit especially if others also do the same.

Potentially exposing oneself to a lethal virus is indeed a huge burden, and I do not agree that anyone is obligated to put themselves at risk.

No one is obligated to put themselves at risk but you can think of it as a choice that you can make to benefit others. All the doctors and nurses who have come out of retirement to work in hospitals at a time where their personal risk is high are doing it not because they feel obligated but because they recognise it is something they can do now so that fewer people suffer in the future. Choosing to go to the grocery store even when there are lower risk alternatives should be the same.

Yes, the UK supermarkets are asking you to do this. Tesco's CEO "We are at full capacity for the next few weeks and we ask those who are able to safely come to stores to do so, instead of shopping online, so that we can start to free up more slots for the more vulnerable".

At this point everyone is in immediate danger of catching the virus, the ill and the elderly are just the ones less likely to recover from it.

Probably the fairest way to do it is everyone gets in the queue instead of competing for spots, and limits are enforced to help spread demand.

In Bulgaria between 8.30 and 10.30 shops are open for the elderly only. I'm not sure how effective this is but it strikes me as a good low tech move.

Some stores in Washington state (US) have elected to implement this, but it is not a legal requirement.

Happening in Canada as well, Save-On-Foods, Shoppers Drug Mart, Real Canadian Superstore and London Drugs at least - there may be others. London Drugs have also added an hour for healthcare workers and first responders at the end of the day.

Singapore does 7am till 8am, you need a senior citizen card, or be taking care of a senior citizen / person with disability to get in during this time.

I think this is really great. When I go it's often crowded because people are still panicking.

I’m not sure it’s a great idea because older people are far more likely to catch the disease, so you’re crowding the most susceptible, most at-risk demographic together. But I’m not sure there’s an alternative. It’s better if the hours are longer.

Alot of super markets in Singapore are 24/7, a lot of the elderly wear masks, and for the most part people adhere to social distancing. The spread is minimal in Singapore.

British supermarkets have bizarrely cut their hours. Presumably the idea is that staff can stock shelves while distancing from customers, but the effect is to crowd customers together and increase queues.

Same is Slovakia. When looking on handling the covid spread, it seems that generally central/east european countries are doing surprisingly good job compared to western europe, which is rather clueless and is paying/will a rather pay high price for this incompetence.

I think it’s far too soon , and naive, to make such a statement. It’s easy to see some of the mistakes done by Spain and Italy in hindsight, but I don’t see central and Eastern Europe doing any better.

I'm afraid little testing helps keep numbers low. They tested 5000 people since a few days ago in Bulgaria.

Whilst I may be young and relatively healthy (mild asthma) I'm still sticking to delivery. Near to me there are few grocery stores and if I wanted to go to a larger store I would typically take transit. I've tried to minimise my impact by only ordering every 2 weeks as opposed to every week, but it seems like this is how I can have the lowest impact given the situation.

I agree; however, a plea to young people on HN is much less effective than having it as a company policy. They could reserve some/most/all spots for seniors.

Not giving you it to young people prevents them from giving it to vulnerable people further down the road.

If 1 person infects 10 prevention of one transmission can save a 1000.

Everyone ought to do curbside pick up or delivery for everything.

Actually, it depends on where do you live. In OP's country elder people have been given rights to call social services and even police to get their shopping done. Other than that, in Turkey family relations are similar to Italy: the old people and younglings usually live together. So there is always a risk to get the disease in the market and then pass it to an elder back at home. OP's doing the right thing keeping his head down and order online.

It seems like a lot of demand issues highlighted by the pandemic could be solved by queues. Instead of saying "there are no slots available", put the user in the queue. Process the queue in order. The improvement in wait time and demand accuracy would be a win for both consumers and the business.

Some websites do put you in a queue to get to the actual website/shopping. You don't even get to access the website until it's your turn. Like waiting in front of a shop.

Not a great experience if you want to just access your account, or some general information on the website, like contact info, or delivery times.

Web queues are ineffective as shown by sneaker bots. You’d need actual ID style authentication of accounts. If you’re talking about order queues than people will stuff the queue and cancel their orders or just have more hoarding if their multiple deliveries get processed. If it’s FIFO same technical advantages will apply: bots can click much faster than humans ever will. I think the issue is more human nature and things moving to the web creating inequality between the technically adept and non-adept.

Require SMS or callback confirmation of the order slot using a non-Google/Twilio/SIP number (mobile phones or landlines both work here). Provide for one time order cancellation at no cost, cancellations after that incur a $10 cancellation fee.

This should negate any technical advantage from a sophisticated entity. Queues still work, if managed properly.

Cancellation costs may be against local consumer laws.

But I think you wouldn't need it with upfront payment and some simple anti-hoarding rules.

Also the delivery address or payment id could be used in a way to make this fair.

>Require SMS or callback confirmation of the order slot using a non-Google/Twilio/SIP number (mobile phones or landlines both work here).

How does this deal with bots that places orders on other people's behalf?

https://news.ycombinator.com/item?id=22715875

It does not provide for them.

Bots are typically used to buy out a limited supply of something like concert tickets. Clicking quickly or responding immediately to availability changes wouldn't give bots any advantage when there's nothing but a FIFO queue.

If I were setting this up for a grocery delivery platform, I would require the delivery address be provided upfront to enter the queue. Validate and standardize the address using the official Postal Service API (in the US). Prohibit changing the delivery address once in the queue, and prohibit the same address from appearing in the queue multiple times. Those requirements eliminate the mechanical possibility and incentive to partake in any sort of hoarding or profiteering using bots.

But unfortunately there are a limited number of per day delivery slots, so bots will own anything FIFO. They will monitor the page and be the first ones in. There also aren’t really fully standardized addresses per se, it’s actually kinda weird, but what’s stopping anyone using the apartment number next to them and receiving the delivery when notified?

>But unfortunately there are a limited number of per day delivery slots

I guess the idea is to put everyone in one queue, rather than having separate queues per day/slot. Presumably this isn't an issue since everyone's on lockdown anyways, so they're always available to accept a delivery.

But there are only so many delivery people. 1,000 delivery people simply cannot give 2,000 people their groceries within a 1 hour window. The timeslots manage when the deliveries occur. People also sleep at different times, and some essential employees are still working, so you can't simply deliver the groceries to everyone without a known time.

I guess the idea is to put everyone in one queue, rather than having separate queues per day/slot.

And now you've just excluded all the doctors, nurses, police officers, firefighters, and their families from using the service.

Just let people order far in advance then?

Sneaker stores rely on exclusivity so the bots used by resellers are considered a problem. Of course, the real problem is the idea of exclusivity in the first place but that’s besides the point.

Supermarkets don’t have that problem - their goods are not that exclusive and not sought after at scale (anyone wanting to resell the products at scale can get them cheaper from the manufacturer directly).

If it wasn't for the reselling market, there wouldn't be the same level of demand for sneakers. They're just shoes.

>If it wasn't for the reselling market, there wouldn't be the same level of demand for sneakers

You can say that about literally anything. The value of anything goes up if it's liquid.

Not quite. The spread narrows if it’s liquid. All else being equal, a liquid asset is preferred over an illiquid one, but liquidity does not mean something is more valuable.

It just means the price paid or received is closer to the consensus valuation.

I think its primarily the "limited" number produced, which created the market and brought resellers. There wasn't a huge secondary market when Jordan came out, they had to build a brand etc. Eventually there was more demand than the number of products and reselling became a bigger thing. Also I put limited in quotes because a lot of the bigger runs are huge world wide. The main profit that is left is extremely small runs or the smaller sizes (which themselves have smaller runs). But just like a Black Lotus MTG card is just a piece of paper with some ink on it, they aren't necessarily just shoes, they're "limited" shoes.

>Web queues are ineffective as shown by sneaker bots. You’d need actual ID style authentication of accounts.

...and that's rendered ineffective by SaaS bot companies that take your account logins and payment details and places the orders on your behalf, for a fee of course[1]. The economically efficient solution would be to auction off the slots, so at least the grocer is reaping the profit rather than some third party. It also has the benefit of capitalizing the grocer so they can hire more people and/or buy more equipment so there are more slots in total. Also, if it bothers you that some FAANG programmer can afford deliveries than some old granny, this can be alleviated by reserving a certain % of the slots to "needy" people (eg. old, disabled, sick, immunocompromised) subject to validation.

[1] kinda like https://en.wikipedia.org/wiki/Domain_drop_catching

This assumes it's easy to resell a slot, which may not be correct (eg if you have to put in your delivery address when entering the queue). Tickets are a very different commodity.

>This assumes it's easy to resell a slot

That's not what I'm saying. Rather than buying a slot (or an account) from the bot operator, you'd give the bot operator your order and address/account/payment info, so when the slots become available, the bot operator can place an order on your behalf.

Isn't that just what amazon should already be doing?

They need an option for people who are flexible with their delivery times, put in your order and they assign you a delivery slot when one opens up.

If a bot can do that, then amazon can just do it themselves and actually improve the user's experience.

How many people are working from home right now that don't can _when_ groceries get delivered other than "the next available slot"

Totally agreed. It’s the classic “hiring people to wait in line for you” solution. Eh I don’t think surge pricing should be put anywhere near this problem and it’s certainly not economically efficient in the classic sense, that is creating the most economic output and benefit for the consumer.

You are solving the wrong problem. Curbside pick up is about as safe as delivery for 1/3 the labor cost and the grocer already has enough people if he uses everyone.

A food delivery auction would in some segments drive up the cost drastically when some can least afford it fuelling panic. This panic would fuel buying and hoarding.

Meanwhile the poor flock to the shelves to buy before the rich can pay to have it all delivered for hoarding.

Surge pricing for groceries during a crisis may be one of the worse ideas I've seen on hacker news.

Grocery delivery needs an address. You could pretty easily limit to one queued order per address, and limit order changes.

One hack around address limits is to put in a unit number. 1 Park Avenue or whatever in Manhattan will have multiple units in it. And if you live in one of these units you can add a letter, like "Unit 522B" which will just get delivered to your unit 522.

Some companies have databases that know the layouts in each address but many do not.

And when the grocery delivery person comes and sees the addresses don't match, your address gets blacklisted.

Never happened to me. If there are no letters after unit numbers, and the delivery person sees a delivery for unit 522B, they'll leave it at 522.

I mean for Amazon Prime you have to have a Prime susbcription to even play, so the $129/year fee is a good barrier to bots.

It seems to me that that they already have a 4-day-long queue, and the error message is in fact a "the queue is full" error.

Why not a longer queue? I don't know, but I'd guess people forget about their order and buy redundant things, causing support and customer satisfaction problems, if the queue is longer than 4 days.

Probably because if I'm going to wait four days anyway, I'll probably just say "Gee, guess I'll just drive down there and get it".

Ocado in the UK do exactly this, using queue-it [0].

On some sites, apparently queue-jumping is as simple as disabling javascript (n.b. Ocado [1] have been in the game longer than most, and this does not work on their site).

[0] https://queue-it.com/ [1] https://ocado.queue-it.net/

This does work on Ocado, but you also need to disable redirects also by the way

Ocado in the UK has been having queues to get onto the site and limiting people to one slot per week. They are now giving people who have been paying for delivery passes before 2020 priority access to the site one day per week (divided randomly into 7 groups) to book slots. I presume they are also doing something similar for vulnerable customers. I have wondered whether if it'd be possible to hack the queuing process somehow but imagine the number of people capable of doing that would be tiny and am too honest to consider myself! I'm more wrangling with whether it is ethical to take a slot at all (but we are getting allergen-free stuff for another family and don't want to let them down).

It is a good idea. And may be telling the waiting list number or queue number will help. Because otherwise the buyer has to every-time visit the page. User acknowledgement through queue token or waitlisting number will help.

I had to do the same thing to get an Amazon Fresh order. Well, not quite the same thing, I ended up writing an Selenium project that automated the Amazon Fresh checkout process till it got to a delivery window page that didn’t complain there were no slots available.

If I can’t get groceries, it’s not the end of the world for me, since I can call on family if necessary, but I’d rather not make their stressful life any worse right now, since I’m not the only one in my family’s circle that needs help.

There are no great answers, since I might have taken a slot from someone that can’t write a web tool to scrape Amazon, and doesn’t have a family to fall back on. It sucks on many levels.

Anytime I try to automate/scrape Amazon I get captcha'ed after a 1-2 attempts. Anything special you're doing with selenium?

I have made Amazon Fresh bot doing exactly the same using puppeteer.

Amazon Fresh (until a few days ago) was releasing slots 3 days out each night at midnight, so I've been saying up to place orders. Now it seems incremental throughout the day, but I've gotten lucky and nicked a spot. However, they don't have everything I need even normally. Now, it's even worse, so I have to order from multiple places. The local Mart's that deliver are still opening their slots at midnight, though usually for 8 days out, and again items ordered often go out of stock before delivery. It feels like a sureal video game.

Still better then Wal-Mart’s site where everything is in-store pick up, and Target’s site where everything is out but you can’t tell until you click into the item and try to put it into a basket. My only gripe with Amazon is that it’s too hard to distinguish between them and their third party sellers, and it’s a pain to deal with the sellers. I also wish there was a button to report price gouging - I was looking for some emergen-c packs to flavor my water, normally it’s cheaper then cool-aid and less hassle to mix, some fool is charging $109 for a box of thirty packets.

Amazon profits off price gouging, though, because they get a percentage of the profits. And since price gouging isn't strictly illegal -- it's seen as a perfectly valid part of capitalism by the state -- there isn't really any way to disincentivise it.

Yeah, emergen-c is either sold out or price gouged, a pattern seen across most vitamin-c products. But take a look at this-- it's a powdered form that can be mixed. Water works, but OJ might taste better. And the price per dose is very good: https://www.amazon.com/Ascorbic-Antioxidant-Smoothie-Pharmac... As of this comment at least, it's in stock, about $28, there are other brands available too (search vitamin c powder) and it's at least as price-efficient per dose as normal pills/chewables.

Also, if you have Amazon Fresh, Amazon's Whole Foods, or Amazon Pantry (just opened again) available in your area, 3rd party price gougers aren't really an issue.

>> everybody stays at home and especially avoids crowded indoor places like supermarkets.

I know I shouldn't do that, my doctor has advised me against it, but I have a headache today and this made it worse and I can't help myself so: this is an oxymoron. If everybody is staying at home, then supermarkets cannot be crowded. If supermarkets are crowded, then everybody is not staying at home. Both things can't be happening at once.

Everybody stays at home except for essential shopping in supermarkets. Not every statement is meant to be taken 100% literally, but I'm guessing you knew that and are being pointlessly pedantic.

You mean you're guessing I'm a pointless pedant? Thank you for your contribution to my Sunday headache.

I think we've all underestimated just how many people "everybody" is. Normally we're spread out. Now we're either at home, the park, or the grocery store, and suddenly it seems more crowded even though "everybody" is staying at home.

Nobody drives in New York - too much traffic!

Shameless but very relevant plug: Monitoring a website for changes and sending data somewhere else is exactly what we do at https://monitoro.xyz

It’s the difference between maintaining a bunch of scripts and making sure they don’t stop working abruptly, and having the data you wanted without headaches.

It also wouldn’t make for such an elaborate article, but that’s the cost of simplicity if you just want to get things done.

We recently ran into this problem at Cooklist in the US. We ended up setting up the ability for our users to place orders up to 3 days out.

Our system then books their pickup order at midnight when the new pickup slots are opened. We are seeing that most pickup slots are filled for the next day by 9am.

It is interesting that the stores don't allow booking several days in advance anymore. It makes sense though because even with booking for the next day we are seeing that 25% to 50% of the order is substituted due to out of stocks.

I've done the same thing in The Netherlands, as I'm doing the shopping for my grandparents. I noticed that I don't need it yet, but I did see that I might need it in the future since I need to stay awake until 23:59 at night, so I can snatch a slot at 00:00 at the most famous supermarket here.

My script isn't fully ready, but it's 75% complete and I'll complete the final 25% when I really do need it.

Also, these things are quite easy to make, so I expect a lot of techies to eventually do this.

Few months ago someone had posted open source scraper that got 1k+ votes (or maybe not) - it was used by NYTimes for realtime alerts.

Someone remind me please.

You're talking about Huginn, right?

https://news.ycombinator.com/item?id=21772610

Yes, yes, thanks.

To clarify, Huginn is a framework/app server for scrapers and doing rest calls etc based on them

Tips: the first session cookie is probably returned if you do a GET on the homepage of the website, so you can simplify your script by

Declaring the requests session, doing get on the homepage and then doing get on the endpoint

It will also prevent issue with session expiration

Hey do you have any suggested readings on how to better understand these structures and your reasoning?

I’ve done a few of those scrapers and mail to notify projects in the past, but I’ve recently switched from using email to a telegram bot, it’s pretty great. Very easy to implement and doesn’t spam your inbox.

I must say, from the title I was expecting to be disappointed - but this is a very nice and clear example of a useful little hack combining python requests module and a little web page introspection.

I think maybe I'd have gone down to every 5 minutes, to be a little nicer to the end-point (especially when sharing the code) - otoh once a minute isn't to much of a hammer, I guess.

I had a similar idea on Saturday and wrote a puppeteer based script which monitors Tesco.com and then pushes a screenshot to my phone if it finds a slot: https://github.com/paulmaunders/delivery-slot-bot

hey - just tested your code - works great. I am no coder, but have installed and it runs fine. Quick question 1) Its now only checking for Group-1 which is 1hr fixed. Would like to include Group-4 which is the flexible slot. 2) Not just send notification - book the earliest one available.

How to do this?

The supermarket is a post WWII invention. Labour was expensive, so why not have the customer walk around and pick. Typically before WWII shop clerks handed items to waiting customers across the counter.

As any invention it can be reinvented.

Organic farmers started doing deliveries 20 years ago, to compete with the big warehouse boxes.

Would be simpler to just plan four days in advance. Shopping used to be done once a week, or even once a month for bulk and frozen goods. Now it seems to a be what people do instead of going for a walk in the countryside. Makes even less sense now that you don't have to leave your bed to do it.

They didn't mean that there were free slots after 4 days.

Maybe there are local dynamics at play (we are in a city in The Netherlands). But my wife has been doing the grocery shopping (since I don't have a driver's license). We would normally shop every day, but she is now going every 3-4 days. And after the initial 3-4 days of hoarding, the grocery stores are far less busy, probably because people are buying for multiple days at a time. She has had no problem keeping the necessary 1.5-2m distance.

Greasemonkey seems a much better weapon.

how?

The picture with his cookies contains the suspicious keys "hphone" and "hmail", which likely show md5 sums of his phone number and mail respectively.

Especially the phone one can easily be brute-forced (small search space). I wonder why they hash this in the first place?

I hope they're salted...

Everyone should be ordering online regardless of age or increased risk. Limiting contact with individuals is the only way to stop the spread of this disease. The answer is that the surge in demand for delivery should quickly be filled on the supply side by delivery companies. The increase in business should go to providing hazard pay, PPE, and benefits for the supplying workers. This limits the number of people in contact with others while supplying jobs for those who have been unemployed during this time while increasing pay and benefits for those workers due to their increased risk vs reward. This is how capitalism is supposed to work in theory at least, to quickly adjust to market needs.

I literally build the same thing here in the UK. I knew I couldn't have been the only one lol.

Good job.

The best part of "blood in the streets" is old school hacker news posts like this. Kpi of creative progress to come!

next, booking your grocery pickup as a service /s

I couldnt find an appointment slot with safeway this past week. Ended up writing a quick script on lunch that did the trick. Ran it as a schedule task to run every 5 mins. Pretty hacky and unpolished but worked, got a text message a day later and checked out. Not sure if anyone could use it but here it is. https://github.com/christru/safeway-delivery-check

I’ve just been going to the grocery store as usual and it works great. You show up in the morning and there’s no line yet and shelfs are full. Later in the day they meter entry so you can keep distance inside.

Don’t be that ass who makes others do your shopping just coz you’re too afraid to.

In store: you need to have someone stock the shelves, someone walk through the store and pick groceries, and someone handle the payment in person. Chance of infection: depends how long since someone coughed in the store.

Online: you need someone to stock the orders and wait for people to drive up for curbside pickup. Chance of infection: low.

Surely the second option is both safer and more efficient?

If you work in a supermarket you know exactly where each product is. Standing in line is also a huge waste of time.

A lot of online options send a person who goes into the store and walks around to get all your stuff. Instacart for example.

Ah, that's quite inefficient (and probably costly) because it's a third party without integration in the logistics chain.

I've tried food deliveries provided by two local major grocery chains. As far as I understand, one of them has the stock-shelving people pack the deliveries (so, before hitting the shelf), and the other seems to pack pack the direct deliveries in the same logistics centres from which they supply their smaller shops, so that stuff never even gets delivered to a store; and the limited overhead means that they offer free delivery for non-small purchases.

Probably different regions but none of my grocery stores do online like that. They have someone walk the aisles and shop like a regular shopper, and check-out at the cash register like a regular shopper. They will even give you their receipt to verify the order. You're just offloading the risk of infection to someone being paid to grab groceries.

Think of the systemic effects. If everyone ordered online, fewer shoppers would go to the stores, changing the topology of infection from a mesh to a star. Spokes would likely develop immunity, and would be less surface area to test, if we had any tests /s

edit: not star, hub and spoke, sorry

Partially I'm guessing this is an artifact of just-in-time delivery - almost no storage outside of shelves (there's no central storage location that could fulfill orders).

Still, if the stores handle this themselves - it's much more feasible to take extra precautions - from using a locker room to change into a uniform (not bringing outdoor/home clothes into the store) - to using disposable gloves etc.

Ideally, stores would offer only pickup and delivery - and not allow in-store shopping at all.

Would probably make sense to increase staff as needed, as any additional staff needs would scale with increase in revenue.

Having fewer people be out and about in supermarkets protects against more spread and helps flatten the curve.

Ordering groceries creates employment at a time when unemployment is higher than literally ever before.

You are absolutely correct. There are not enough delivery capability for those who really need it. If you are young and healthy you should be ashamed to be getting your food delivered. It is cowardly.

My esteem goes to those working in the stores, at the tills etc.

Hehe, I had to do the same a week ago. The only difference is Slack’ing instead of mailing and also cart prepopulation because those delivery slots are taken out almost instantly in my city. This is in the EU btw, not specifying the country or chain to, well, to be able to survive longer.

Reply on news.ycombinator.com