Shopify live dashboard | Hacker News Reader

Shopify live dashboard

bfcm.shopify.com

359 pointsludovicianul108 comments

We have been doing BFCM dashboards for the past 10 years now. It's always one of the most fun projects at Shopify and a great tradition now.

You know, it's pretty abstract to run internet software companies. Seeing a visualization like this makes it a lot more relateable even for the folks that work here.

This year we packed it with lots of fun details ( https://twitter.com/Shopify/status/1728040379163771020 ). Like you might see fireworks when a store has their first ever sale, and there are lots of fun city stats which are part generated by GPT4. On desktop you can also open the nerd menu and play around with the settings and share the link to your creation.

Awesome work! You should break down the development in a blog post

+1, it's especially cool how it breaks down different statistics by city. Really great work.

There's some interesting tech details on this thread: https://twitter.com/gilgNYC/status/1728066825013866543

Found this on YouTube: https://youtu.be/T1kkr8VMVqc?si=qq6JvEX2-kBL6jjL

why is looking up Jerusalem, Israel blocked?

It doesn't seem blocked, it says "Not enough data" when you select it.

Is BFCM a well known acronym? I get that it’s about Black Friday but can’t figure out the CM and don’t see it written out anywhere

Cyber Monday :)

Only Cyber Monday started in July, on a Wednesday.

Cyber Monday

that acronym use became steadily more common/popular with marketers / social media in the past 5 years

But, but... Rails doesn't scale!!!!!

You beat me to it.

In all seriousness though, I'd love to know more about Shopify's infrastructure. Pretty marvelous how they handle such an enormous influx of load.

The dashboard shows 25k orders per minute right now (~400/s). A modern server shouldn't even be close to 1% utilization to handle that.

While 400 TPS might not be huge, there is still a lot of processing that goes into placing each order, and it's easy to underestimate the necessary capacity. Latencies add up (a call to a tax service, a call to a shipping service, etc), state adds up, and suddenly it's way over 1%.

Latency doesn't affect throughput. As soon as you fire off a call to an external vendor, your server can work on processing the next request. It doesn't matter if that call takes 10 ms or 5 seconds. Even if it took 30 seconds, that means you need to be able to handle 12k concurrent requests (almost all of which are asleep), which is nothing.

Creating an order isn't a stateless process.

Are you assuming that 1 order is one inbound request, one database transaction, and one outbound response? Because it isn't that. You need to integrate with address verification services, fraud detection, payment services, stock management, communications like email and alerting, telemetry, and more. It's all manageable and a lot of it can be queued, but it's still hard.

I'm not, otherwise something like 0.1% would be a reasonable upper bound. Even if you figure each order involves 20 async steps, that would be ~8k steps per second, which is nothing for a modern server. My old i5 can do 70k simple requests/second (the first type you describe) on the JVM with the app and database on the same computer (so contending for CPU time). A modern server CPU is ~15x more capable than mine without getting into multi-socket.

Put another way, a dual socket Xeon platinum can give you 120 cores. To do 400 orders/second on such a machine, you need to do less than 300 ms of core-time per order, which is an insanely large number. The things you mentioned spend almost all of their time waiting.

I can build that in a weekend!

Why even have a whole system for it? Just expose an FTP server and let users edit an “orders.txt” file

1) Orders are complex and require many steps to process correctly.

2) Orders are only the writes, we have to serve many magnitudes more read traffic and not all of it is cached (invalidations, inventory counts).

1. Yes, of course.

2. How many is "many" magnitudes? The word suggests at least 4, but if so, it'd make more sense to brag about that.

And anything is cacheable if it doesn't affect transactional logic. If the user is looking at a page for 10 seconds, are they going to care that the inventory count is 3 seconds old? At your many orders of magnitude, caching for a couple seconds would give you hits almost all of the time for popular pages.

Also, aren't inventory counts that are displayed to users typically completely made up? Why would you make something like that dynamic and accurate in a high load system? The only reason to have it is to create FOMO or for b2b bulk ordering, and the latter is a whole different thing.

This page doesn't answer that question because it doesn't include how many servers are being used to handle it.

It also could be that something like Java/Go/Rust is doing the hard work.

I’m not sure what the globe is written but I’m pretty sure the comment is referring to the shopify core platform scaling to that many dollars of throughput, and while, yes, certainly there are peripheral bits in a variety of languages, the core is still very much a rails project.

Amazing how concentrated the wealth of the world is.

I think first and foremost it shows in which countries Shopify is the most present.

Other measures indicate similar concentration in roughly the same places (excl. maybe China), such as a light pollution (indicator of energy usage): https://www.lightpollutionmap.info/

China obviously, there's not much they wouldn't just buy on domestic platforms using domestic payment services, but almost all of Asia is pretty dark on Shopify, compared to your map. I guess they're just not a thing there, plus maybe black friday itself. Also, the UK is disproportionately brighter on the Shopify map compared to the rest of Europe. Interestingly, light pollution seems pretty low in Germany compared to the Netherlands, but in Shopify it seems to almost be the inverse.

It was either night or very late evening in most of Asia when you wrote that comment.

Definitely a factor, but Japan and Hong Kong were going pretty strong compared to the rest, so there's at least that. Would've expected Taiwan to show up in a similar fashion, but also almost nothing there.

Black Friday is mostly an American thing that American companies have attempted to export to Europe.

Time zones may also be a factor. And Black Friday began as an American phenomena, which has been exported.

I was surprised how much it has been adopted up in LatAm. For example, you can pick any country on Mercado Libre's list[0] and they'll have Black Friday going. Not "Viernes Negro" either, but the actual words "Black Friday".

---

[0] https://mercadolibre.com/

I find it interesting that of as I look (around 18:00 GMT) that there is so much traffic in GB but not the other parts in europe.

Image reference: https://i.imgur.com/2Les0QX.png

Is it safe to assume that brexit is the reason?

I don't see how Brexit could be part of the reason. Unless the economy is so bad more people are determined to find a bargain.

There's more Black Friday marketing in Britain than some other European countries, and maybe more use of an American credit card based shopping platform.

Probably the fact that Black Friday is an American phenomenon that is successfully exported to mainly Anglo countries. Doesn't impact mainland Europe as much.

Black friday is not small in europe. We (germany) have BF ads all around.

There should be a something similar to Godwin's law but for any discussion about UK. The longer the thread the more likely someone will mention Brexit.

You can flip the argument and say how corrupt and anti-capitalism are other parts of the world.

Despots don't want people they rule to unleash their potential, because that would challenge the status quo and their position.

What is their basis for the "Carbon Removed" stat?

Hilarious, should be carbon added by all this unnecessary consumption made easy by them.

Define necessary consumption. You cannot, because everyone's wants, needs, and desires are different. People will consume. If they do, behaviorally nudge them in ways to offset the harm (verified carbon offsets; not junk ones of course). Similarly to how certain cohorts loudly complain that EVs are not a component of a climate change mitigation plan. Almost 90M cars a year are sold globally; no one cares if they don't think they're the answer, they're going to get sold, so make sure the units that move are EVs.

TLDR Provide solutions not opinions, opinions are like startup ideas without executive behind them: worthless. Direct air capture is the gold standard for carbon offsets. It is expensive, hopefully it will get cheaper, but we must start somewhere until the entire supply chain electrifies.

Well you dismissed the idea of defining it before the commenter had a chance, so are you really making an honest argument here? If a policymaker wanted to ban “unnecessary consumption” they would come up with a list of what that is and whether your purchase would be banned or not would depend on whether you’re buying something from that list, which they can also incrementally grow over time.

I encourage someone to try it to speed run their political career to provide us the natural experiment we can observe.

Constituents will accept taxes and other disincentives (see alcohol), they will not tolerate some political rep telling them what is unnecessary consumption and banning their desired products (broadly speaking, there are always exceptions and nuance).

Walk the pavement like you would for a day running for office, ask your potential voters if they would like this policy. I’ve canvassed multiple times for US political reps, so I have an idea what this looks like.

A small amount of fiat to offset harm is far more palatable. Look at this dashboard spin. Don’t yell at humanity, provide it better options.

The politician’s career is irrelevant. We’re just talking about whether it is possible to define and you’ve been given a rational response suggesting it might be. Heck, it might be possible to ask AI to generate a list of items or criteria.

Many EU governments and the UK have defined lists of untaxed goods, mostly staple foods.

The UK:

> Food and drink for human consumption is usually zero-rated but some items are always standard-rated. These include catering, alcoholic drinks, confectionery, crisps and savoury snacks, hot food, sports drinks, hot takeaways, ice cream, soft drinks and mineral water.

https://www.gov.uk/guidance/rates-of-vat-on-different-goods-...

If you had two chickens, should you give you neighbor one?

Can you define it then? Are Kitchen appliances unnecessary spending? What about designer clothes? VR Headsets? Video Game Consoles?

Sure. You can start by making unnecessary consumption mean “net pollutant” products where the cost of producing them and the waste they create exceed the value that customers pay for or gain from the product, so easy suspects are the types of inventory which are always left over and unsold even during huge sales and promo periods.

They are just pointing out how funny it is to have any sort of environmental savings metric for a sales event known for impulse FOMO purchases.

But that’s the entire point. It isn’t funny, it is legitimate effort to offset the harm caused by consumer demand that won’t go away. My problem is that it is flippant, ignores the nuance, and is borderline ignorant. It has no place here, whereas the math around the topic does. Curiosity and discovery over shitposting.

I agree, but there's another side of the coin that's perhaps best demonstrated by "Climate-friendly" burger patties. We'll do anything to roleplay that we care instead of the hard things.

https://www.vox.com/future-perfect/2023/9/8/23863100/tyson-c...

That said, I don't think responsibility falls on Shopify here. But I did find myself nodding along with them as I find myself scrolling Amazon for "deals" on things I don't even want!

> consumer demand that won’t go away

We created it, it didn't exist even 100 years ago... let's not act like this is some sort of universal law we have to abide to

Id define it as people's needs and not their wants and desires

The site needs to be hosted without TLS at the very least.

Blaming a reduction in transaction costs for increased consumption is seriously missing the mark. I don't know if there's data for this, but I can imagine that by improving the online shopping experience Shopify could have reduced the amount of unnecessary physical shopping trips thereby indirectly reducing carbon emissions.

This analysis out of MIT would support that theory: https://ctl.mit.edu/pub/thesis/environmental-analysis-us-onl...

Fortunately, you are not the dictator of "necessary" consumption.

Looking at Fig 2 in the Executive Summary pdf, most of the CO2e emissions for physical shopping is from customer transportation to store - between 50-80% for the various scenarios they considered. For online shopping about 50% of the emissions are from packaging.

The main lessons I see are

- If you drive, then switching to online will reduce emissions by about 30%.

- If you don't drive, and instead bus or walk to shopping stores, then your emissions will be about 10-20% of the car-drivers or online shoppers.

https://www.shopify.com/climate/bfcm (From the link on the dashboard when you click that stat)

> All orders made through our platform over the Black Friday Cyber Monday shopping weekend will ship carbon neutral.

> Here's how it works: A rigorous formula built by Shopify data scientists calculates the estimated carbon emissions for each Black Friday Cyber Monday shipment. We then team up with Shopify Sustainability Fund partners to remove CO2 emissions equal to the BFCM shipping footprint.

> The result? You can ensure carbon-neutral shipping on all of your BFCM orders.

Which is BS, of course. There's no way to really measure this.

Would you mind sharing your credentials and domain expertise on the topic?

Well you can somewhat calculate the CO2 emitted by a parcel shipped from X to Y. Of course it will depend on loads of factors, like did they use a plane, a truck, a barge with one guy sitting inside rowing gently down the stream, etc… so im not sure how exact their numbers will be.

Basically it means nothing anyway, but companies love to tout how “green” they are, dont they?

I get that its virtue signaling but who is the intended audience who would actually believe it and appreciate it?

Good point, better not try.

It's a cool thing to add to your company website.

They basically pay someone else to “remove carbon” and can the throw this magic number in there. So yea…

Why pay someone else when you have your own “carbon removal” program and just pocket the profits?

Of course, you’ll still need to purchase the Certified Genuine Carbon Credit Approved Organic Green Sticker (R) (TM) (501c3) (NGO) (LLC) to prove you are worthy to sequester other people’s consumption sins.

I see lots of criticism in this thread and elsewhere about that carbon removed stat. I would love to see more productive criticism, where people make suggestions on how they could do it better, instead of just criticizing a company of this scale at least doing something. Or examples of companies that are doing a better job of it.

Personally I'd prefer if they didn't call it carbon removed unless they were actually doubling the shipping carbon removal, and they should call it offsets unless it's all coming from some of their direct capture partners (which iirc Shopify does buy from).

>I would love to see more productive criticism, where people make suggestions on how they could do it better

Not do it at all, because it's obviously bullshit. Even in the best of cases, it's a speculative con. The only purpose is to make people feel good about consuming more, which they do to distract from their miserable and alienated existence in late-capitalist society.

> which they do to distract from their miserable and alienated existence in late-capitalist society.

Is that so?

Shopify has been able to scale to this using a Ruby on Rails monolith and a few Go/Rust services.

Impressive.

This is really cool, and (unlike the Stripe one) doesn't crash my browser nor does it lag on my devices.

The city facts are a nice addition, but most of the cities I search for don't have enough data despite showing up in the suggestions, which is annoying. The customization options are fun to play around with though, and especially airplane mode is a nice touch.

Apart from the actual thing being impressive, I think I’m even more impressed that this runs smoothly on my 2016 iPhone SE.

What library is used to make the frontend globe rendering? Its pretty fast and not laggy on my chrome browser.

three.js Probably something like that https://github.com/vasturiano/three-globe

We built it from scratch with React Three Fiber, a React library on top of three.js. Super powerful framework.

We'll be doing a full technical breakdown in a few weeks :)

Are the obvious reasons that there is so much more activity in GB and RU than other parts in europe others than one can assume (brexit, sanctions)?

I would love to read about changing patterns because of real life events. (Like the rumour that american water plants have problems during superbowl commercials because of toilet flushing but in a larger scale)

Aside from the globe itself, most of the movement in this animation is the frenetic ticking of digits as the "sales per minute", "unique shoppers", "orders per minute" and "carbon removed" numbers change. All of these numbers are actually updated only once every 10 seconds, but the animation makes all the digits scramble for 8 out of those 10 seconds for no reason other than to look busy.

Don't do that.

Don't distort the information. Don't make it harder to read the information. Don't waste the user's visual attention on things that don't matter. Spend it on things that are meaningful.

Wow, these customizations are crazy. Flying in airplane mode makes it look like a war zone.

All I see is all your base belongs to me (Shopify)

The airplanes are doing .001c

Plane or I.S.S.?

Planes on this map.

That arkward moment when sarcasm doesn't convey... Sorry, I am British.

'Sorry, I am British'

Are you trying to be funny mate?

Sorry, couldn't help it.

I like it, but a comparison to last year's Black Friday (at this time of day) is what I would really like to see. But, still, cool.

According to my rough math, they've charged $100M worth of transaction fees at this point

GBP and EUR volumes are pretty similar (just north of 500K atm). I find this interesting. How can one country equal sales of the whole Eurozone? Is that more showing of commerce activity or the popularity of Black Friday in those countries?

I'd guess a lot of shops in the Eurozone don't use shopify but locally-supplied competition.

This somehow crashed my browser. Cool interface, though. I liked the Stripe one a bit more in terms of actual stats and overall I would've liked to see more stats from both. Maybe they'll do reports after.

Some insight on this by one of the devs: https://youtu.be/T1kkr8VMVqc?si=qq6JvEX2-kBL6jjL

Love this. Cool to see how the west coast is still mostly asleep :D

The US economy is something else. Look at all that activity!

Black Friday is a mostly local to US.

Comments in this thread seems otherwise

Love the city facts. Well executed.

Stunning! I wonder how they did the globe animation with real time data.

How do you find shopify shops?

2 fps on my machine

The 'carbon removed' figure is so extremely cringe

Do you have any better criticism than that something is "cringe"? Otherwise this reads like something a teenager would write.

It's obviously utter nonsense. No serious and/or somewhat intelligent person could interpret said figure to have any useful meaning or need this explained to them. It's just plain bullshit.

So I take it you're not a serious or intelligent person if you can't describe what you find "cringe" about a "carbon removed" statistic. Perhaps you should ask yourself why such a simple statistic bothers you so much.

No carbon has been removed from anywhere. It's just made up. It's as simple as that.

Shopify will quite literally remove the carbon from the air via the Direct Air Capture plants that they've purchased [1]. They're not saying "hey buy from us and we won't knock down a tree," they're literally pulling carbon from the air via DACs.

It's as simple as that.

[1] https://news.shopify.com/shopify-purchases-more-direct-air-c...

https://www.reuters.com/business/sustainable-business/stripe...

https://www.shopify.com/ca/blog/carbon-removal

Maybe they could seriously claim carbon reduction if they had an optimized logistics network like Amazon. Otherwise they don't really know how merchants handle and deliver their wares.

Reply on news.ycombinator.com