Comment by DivineTraube | Hacker News Reader

Comment by DivineTraube | Hacker News Reader

Parent Full thread

> It is true that with Shopify or similar platforms you aren't able to make backend choices or implement caching like the article has described, and yes it's true that all the themes in the theme store aren't perfectly optimized for page load time -- but the question really is do you want to do any of this stuff yourself?

You're right, that is surely a thing you do not want to do yourself. This is why Baqend does all the browser and CDN caching, TTL estimation, query invalidations and so on automatically.

In terms of the frontend if you do not want to rely on a stock theme - and many larger ecommerce site won't - then you will have to do these optimizations yourself. Fortunately with good tooling available as outlined in the article.

I think Shopify is in fact a great improvement over systems like Magento. But if Shopify is well-enginneered (which I'm convinced of), how do you handle sudden load spikes in the database? Do you reshard and migrate data in real-time? How does Shopify scale the application tier? How can tenants run scalable custom backend code? What happens if a database or server node crashes?

The advantage of Baqend is that its inner workings are published in research papers and blog posts, so people can judge the performance and scalability not only through benchmarks and real-world examples like Thinks but also by looking at the design.

In contrast to Shopify, Baqend is backend-as-a-service which means customers have full freedom on how they design their site or shop. They do not have to live with a server-generated HTML that they cannot change - they can for exmaple easily build a React/Angular single page applications, run server-side JavaScript code, do custom queries, build a native app around the same business logic and APIs, subscribe to websocket-based change streams etc.

While I respect that it took you 10 years to build Shopify, this is usually not an indicator for scalability and a low-latency design. Often such projects end up with solutions like MySQL sharded at application-level, intermingled technology stacks and legacy systems.

Disclaimer: I'm the Baqend CEO and its caching approach is my PhD thesis.

> then you will have to do these optimizations yourself

For the optimizations you listed they actually all still apply with a custom theme. The author doesn't have to do anything. Shopify caches all generated HTML, serves all assets from a CDN, and controls expiry without using TTLs because we know when anything changes. The stuff custom theme authors do have to do is things like bloat reduction / post-first-paint behaviour addition / css pruning etc etc. We're considering shipping mod_pagespeed for all customers too which would help automate a bunch of this stuff.

> how do you handle sudden load spikes in the database

Most of our read load is served by a datablob caching layer[1] so the DB is doing mostly stuff only it can do, writes, and we then have a carefully tuned schema that allows for massive throughput. We also buy really expensive databases -- we just took shipment of over a ton of database hardware in one datacenter in preparation for Cyber Monday. If we do hit our limit, we're able to throttle the number of people in the checkout while maintaining a good experience for customers.

> Do you reshard and migrate data in real-time

Yes

> How does Shopify scale the application tier

We shard by shop and build failure domains called pods that are self contained versions of each piece of Shopify that can failover independently between infrastructures.

> How can tenants run scalable custom backend code

Using Shopify Scripts. They're only available in the domain of carts and checkouts right now, but are rolling out to all our backend domain models in time. See https://help.shopify.com/api/tutorials/shopify-scripts

> What happens if a database or server node crashes?

We have automated failover in place at almost every level of the system, so unless we've screwed up somewhere, no one notices!

> they can for example easily build a React/Angular single page applications, run server-side JavaScript code, do custom queries, build a native app around the same business logic and APIs, subscribe to websocket-based change streams etc

Did the Thinks store end up using any of these features of the platform? I am sure lots of apps need these things, but for web storefronts a lot of this seems like overkill. Shopify also supercharges you when you want to move beyond just a web storefront: we have a mobile SDK for building native apps, connectors to let you sell on Amazon, Facebook, and anywhere you can inject HTML, a world class mobile administration app, a POS solution, a massive app store for other integrations, etc etc etc.

> this is usually not an indicator for scalability and a low-latency design

Again, sub 100ms response time and an order of magnitude more traffic every minute than the peak the Thinks sale saw... that's pretty good!

I'm trying to get at the fact that building a web store from scratch, be it with a handy backend as a service or not, is going to be really hard in comparison to using Shopify. You sacrifice some small amount of flexibility, but you gain so so so much more by buying into the ecosystem. I still think it is unwise to try to compete with other platforms who's core competency is commerce.

1: https://github.com/shopify/identity_cache

Reply on news.ycombinator.com