Hacker News Reader

Comment by erulabs | Hacker News Reader

I hope fly is able to make it. I’m rooting for them - however - I’m starting to wonder if the SQLite push isn’t more “this is fun and interesting to build” and less “customers want this”.

Don’t get me wrong - this is neat - but I’d never suggest anyone to actually use this outside of a fun experiment. The problem with existing SQL dbs isn’t really the architecture - its the awful queries that do in memory sorting or make temporary tables for no reason or read-after-write, etc, not network latency. SQLite won’t fix your current production problems.

If it turns out they’re building this for customers throwing cash at them, awesome. I just somehow doubt it. I think Planetscale has the better approach: a drop in replacement for MySQL/RDS with a smarter query planner. As a production engineer that’s what I want to pay for!

View on HN

I’m excited about SQLite for web apps if only because it is one less moving piece in my stack, which focuses on prototyping and finding product market fit.

If I start hitting hundreds of writes per sec, then, thats either awesome or I wrote some horrible code.

This is exactly where I think SQLite shines! Unfortunately, I don't get to do much green-fielding in my career.

I recently implemented SQLite for a metadata application at work. My constraints in design were pretty concise; the application will only ever need to scale vertically, my data persistence story is a matter of speed and accessibility over longevity, I have plenty of options for scaling disk IO, and my read performance is much more paramount than write performance.

The outcome is that my persistence later is not adding $200/m immediate service overhead cost and my deployment was easy to manage, which is a strong promise made to the rest of the team. I think there's a place for SQLite, but just like any tool you need to know that your design constraints match the constraints of the tool.

I think you're skipping the replicated read only use case, which is our use case, and it's super handy there. but i understand this is a restricted scenario where little could really go wrong, and it could be done other ways.

Lots of smaller businesses could do fine with this if they don't have a write-heavy workload. Like an ecomm shop, for instance.

Yeah it seems to make a lot of sense in ecomm. Product search and filtering on tables in the 1000s rather than the millions.

Any self respecting e-commerce site would want fault tolerance and strong consistency even with potential network partitions, so definitely not SQLite as described in article

They aren't necessarily going to have all this. Lots of smaller ecom shops can run on a single server per region. If you're a North American company selling a few hundred t-shirts per day in NA and EU, it could probably be fine, no? I'll admit I'm not speaking from experience. Rather, I have experience in everything I just said only I was using pg, not sqlite. But I've been very interested in sqlite recently.

Why even build your own e-commerce Web site at all in that case? It’s undifferentiated work.

As with anything, the typical business can use something off-the-shelf, but a certain percentage are doing things differently enough that custom development becomes practical. I had to custom build the billing system for Beaker Studio to properly meter customers. Even Stripe's metering API wasn't flexible enough to handle per-hour metering.

I find it hard to imagine a reason a shop selling 100 tee shirts in a day would need any custom functionality since this is basically the exact use case all these OOB e-commerce tools are built for.

Yeah, you're probably right about that. Main thing that would likely be custom would be the design.

I've worked for such a place. Off-the-shelf solutions are trying to be everything to everyone and you can end up customizing the crap out of them to the point where it can become more onerous than just building your own. The place I worked was also print-on-demand service and had hundreds of thousands of SKUs as well as allowed customers to make custom products and we also hosted some peoples' shops. Shoehorning that into a custom solution was painful.

I work at a very similar place now that uses Shopify. Managing that many SKUs on Shopify is crazy painful.

The thing is is that custom ecom solutions really aren't that hard. The off-the-shelf ones are complex because, as stated above, they are trying to be everything to everybody.

One of the first commercial projects I worked on nearly 20 years ago was a tshirt shop. And the precious company I worked for was a comparatively huge logistics startup.

I’m fairly confident I could spend a month or so writing a custom solution for a shop selling and shipping a few hundred tshirts a day that would save them enough money to break even on the software in a few years (compared to off the shelf solutions).

If I was starting my own tshirt company, I’d definitely do it.

In our B2B space there are tons of custom business rules. For example, you place orders per manufacturer and each manufacturer has its own minimum and reorder amounts and reqs on an order, promotions, business rules, etc. This is for business that have been around 25-30 years.

We did an evaluation on several out of the box ecommerce solutions and none of them were able to meet the requirements that absolutely had to be there. Shopify flat out said no, they can't help us, etc.

> the typical business can use something off-the-shelf

I think we generally have this problem. It's not the typical business that can use that stuff, it's the average business. Unfortunately no business is the average business.

I don't know about nowadays.

8 years ago we built our own ecommerce site + warehouse app (inventory tracking, fulfillment, receiving) for a few hundred orders per day. The goal was to be able to better see profit margins by product, track where the money was going/coming from in detail, along with cleaning up the inventory management part of the operation.

The warehouse people loved the change because it really streamlined the fulfillment process and it basically reduced their errors to zero. Owners loved it because of the all the cost/price tracking. It took about 6 months of work by 2 devs.

Before this they were on woocommerce. It was slow as shit. We looked at shopify but the integrations with 3rd party warehouse stuff was bad. The other solutions we looked at were overly complicated swiss army knives.

This is very close to my experience as well. Unfortunately, it was in reverse for me :( We had a custom solution and people were happy. A new tech lead came in and didn't like that the custom solution was PHP that still had some legacy spaghetti in it, so we switched to an off the shelf solution. It was very painful. People were unhappy.

> that still had some legacy spaghetti in it

I'm going to take a wild ass guess here that you are wildly understating the "bit of legacy spaghetti"

Your wild guess is flat out wrong. When the previous tech lead came in the entire thing was spaghetti. His strategy was to slowly start bringing in some order to the chaos which we did, diligently, over two years making tons of progress. It was a homebrewed framework but it was really nice (he'd had years of experience contributing to Drupal). All the business critical functionality had been converted to this. The spaghetti parts were features that were essentially "complete". Since they never needed updating, they worked fine as they were.

Drupal is the one piece of technology that I did 1 project in and would never work with again no matter what the pay was.

I wouldn’t know I’ve never used it.

It’s not about the size. Issues result in lost sales which means lost revenue. You would want fault tolerance regardless of how big you are.

There is a cost to fault tolerance as well, if you lose $100 per month due to fault tolerance thats worth about 30 developer minutes. Do you really get good fault tolerance for 30m of monthly work?

You do with RDS and/or other hosted database services, hence their popularity.

That's something for me to look into. Like what would it would look like in each? I've never been the dba-type in past jobs. I'm relatively well-versed in SQL but not administration.

https://rqlite.io/ at least looks like an interesting option, here

Why would they need strong consistency? What's shown on the page the user sees is never strongly consistent with the db anyway.

I have this same discussion at work frequently. Everything thinks they need instantly up to date, yet our customers work for 10 minutes on a stale document and no one cares.

Lots of small businesses invent write heavy workloads and don't realize how many thousands of dollars they are spending a month on being nosey.

If you're a smaller e-commerce business, your whole site can probably be cached aside from auth, checkout, and order history.

I can say with authority that this is true! We used to store our whole massive catalog in Varnish.

> The problem with existing SQL dbs isn’t really the architecture - its the awful queries that do in memory sorting or make temporary tables for no reason or read-after-write, etc, not network latency. SQLite won’t fix your current production problems.

In my experience SQL databases are pretty poor at executing the kinds of deeply nested joins needed to return all of the data needed to render more complex UIs. It almost always ends up being faster to simply make nested selects in batches. So latency ends up mattering in these cases.

You can work around this by denormalizing the data, but this often explodes your data size.

Which is exactly where SQLite shines, but I guess that was your point: https://www.sqlite.org/np1queryprob.html

If you want a read only database that’s too large to simply be a JSON file hosted on a CDN it makes sense. That’s kind of niche but not unheard of.

I can't imagine many suitable production use cases too. Alternatives like Planetscale, Supabase or Neon are even easier to use and at the same time much more powerful. If latencies around 50 ms are too much for the specific use case, SQLite with Litestream could be a great solution. Otherwise I'd go with managed Postgres/MySQL solutions.

Replace SQLite with Excel and read it again.

> It's the awful queries that do in memory sorting or make temporary tables for no reason or read-after-write, etc.

Are there any viable alternatives though? I often wonder what an SQL-like language built from the ground up would look like.

The alternative is to write good queries! Alternatively, a better query planner that has a more liberal approach to "I know what you want, not what you're asking for". A great article on this re: Planetscale's approach is at https://vitess.io/blog/2021-11-02-why-write-new-planner/

The first iteration of my website used sqlite. I only switched away because the growing audience made me nervous.

Reply on news.ycombinator.com