Data loss is a pretty serious problem.
Do you have any more information about the situation?
Could it have been in the hand spun partitioning logic instead of SQLite?
What was the ingestion throughout roughly?
Do you have any more information about the situation?
Could it have been in the hand spun partitioning logic instead of SQLite?
What was the ingestion throughout roughly?
It's been more than 5 years but from what I remember, it definitely was _not_ the partitioning logic (the sharding just meant we had a huge amount of files that were hard to organise). But a single consumer doing heavy writes on a single SQLite file would see enough traffic that pretty soon you would start to see errors and your writes would start to break.