Hacker News Reader

Comment by abetusk | Hacker News Reader

Could you go into more detail about this issue and provide some links to the documentation?

I'm absolutely no expert, I'm just reading about it now, but from the SQLite WAL documentation [0]:

> 7. There is the extra operation of checkpointing which, though automatic by default, is still something that application developers need to be mindful of.

and from their "checkpointing" documentation [1]:

> By default, SQLite does a checkpoint automatically when the WAL file reaches a threshold size of 1000 pages.

I'm only skimming but I see no mention of waiting for an idle time window.

Have you been successful in DoS'ing lobste.rs?

[0] https://sqlite.org/wal.html

[1] https://sqlite.org/wal.html#ckpt

View on HN

On the page you linked:

>However, if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound.

>This scenario can be avoided by ensuring that there are "reader gaps": times when no processes are reading from the database and that checkpoints are attempted during those times.

Dunno, maybe Rails has a built-in workaround for this.

My workaround was to run a separate thread that monitored the WAL size on disk every second. If it went above the target size of 8 MB, my framework would enter "slow down" mode, where all reads and writes were artificially delayed by calling "sleep()", starting at 16 ms and gradually increasing the sleep time based on a few heuristics.

This allowed the application to have short gaps with no reads or writes, so the checkpointer could actually proceed.

https://github.com/openclaw/openclaw/issues/72774

Seems to indicate a collection cron with a manual query to truncate can fix it. This is fun little things I like to learn. SRE Easter eggs.

Those SRE people live on excitement, its basically a gift ;)

Why not grab a rwlock so that no readers or writers can proceed and all existing ones are drained, and then force a checkpoint? What's with the exponentially increasing sleep?

Thanks, this is really embedded in the documentation and is not at all even hinted at in the above documentation.

This seems like a huge DoS security hole.

I encourage you to write a blog post about this. It seems especially relevant.

Reply on news.ycombinator.com