HNHacker News
TopNewBestAskShowJobs

hyc_symas

758 karma · joined February 21, 2013

submissionscomments
hyc_symas··on Garage S3: LMDB Corruption Post-Mortem
> A power outage interrupted Garage mid-write on the LMDB metadata database. LMDB uses mmap’d I/O and, by default, operates with MDB_NOMETASYNC and MDB_NOSYNC (Garage sets metadata_fsync = false by default)

No, Garage disables LMDB's safety mechanisms by default. By default, LMDB performs whatever necessary syncs and is 100% crash proof, even in the face of power outages. Garage's problems are of their own making, not because LMDB has unsafe defaults.

hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
The txnID was added to the page header to enable support for incremental backup. As a consequence, it's sufficient to compare a page's txnID to the current txnID to know if it's dirty or not, and spilled pages don't need a cleanup pass to clear their dirty bit on commit so commits of large txns are faster now.
hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
write/fsync can be faster in a large dataset because writes let the filesystem know an explicit list of dirty pages, so fsync only needs to deal with them.

mmap/msync gives no hints about which pages are dirty (unless the app tracks them itself and msyncs them individually, which would completely defeat any reduced syscall advantage of using a writable mmap in the first place) so the entire map must be scanned for dirty pages.

In practice, the expected performance advantages of using a writable mmap just aren't there, and coupled with the ease of silent corruption, it's best to never use that approach.

hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
By the way, you're wrong on both points - the cache of page mallocs is not infinite, and it does spill dirty pages to disk when necessary. And the latter is what bounds the number of malloc'd pages.
hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
You're dreaming. None of your explicit memory control operations mean anything in practice, because today everything runs in VMs with no actual control of the underlying hardware. Probably co-resident with an unknown number of other tenants.

As for what you claim the paper's authors were saying - I quoted their text verbatim. Your interpretation is not what they said.

They claimed using mmap safely is impossible, and using it correctly requires more complexity than a traditional DB design. The safety claim was already disproven by multiple researchers. To prove their second claim they would have had to produce a DB that did traditional buffer management and was simpler and more performant than using mmap. They never did any such thing, nor could they.

hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
In the time it takes for your query optimizer to dissect a query and "look ahead" LMDB would have already answered a million queries. You think your magical "future oracle" is zero cost? How many KLOCs is it? LMDB's hot paths fit entirely inside a CPU's L1 cache.
hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
Indeed, there's no need to use the doc website. There's nothing there that isn't embedded in the LMDB source code. All of the docs are generated from doxygen comments in the source.
hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
If you're downloading binaries from a plaintext documentation site, I think that's on you.
hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
Nonsense. The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm. But when you do it yourself you have to juggle the fine details of which indices to prioritize, and you will never get it perfect. If you're not running a dedicated machine, as soon as any other processes run all your careful tuning goes out the window.

Since LMDB manages multiple tables as a tree of trees, no fine tuning is needed. The internal paths to every hot page automatically take priority, regardless of which index or how large each index is. So a simpleminded LRU always makes optimal use of available cache, regardless of access pattern or other load on the system.

hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
Obligatory "that paper is garbage" https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...
hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
Irrelevant, since LMDB doesn't dirty the pages in the mmap. The mmap is read-only.
hyc_symas··on Lightning Memory-Mapped Database Manager (LMDB) 1.0
LMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.
hyc_symas··on How Monero’s proof of work works
> The price is intended to decrease.

No. Monero's tail emission rate was specifically chosen to be less than the rate of global gold production. Do you claim the price of gold is intended to decrease?

A continuous emission like Monero's doesn't equate to inflation/devaluation. It allows its userbase to expand organically, without artificial scarcity pumping its price.

hyc_symas··on How Monero’s proof of work works
I don't know what the current versions do, it's been a while since I touched that code.

I have no reason to lie, I'm not selling anything. Bitmain is selling mining hardware, take a look at their claims. They've had 7 years to try to crack it.

hyc_symas··on How Monero’s proof of work works
FPGAs have no particular advantage. You could dedicate a chunk of their resources to implement a softcore CPU but it'd be several times slower than a real CPU.

The random programs change too quickly to just implement them directly on an FPGA. Reprogramming the entire chip like that takes too much time.

hyc_symas··on How Monero’s proof of work works
> Loved your 2019 talk on "is XMR still ASIC-proof" – is it still, in 2026, in your opinion?

Yep. Nothing about computing architecture has changed.

hyc_symas··on How Monero’s proof of work works
>> just killing the process will never corrupt the blockchain DB

> I would love to show you how easy this is to reproduce, even on fresh installs of Ubuntu and/or MacOS on otherwise-stable hardware (never tried Windows... easier?).

If it's so easy to reproduce, you should be able to screen record a session with two terminal windows:

1 with monerod running and syncing the blockchain

2 send a `kill -9` to the monerod

1b restart monerod

And then we should see the error message you're referring to.

hyc_symas··on How Monero’s proof of work works
> If you don't sync then you're not (cannot be) a fullnode / network verifyer / ringsigner.

I was talking about database sync, not blockchain sync. You don't need to use safe sync mode if you don't have to worry about machine crashes. And just killing the process will never corrupt the blockchain DB.

> This may be old behavior... I go way back

On this particular point, I go way back further than you.

hyc_symas··on How Monero’s proof of work works
Light mode is used mostly by the monerod, validating incoming blocks. But on a machine with sufficient free RAM, monerod can also be set to use Fast mode instead.

The post described both modes. The only difference is that Fast mode processes the cache to generate the full 2.1GB dataset, so subsequent programs can just reference it as needed. Light mode uses only the 256MB cache and generates the required dataset values individually, on each access. That saves RAM but costs more CPU time.

hyc_symas··on How Monero’s proof of work works
The limit we set at the beginning was "no one can design a custom device for RandomX with more than a 2:1 efficiency advantage over general purpose CPUs". That is and will forever remain true.

In reality, no one has been able to build any device for RandomX that isn't actually a CPU. The closest thing to a "mining ASIC" is just a bunch of RISC-V cores.

hyc_symas··on How Monero’s proof of work works
Since the programs are randomly generated, there's no guarantee that any particular program always uses some number of any particular instruction. There's only a probability, X/256 instructions will be somesuch operation.
hyc_symas··on How Monero's proof of work works
Monero is disinflationary, not inflationary. The rate of new coin emission is only enough to maintain equilibrium with the rate of coins being lost (due to people losing wallet keys, etc.). So your comment about being forced to keep earning doesn't apply to Monero.
hyc_symas··on How Monero’s proof of work works
You must be on drugs. There is no separate specialized verification function. It's the same algorithm for verification as for mining.
hyc_symas··on How Monero’s proof of work works
PoW is useful in far more situations than PoS. A derivative of RandomX is now used to protect TOR too (Equi-X). https://github.com/tevador/equix/blob/master/devlog.md
hyc_symas··on How Monero’s proof of work works
> Once you kill the process, your local blockchain is [most likely] unusable.

Totally false. LMDB is perfectly crash-proof in that scenario and killing the process never damages the DB. The only thing that's not guaranteed is turning off syncs, in the face of an OS crash/power outage.

If you don't sync, you're not abusing the SSD. If you run on Windows, the OS is too unstable to use without safe sync mode though.

hyc_symas··on How Monero’s proof of work works
No, Bitcoin is absolutely a waste by every measure. It was a prototype that never should have continued after all of its flaws became apparent.
hyc_symas··on How Monero’s proof of work works
The design has been working perfectly from 2018 till today.

https://old.reddit.com/r/Monero/comments/1h6e4nk/randomx_5_y...

Most miners use AMD Ryzens. Couldn't tell you the actual breakdown of CPU types in use. Apple's M series CPUs are quite efficient at it too. Bitmain now sells a "Monero RandomX Mining ASIC" which is just a bunch of RISC-V cores, seemingly based on Sophon SG2042 SOCs. There's nothing special or more cost-effective about their product.

You can mine on old smartphones quite easily. I use a bunch of old Android TVboxes myself. Their hashrates are nothing to crow about, but their hashes/watt are still competitive with faster CPUs.

There is a RandomX V2 that will be deployed soon. Its main improvement is even cheaper verification cost.

hyc_symas··on How Monero’s proof of work works
Sorry, copy/paste error. Fixed now.
hyc_symas··on How Monero’s proof of work works
ProgPow was ridiculously simple and would never have accomplished its goal. I covered it briefly in my Monerokon talk as well.
hyc_symas··on How Monero’s proof of work works
Just use a Linux laptop with a working battery so you never have to worry about power outages or other system crashes. In that case, you don't need safe sync mode, and you don't have to kill your SSD.
Page 1 of 16Next →