Comment by n2d4 | Hacker News Reader

Comment by n2d4 | Hacker News Reader

Hey HN! For a few months, I've been building an in-memory version of Postgres at work. It has full feature parity with production databases.

The cool thing about it is that you don't need any external processes or proxies. If your platform can run WASM (Node.js, browser, etc.), it can probably run pgmock. Creating a new database with mock data is as simple as creating a JavaScript object.

It's a bit different from the amazing pglite [1] (which inspired me to open-source pgmock in the first place). pgmock runs an x86 emulator with the original Postgres inside, while pglite compiles a Postgres fork to native WASM directly and is hence much faster and more lightweight. However, it only supports single-user mode and a select few extensions, so you can't connect to it with normal Postgres clients (which is quite crucial for E2E testing).

Theoretically, it could be modified to run any Docker image on WebAssembly platforms. Anything specific you'd like to see?

Happy hacking!

[1] https://github.com/electric-sql/pglite

This looks really cool, awesome work!

Correct on PGlite only being single user at the moment, and that certainly is a problem for using it for integration tests in some environments. But I'm hopeful we can bring a multi-connection mode to it, I have a few ideas how, but it will be a while before we do.

There are a few other limitations with PGlite at the moment (related to it being single user mode), such as lacking support for pg_notify (have plans to fix this too). Whereas with this it should "just work" as it's much closer to a real Postgres.

I think there is a big future for these in-memory Postgres projects for testing, it's looks like test run times can be brought down to less than a 1/4 with them.

(I work on PGlite)

Ooh! The 'docker image on WASM' thing sounds promising for a wide range of problems. Recently I wanted to run a FFMPEG/SoX pipeline on the client - too many dependencies to easily recompile with Emscripten; could your approach help there?

Yeah, that should be possible! Though, for audio processing, the performance will probably be terrible (because it's all emulated).

There's already ffmpeg wasm. I've used it in projects. Works great.

https://github.com/ffmpegwasm/ffmpeg.wasm

Thanks, looks good... doesn't look like there's a SoX though.

I googled - haven’t tried it https://github.com/rameshvarun/sox-emscripten

There’s also https://github.com/IPS-LMU/wasmsox

Lots of cool software that people hack together for fun. Just site:github and search!

If it could support the pgvector extension it would be a super fast vector database with all the power of Pg - the relational aspect brings the ability to add and query using rich domain specific metadata usually contained in relational databases.

I spent last week trying to do that with some of the other pg embeded libs.

And then lancedb released their embedded client for rust, so I went towards that. But it's still lacking FTS. So I fell back to sqlite. have some notes here https://shelbyjenkins.github.io/blog/retrieval-is-all-you-ne...

As a data point, the online demo seems broken for queries it doesn't like:

    select foo();
    Error.captureStackTrace is not a function

That's when using Firefox 124.0.2 on Linux.

Yes, that's a nonstandard function provided by v8, so it wouldn't work on Firefox. [1]

This can be worked around by just constructing an Error and taking it's stack property, captureStackTrace is just a convenience function, so hopefully they can fix that.

[1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Thanks for suggesting that. Looks like they've implemented that here:

https://github.com/stackframe-projects/pgmock/commit/f80d9fa...

It seems to have beendeployed out to the demo already as it's working now. :)

My bad, gotta fix that one!

That's great. But doesn't the whole concept of E2E test mean that you use real environment without mocking the components?

Explicitly mentioned in the comment as a drawback. In practice E2E means "E2E as much as humanly possible", and I'm glad to see any work that can help.

It is, however, humanly possible, to run a real Postgres instance in a Docker container. You may use for example Testcontainers. So, I would not resort to an inferior Postgres "mock" for E2E testing.

For unit testing (also mentioned in the tagline of the project on GitHub) I could see you wanting something snappier than a real Postgres in Docker, so then... maybe? Purists of some following will tell you to use a mocking framework instead, but I think that something closer to the real thing would be better in all cases. This might be it, just be careful not to lull yourself into a false sense of security about how close to the real thing this is (isn't).

It does seem to be running a real version of Postgres though, unless I'm misreading?

On first reading of the main GitHub page, I didn't take it that way. On second reading, I guess it could go either way. But there's talk about having reimplemented the network stack, so maybe that implies they just repurposed the rest of Postgres. I just don't know enough, happy to take your word.

It depends on the humans and what they're up to. Fast feedback is a very valuable attribute of a test suite, so reasonable people may well be willing to go down this road.

The reality is that testing is as nuanced as anything else in software. Tradeoffs are different in different situations, and reasonable people can disagree about appropriate tradeoffs in a given situation.

> pgmock runs an x86 emulator with the original Postgres inside

Why can't Postgres compile to WASM instead of x86?

Postgres uses multiple processes, shared memory etc. The single user thing that OP referenced is single user because of that...

Reply on news.ycombinator.com