burmecia

Thanks, the "Top Comments" is sourced from the "kids" field of the "items" endpoint. As the API docs (https://github.com/HackerNews/API?tab=readme-ov-file#items) says, it is "in ranked display order" so I just used it to sort the top comments.

That's a good idea, will put it on to-do list.

Thanks for the ideas, heat map looks great! Will also consider add custom queries, it can be quite useful.

yes, the keywords search is using full-text searching and it is not efficient so I hardcoded them. Will consider switch to use embedding, that will be more efficient and accurate, also can support custom keywords search. Thanks for your advice.

AFAIK, ChatGPT's training data is until 2021, can it answer questions based on the latest Supabase documents?

Next.js is more about frontend, but Supabase is more on backend. I cannot see any conflicts of the two, they can work with each other flawlessly.

You can create a custom function doing all the delta logic and use pg_cron to schedule it to periodically materialize data using FDW.

| At the end of the day, are these just FDWs that run on top of any Postgres instance?

Yes, the FDWs developed by Wrappers can be used on any Postgres db.

| Do I have to run Supabase to take advantage of the FDWs created through this framework?

No, you don't need to. Those are just normal FDWs can be used on any Postgres db.

| Is there a generic JDBC/SQL/etc FDW for any SQL based database like Snowflake, Oracle, etc?

The Snowflake FDW isn't developed yet, the example is just for concept demo. It has ClickHouse and BigQuery FDWs although they are not JDBC based. Generic JDBC FDW will need JVM embedded so it is not in plan at this moment.

Disclaimer: I am Supabase developer.

| How well do foreign data wrappers work with Postgres's query planner?

You can provide table size estimation to Postgres's query planner in wrappers using `get_rel_size()` function, here is doc:

https://docs.rs/supabase-wrappers/latest/supabase_wrappers/i...

| Also, is the Firebase connector for Realtime Database or Firestore?

It is for Firestore, Realtime Database is not supported yet.

Another interesting term not mentioned is 'qual', which basically has same meaning as 'restrictions'.

Our price model based on database usage which includes db egress, so we use it to monitor db egress traffic just for now. We are all db fans and we like put things into db, so all the following processes, like searching, tracking, analysis and etc., are a lot easier.

I like old solid tech like Postgres, libpcap and etc. Putting them together with new tech like Rust and pgx is also fascinating.

That's a nice approach, thanks for sharing with us.

Yes, you need root to grant capture permission to Postgres, but Postgres doesn't need to run as root. The main purpose is to easily ingest network traffic data into Postgres for further process. Yes we can use tcpdump to do the same thing, but that needs another tool to load data into Postgres.

Hey HN,

I have spent some time to search for a tool that can ingest realtime network traffic data to Postgres but have no luck, so I developed this extension and used it internally in our team. Thanks Rust, pgx and libpcap, the development journey is easy and enjoyable.

Would like to hear more feedbacks. Any contributions, feature requests, bug report or ideas are welcomed.

Thanks.

I am building an encrypted file system that runs inside application (https://github.com/zboxfs/zbox). It focus on security, privacy and reliability. The interesting part is it can utilize different storage, such as memory, OS file system, RDBMS and object store. I learned a lot and enjoyed working on this project.

Totally agree. To make file system crash-proof you really need an ACID compatible underlying storage layer or implement it inside the file system itself. In fact, file system and database are quite similar in case of atomicity and consistency. If an embedded ACID component is too heavy, utilizing a reliable database as storage, for example sqlite, might be a good idea. That’s exactly what I did in ZboxFS (https://github.com/zboxfs/zbox) which can use sqlite as a underlying storage to achieve ACID compatible behaviours.

I am using Rust building a privacy focused in-app file system ZboxFS(https://github.com/zboxfs/zbox). The fun part is not only joyfulness Rust bring to me, but also learned a lot about cryptography, file system, transaction control and many more interesting areas.

Database and filesystem share a lot similarity, but still have their own characteristics. To use database as a back-end for filesystem has some benefits, such as reliable persistence, ACID transaction and etc. It reduces complexity to achieve durable storage which filesystem should deal with.

Many filesystems are able to use DB as back-end, I also made a filesystem ZboxFS (https://github.com/zboxfs/zbox) which can also use sqlite as underlying storage.

I don't think the encrypt-then-dedup is a safe way to protect data privacy. In this case, identical blocks need to produce same cipher text, this will actually leak your data pattern even though it is encrypted. A better way I think is using randomly-seeded derived keys to encrypt each block, thus the identical blocks' cipher text will always be different.

I think any applications need store confidential files on client or remote can adopt it. Web and mobile app might be the best to use it at this moment.

As there is transaction control, the conflict handling should be straightforward. That is, the thread got write lock can write the file exclusively and each write is a transaction and commit will form a new permanent version.

GDPR might be a good reason, but I think it can be more general. Any apps need store confidential data can use this, no matter the data is on local or cloud.

I think the best approach is never save unencrypted data on cloud. Always encrypted on client first. But by that way we lost dedup capability, so we have to do everything, such as encryption, dedup and compression on client side. I made an in-app file system dedicated for that purpose. https://github.com/zboxfs/zbox

Another approach I can think of is using rolling hash (https://en.wikipedia.org/wiki/Rolling_hash) to split file to different size of chunks, those chunks can be saved on different sectors which don’t need to be continuous. The file keeps a list of indexes for all those chunks, just like inode did for blocks. When insert some bytes in the beginning of file, thanks for the rolling hash, most likely only the first several chunks need to be re-chunk and re-hash, thus the change is localised and can be cheaply done, all the rest chunks still untouched. When this combined with append-only storage, things are even easier because it only needs to deal with bytes around that beginning, and the update chunk index list in file.

This approach is already implemented in ZboxFS (https://github.com/zboxfs/zbox) to do content-based deduplication.

Totally agree. Log based file system makes things much easier, such as atomic write, data consistency and failure recovery. Along with COW semantics, transaction isolation also becomes easier. By its append-only writing nature, file versioning is trivial as well. Garbage collection is performance downside indeed, but it can be mitigated by COW switching and delayed bulk collection IMHO. Overall, the cost is worthwhile and that is what I am currently doing in ZboxFS: https://github.com/zboxfs/zbox.

There is an userspace implementations of filesystems ZboxFS: https://github.com/zboxfs/zbox.

No, I didn't. And I don't think it is necessary as Zbox is much more like an 'application-level' fs, obviously can't match the system level fs.

Nice work!. I also made a similar file system Zbox: https://github.com/zboxfs/zbox. The difference is Zbox is a in-app file system focused on privacy, so FUSE is not supported intentionally. Although it already supports key-value store now, I am currently trying to extend its capability to cloud storage.

IMO, "cd", "grep", "cp" and etc. are provided by shell and os, not by file system. They are just human interface to help you interact with the underlying file system. Can you call "cd" on an ext4 file system directly? No. The shell will invoke "cd" or similar process and that process will do the system call to kernel, and then kernel will call the ext4 kernel module through VFS.

https://www.kernel.org/doc/Documentation/filesystems/vfs.txt

For Zbox, it is just like a kernel module but runs inside application memory space. To interact with it, you have to use a set of 'standard' API, in kernel that is VFS, in Zbox that is Rust's fs API (not exact same, but similar).

Not yet, but could be possible if you can compile it into WebAssembly.