DeusCodex

There were some pretty big changes added with v1.0.3 of blockframe. Removed mmap conditions and thresholds to enable default mmap for file read in tier 2 commits. Previously mmap would be selected if a file was greater than 10mb, use_mmap would be checked within the segment loop constantly, even though use_mmap was a constant. Presence of conditional mmap meant doing variable sized buffer fills which would eventually lead to short reads and corruption of segments due to filling EOF segment with dummy data as the buffer would be statically sized for reusability.

Added padding to generate_parity_segmented. The Reed-Solomon erasure coding is ran on SIMD instructions, meaning the data which is being processed on the specific lanes must be 64 bytes. Prior if 65 bytes were passed, the 1 left over bit wouldn't be filled into the SIMD vector which created undefined behaviour. Added next multiple of 64 padding for exception chunks.

Changed commit boundaries for tier 1 from 10mb to 25mb. Changed sha256 function name to blake3_hash_bytes.

Feel free to check it out, clone or download the release for the platform of your choice. And if you appreciated what I built, please consider giving it a star and sharing, that would mean a lot. Thank you, have a cracking day :)

https://github.com/crushr3sist/blockframe-rs/releases/tag/v1...

There were some pretty big changes added with v1.0.3 of blockframe.

Removed mmap conditions and thresholds to enable default mmap for file read in tier 2 commits. Previously mmap would be selected if a file was greater than 10mb, use_mmap would be checked within the segment loop constantly, even though use_mmap was a constant. Presence of conditional mmap meant doing variable sized buffer fills which would eventually lead to short reads and corruption of segments due to filling EOF segment with dummy data as the buffer would be statically sized for reusability.

Added padding to generate_parity_segmented. The Reed-Solomon erasure coding is ran on SIMD instructions, meaning the data which is being processed on the specific lanes must be 64 bytes. Prior if 65 bytes were passed, the 1 left over bit wouldn't be filled into the SIMD vector which created undefined behaviour. Added next multiple of 64 padding for exception chunks.

Changed commit boundaries for tier 1 from 10mb to 25mb. Changed sha256 function name to blake3_hash_bytes.

You know what, that might be a very good idea, if encoding speeds on my really crappy hard-drive weren't under a second. But no I do see where you're coming from, and for larger files that concept does apply, but blockframe wants to protect your files from circumstances that you can't control, and the concept is still, have the file as its own entity, and to keep it planted, we use RS to make sure nothing funky happens. Small files do tend to cause random access speeds, that's why they're not split up.

I'm open to suggestions for better erasure coding storage ratio :)

Hi HN, author here. I built BlockFrame because I wanted the durability of distributed object storage (erasure coding, bit-rot protection) but for local, single-node archives.

It is a storage engine that shards files into Reed-Solomon blocks (RS(30,3) or RS(1,3)) to guarantee mathematical recovery from disk corruption. It then exposes this engine via a FUSE/WinFSP interface so you can access the data using standard tools (read, seek) without needing custom APIs.

Key features:

Engine Layer: Handles the heavy lifting of parity calculation and Merkle tree verification.

Access Layer: Virtual filesystem driver allows zero-copy access and random seeking on multi-gigabyte datasets.

Self-Healing: The engine transparently reconstructs corrupted sectors during reads.

I’m graduating in May and aiming for systems roles in the UK. I'd love feedback on the architecture and any feedback is highly appreciated, Thank you.

BlockFrame is a file chunking library designed to work as a data-store solution alongside web apps and software. It provides a simple and flexible way to store large files by breaking them into smaller chunks that can be easily managed and retrieved. I'm very interested in file-storage methods and optimising their read and write speeds. Blockframe enables this with security and efficiency.

Source: https://github.com/Wizock/BlockFrame

PyPI: https://pypi.org/project/BlockFrame/

I would really appreciate it if you would download and test it out, take it for a fun run and comment on any issues you had with usage. Please make sure to read the example source code for use types to avoid making errors, however, I hope it's easy to grasp. Oh and the reconstructed folder is required to be created :3 (still working on that, I don't know if people want their own custom path for the file fetch dir)

A star and share would be so much appreciated, I'm very passionate about this library and I hope you guys have some bugs for me to take care of :))

thanks

This is such a amazing little book. I love it, AI is hard for me to understand but this book makes it very easy to grasp some of the concepts.

Thanks for sharing

I aint reading allalat