ThomasWaldmann

Guess now not only RAM bandwidth is important, but also flash storage bw.

Maybe Apple could attach more flash chips in parallel, increasing the bus width and thus the overall bandwidth?

Yeah, it's a pity that TB2 cables and a TB3-TB2 adapter are still that expensive.

Considering the age of that stuff, one would usually expect that to be very cheap.

Hardware guide:

https://github.com/swellweb/targetBridge/blob/main/docs/Hard...

Use an (old) iMac as an external display for an Apple Silicon Mac — free, no dongle, via Thunderbolt Bridge.

Apple dropped "Target Display Mode" in 2014 with the 5K iMac — and it never came back.

TargetBridge brings it back via software, streaming your screen to the iMac at up to 5K over a direct Thunderbolt cable.

Since v2.0.0, you can now even connect multiple iMacs to get multiple gorgeous 5K displays!

FOSS, under the MIT license.

Disclaimer: I am a project contributor, but not the original author of this project.

Bugs?

I don't know about any show-stoppers in borg 1.x.

Design limitations?

Yes, there are some, that's why borg2 will be quite different. But these are no easy or small changes.

Also, borg2 will be a breaking release (offering borg transfer to copy existing archives from borg 1.x repos). It takes long because we try to put all breaking changes into borg2, so you won't have to transfer again too soon after borg2 release.

Yes, it also disallows overwriting.

borgbackup developer here:

TL;DR: don't panic, all is good. :-)

Longer version:

- borg 1.x style “append-only” was removed, because it heavily depended on how the 1.x storage worked (it was a transactional log, always only appending PUT/DEL/COMMIT entries to segment files - except when compacting segments [then it also deleted segment files after appending their non-deleted entries to new segments])

- borg 2 storage (based on borgstore) does not work like that anymore (for good reasons), there is no “appending”. thus “—append-only” would be a misnomer.

- master branch (future borg 2 beta) has “borg serve —permissions=…” (and BORG_PERMISSIONS env var) so one can restrict permissions: “all”, “no-delete”, “write-only”, “read-only” offer more functionality than “append only” ever had. “no-delete” disallows data deleting as well as data overwriting.

- restricting permissions in a store on a server requires server/store side enforced permission control. “borg serve” implements that (using the borgstore posixfs backend), but it could be also implemented by configuring a different kind of store accordingly (like some cloud storage). it’s hard to test that with all sorts of cloud storage providers though, so implementing it in the much easier to automatically test posixfs was also a motivation to add the permissions code.

Links:

- docs: https://github.com/borgbackup/borg/pull/8906/files

- code: https://github.com/borgbackup/borg/pull/8893/files

- code: https://github.com/borgbackup/borg/pull/8844/files

- code: https://github.com/borgbackup/borg/pull/8837/files

Please upvote, so people don't get confused.

at first it was implemented to easily test permission restricted storages (can't easily test on all sorts of cloud storages).

it was implemented for "file:" (which is also used for "ssh://" repos) and there are automated tests for how borg behaves on such restricted permissions repos.

after the last beta I also added cli flags to "borg serve", so it now also can be used via .ssh/authorized_keys more easily.

so it can now also be used for practical applications, not just for testing.

not for production yet though, borg2 is still in beta.

help with testing is very welcome though!

The usual answer: "when it is ready".

For low-latency storage (like file: and maybe ssh:) it already works quite nicely, but there might be a lot to do still for high-latency storage (like cloud stuff).

no-delete disallows any kind of deleting information, that includes object deletion and object overwriting.

borg supports the "obfuscate" pseudo compressor since 1.2.0 (since Feb 2022), that adds random extra length to the chunks, using one of 2 different algorithms to determine the extra length.

borg discussion + wiki page:

https://github.com/borgbackup/borg/discussions/8694

BTW, besides "cloud support" via rclone, there are also some other big changes in borg2 beta.

This is the summary of the major changes (more details are in the change log):

https://www.borgbackup.org/releases/borg-2.0.html

Basic stuff might work in a "posix-like" environment, like when using cygwin or WSL.

The main problem is that there currently aren't active windows developers within the borgbackup project who continuously test and improve the windows specific code parts.

Since recently, we at least have some working CI on windows again, at least that was fixed.

Comparisons might be interesting, but one needs to be aware that they would be a bit apples to oranges:

- unreleased code that is still in heavy development (borg2, especially the new repository code inside borg2).

- released code (restic) that has practically proven "cloud support" since quite a while.

borg2 is using rclone for the cloud backend, so that part is at least quite proven, but the layers above that in borg2 are all quite fresh and not much optimized / debugged yet.

Thanks a lot for writing the rclone backend!

Some people were installing / using borg on android, but guess that isn't suitable for end users, rather for the nerds.

Maybe try SeedVault?

Please note that this is rather recent "bleeding edge" code from master branch.

It is available as 2.0.0beta11, but not suitable for production yet.

"Beta" also means that there won't be repository migration code from beta N to N+1.

Python source code? Yes, it is on https://github.com/borgbackup/borg (if that was the question).

No cloud? Indeed, borg does not integrate cloud functionality.

You'll either need a directory or a server with ssh and borg to store your backups into. Some people use rclone to clone their borg repos to the cloud.

Besides vorta (which offers a lot of options) there is also pika backup (which might be simpler by offering less options).

Not much. rsnapshot is based on rsync and hardlinks. So, you'll get some whole-file deduplication for unmodified files. No compression, no encryption/authentication, no way to verify if a backup is still ok.

Also, if you rename a file or folder, that hardlink-based dedup will not work, because it is based on files having the same path. Also, if you change just 1 bit in a huge file, it will add the whole new file because the dedup only works on whole files.

borg does variable size chunk based deduplication. if a big file changed just a bit, your backup repo will also just grow a bit when backing up that file.

also you get compression, encryption, authentication and the ability to check backups if they are ok still.

also, borg won't create a gazillion of files/hardlinks in the destination filesystem, but way fewer files (each about 500MiB big by default).

borg also archives ACLs, xattrs, flags.

You could have just used the self-contained borg binary (download from github releases page).

It contains everything needed (python, python libs, other libs) except glibc and related libs which must match the OS / kernel and therefore are intentionally not included.

deduplicating backup tools use a symmetric approach: all archives needing some chunk (piece of data) reference it via its chunkid (the keyed hash over the chunk plaintext content).

so you can delete ANY backup archive without influencing any other backup archive. a chunk will be only deleted if nothing is referencing it any more.

also, each backup is logically a FULL backup (it has ALL files, references ALL content data). it is just made in a clever way, avoiding to re-transfer data that already is present in the backup repository, thus it FEELS like incremental (considering speed, amount of CPU and I/O used).

OTOH, full/incremental style backup tools build a chain of incremental archives depending on the previous incremental and the full backup, which gets more fragile the longer the chain gets.

because of that and also because you might want to delete older backups at some time, you are forced to create new full backups regularly (causing lots of CPU and I/O load).

I see quite some full backups in your near future. ;-)

And that is one of the main reasons why chunk-deduplicating backup tools (like borg, restic, ...) are better than full/incremental style ones.

I agree that a multithreaded borg could utilise resources better, esp. IF you have a lot of changed / new data to back up.

But OTOH, for many users this is primarily the case for the first backup, but not for their daily / hourly backups when most files are unchanged - for that, I guess speed is I/O bound and all your CPU cores won't help you with that.

So for N-1 backup runs, it just works good enough already for many users. And for that 1 initial backup, some patience helps. :-)

Implementing MT is planned since long, but due to the above, other stuff had higher priority.

There's this ticket: https://github.com/borgbackup/borg/issues/225

I don't see anything of that in the near future of borg, esp. as long as the fundamental concerns there have not been addressed.

It is still early in the development process, but it might be just for new repos.

The "long and messy" python error messages are python tracebacks and often intentionally displayed by borg to ease locating and fixing bugs.

Of course we could also display shorter error messages (and we do that at some places, if the cause of the exception is well known / expected), but be glad to have the long form and not just "something went wrong" (which is very pretty, but completely useless). :-)

About "Borg’s crypto has known limitations":

We've just merged new crypto code into master, based on AEAD ciphers (AES-OCB and chacha20-poly1305) and session keys - so the potential nonce management issues are soon a thing of the past.

There's current work adding argon2id as default for the KDF (was: pbkdf2), likely soon to be merged.

Also checking blake3 for the ID hash (MAC) right now (but platform / build compatibility has yet to be seen).

Yeah, that's an issue which can be avoided by using 1 repo per client.

There are some ideas on the issue tracker for fixing this long term (like random nonces, session keys, ...), but that stuff will have to wait until after borg 1.2 (which soon goes into release candidate phase).