Comment by UserRights | Hacker News Reader

Comment by UserRights | Hacker News Reader

I found the most important idea regarding backups was to stop searching for the one perfect tool that does it all. The most productive step was to accept that for every [group of] machine[s] and every application / scenario there are different best solutions.

Unfortunately most backup software designers seem to think in an extremely one-dimensional way about how backups should work and I do not know about one tool that offers all the flexibility you need for real life backups.

There is really room for invention here.

IMHO BackupNinja shows the right direction: it is a meta-tool that helps you to manage several different backup strategies. This is the way to go, but it should be generalized and have an API and several GUI options (web, qt, rest).

Also some more brain could be put into the application, like a nice wizard will ask you to define how you would like to have your backup and select the right tools for you:

  [ ] source
  [ ] it is a database
  [ ] which one: ________________
  [ ] very big files (oh, I already know about this)
  [ ] Mickysoft client
  [ ] Outlook
  [ ] other crapsoft that needs special handling
  [ ] look up the plugin repo for the best way to handle this
  [ ] Linux / *BSD
  [ ] destination
  [ ] encrypt backups
  [ ] versioning
  [ ] frequency
  [ ] make backup files browsable by filesystem tools
  [ ] also browsable for users (readonly)
  [ ] make backups available via samba
  [ ] make backups available via nfs
  [ ] make backups available via web gui
  [ ] where to browse: ________________
  [ ] import csv with usernames
  [ ] auto-generate login link for users (no future support hassle)
  [ ] decide which is the best (set of) tool(s)
  [ ] and just do it and let me do the real work

Certainly there is some more to it, but maybe you get the idea.

If you are bored and do not know which should be your next project, please release the world from all these backup pains (and wasted hours and weeks) and build it, thanks! Good Luck!

Sounds like your vision for want you want out of a backup system seems to be almost identical to mine - I would add one thing - it would be lovely if this process could store it's configuration for each backup item / set in YAML.

I think YAML based configuration would make it very easy to work with once created and offer ends less possibilities of integration and generation from/to other applications.

If someone made this - I would donate them money.

Bonus points for a nice visualisation of backups and when they expire etc...

So we implemented a variant of the Bekeley Lab Checkpoint and Restore [0] on terminal.com. You can snapshot RAM state at any given moment and commit it to disk without interrupting operations. You can use it to, for example, start a Spark cluster [1] with a dataset already in memory. I've tested it with a lot of software, so I'd say it works irrespective of the application, but you're welcome to test.

When you use our snapshotting, RAM-state, cpu cache and disk-state are all captured and can be resumed in seconds later. This all happens without a hypervisor.

This sort of obfuscates the need for doing any configuration storage (you snapshot systems at an initial state and then you can bring up new machines at that initial state without config files. If you need to pass an argument to a machine on boot you can do it programmatically by passing the shell commands [2]).

[0] http://crd.lbl.gov/departments/computer-science/CLaSS/resear... [1] https://www.terminal.com/snapshot/c81e6215eba5799335a45b6936... [2] https://blog.terminal.com/tutorial-terminal-startup-scripts-...

So a metatool that is flexible in terms of backup ingest? Almost like backup plugins?

Check out Deltaic, designed precisely for this:

https://github.com/cmusatyalab/deltaic

This tool certainly looks interesting and does support several sources for the backup.

I was describing a meta tool that would act like an experienced backup admin and setup the whole process / environment. E.g. this tool would configure network shares or a webserver with authentication to access the data, that deltaic restored.

However, such a tool does not exist, so I will study deltaic when there is some more time to waste with backup research. :)

I see what you're saying...kind of like an AI or an "automation" tool/framework for backup.

I was focusing on the lower-level building blocks.

Reply on news.ycombinator.com