hikarudo

Awesome, thanks for the suggestion!

PyTorch's serialized models are just Python pickle files. So to load those you need the original classes that were used to build the model. By converting to ONNX you get rid of those dependencies.

I've recently switched from bash to zsh. Checkout the dirhistory plugin, which allows you to use Alt-left to go to change to the previous directory, Alt-up to change to the parent directory, etc.

https://github.com/ohmyzsh/ohmyzsh/blob/master/plugins/dirhi...

> I've been thinking about solving the backup part by letting one of the devices perform btrfs snapshots on its storage.

I do zfs snapshots on my storage server (home desktop), which is in Syncthing's "receive only" mode. I use it to sync my camera roll and Whatsapp media files from my Android phone. I use pyznap for snapshot management.

> Setting up a cron job / timer unit that commits the synced folder to a git repo is also an alternative

Another option would be to backup the synced folder using a backup tool like borg, restic, duplicacy etc. They have options to prune the snapshots.

Feynman talks about exactly this in his book "Surely you're joking, Mr. Feynman":

"Then I got an idea. I challenged them: "I bet there isn't a single theorem that you can tell me - what the assumptions are and what the theorem is in terms I can understand - where I can't tell you right away whether it's true or false."

It often went like this: They would explain to me, "You've got an orange, OK? Now you cut the orange into a finite number of pieces, put it back together, and it's as big as the sun. True or false?"

"No holes."

"Impossible!

"Ha! Everybody gather around! It's So-and-so's theorem of immeasurable measure!"

Just when they think they've got me, I remind them, "But you said an orange! You can't cut the orange peel any thinner than the atoms."

"But we have the condition of continuity: We can keep on cutting!"

"No, you said an orange, so I assumed that you meant a real orange."

So I always won. If I guessed it right, great. If I guessed it wrong, there was always something I could find in their simplification that they left out.

What's the difference between an algorithmic camera and plain image processing / computer vision?

That would be DCT - Discrete Cosine Transform.

Good point. A generic type is a great example of a type constructor.

You are correct, my mistake.

A type constructor is something that takes a type and gives you a new type.

For example, in C, if you have any type Foo, the pointer operator * will give you a new type Foo*.

Another example: in Haskell, Maybe is a type constructor, since it takes a type parameter a (Maybe a). So 'Maybe' is a type constructor, but 'Maybe Int' is a type.

So a type constructor is like a function, but it operates on types, not on values.

Couldn't you have simply restored the original files via the Dropbox web UI?

> Software is completely different, it is hardly ever a repeatable process to create software. Making the same software over and over is nonsensical, but making a billion identical contact lenses is exactly what you want.

I've heard it said like this: Creating a software is like developing a recipe for a cake. Baking thousands of cakes (a "manufacturing process") is what you do once you have the recipe. The two activities are very different.

See also Peter Naur's "Programming as Theory Building". Essentially,the original authors of a system have a theory (what nowadays we would call a model) in their minds of the system, of its architecture. Something that is difficult to convey by code alone. So when a new batch of programmers start working on that codebase, they don't have any knowledge of this theory, so their fixes and new features are of much worse quality than what the original programmers would have proposed.

In the same vein, I recommend the book "Secrets to Winning at Office Politics" [1]. The title is a bit click-baity, but the contents are excellent.

[1] https://www.goodreads.com/book/show/260493.Secrets_to_Winnin....

The Coral TPU is mostly made for inference only, not training, with a few exceptions.

Off topic: shows with audio descriptions are also great for learning foreign languages. You basically get a voice narrating what is happening on screen. "Oh, so that's how you say 'crouching' in Japanese!"

> isn't using lua a problem because of it's memory management causing GC stalls?

Perhaps even more important is the increased memory use due to Lua. Some devices have very little memory to begin with.

He probably just understood "gnuplot", and was used to the confusion.

Embarrassing story time.

I once introduced Richard Stallman to a fellow grad student in Japan like this (in Japanese): "He is the creator of GNU. You know, as in gnuplot."

Stallman understood what I said and immediately corrected me: "gnuplot is not a part of GNU".

Good points. Also, code can change frequently, so the "visual memory" reinforced by pagination becomes less useful, maybe even a hindrance.

I wonder if source code could be read and understood more effectively if it were paginated.

My workflow is the same. The results of the exploration phase are like a prototype - plan to throw it away, keep the learning, then build the final, elegant version.

Thanks, I've been wondering about the "proper" way of doing this in ZFS and this method hadn't come up in my searches.

Even better:

https://www.hetzner.com/dedicated-rootserver/matrix-sx

I've been looking for a solution to the same problem.

So far we've been using a single server for storage, and developers rsync whatever they need locally. A few million images. Training is usually done locally, but when we do use cloud training, we upload just the dataset we need to S3 and use EC2.

We're a small team, and currently considering moving to a cloud-first infrastructure. The idea is to store each image in S3, and all metadata (annotations etc) in Postgres or something like that, maybe using Postgres's JSON/JSONB feature.

I'd appreciate any thoughts and pointers on handling datasets with a few million images.

I like how this story is told in Buddhist contexts:

http://www.1netcentral.com/articles/the-snake-rope-essential...

Your story reminds me of the Indian allegory of the man who mistakes a rope for a snake.

What an amazing story! Thanks for sharing.

There is also the startup founder path. That's what I chose, and I'm glad I did. I was tired of meaningless research and meaningless work. I wanted to build something real.

Any pointers on hot code reloading for C++?