Absolutely possible.

What I'm building into PhotoStructure is typically called "transfer learning."

https://en.wikipedia.org/wiki/Transfer_learning

PhotoStructure is entirely self-hosted, including model training and application: the public domain base models (trained on huge datasets) are fetched and cached locally.

By design, none of your data (or even metadata) leaves your server.

(I expect to ship this in an upcoming beta next month.)