Heh very similar experience with myriad-x there. Going off the beaten path is a pain, especially since the low-level is now so hidden...
I wish you good luck there, but you're up a huge task. You have all my congrats for going open source, and I think now it's mostly the only way forward. FINN is OSS and I'm very happy to have an OSS alternative. If only old Altera would go full OSS on new AI+FPGA stuff maybe we'd see great cross pollination.
Anyway, if Intel FPGA people aren't watching this, I can assure you they'll be looking soon.