I wanted to add something about the xilinx dpu, and you brushed on the subject but I was quite unhappy with the softip thing. It embeds all instructions for all kinds of networks so, taking a lot of gates for unused features, it's not much customizable, and perf for anything else than vanilla conv2d stuff quickly gets down. Buying an Alveo board to get such low inference perf was a gutpunch.

FINN seems far better there. At least you get millions inference/sec on simple quantized CNN1Ds.

The xrt api is simple and relatively ok, too. Stream data, execute inference, fetch results, mostly sync, so you have to wrap a lot of threading there, but the basics are there.