A long, long time ago I coded an MP3 implementation for a TI Fixed Point 16bit DSP@100MHz (family 5XXX IIRC).

I initially wrote it in floating point C (compiler translated float to fixed point) and it was damn slow. I then rewrote everything to use fixed point arithmetic only and dropped down to about half a second of decoding for a second of data. That wasn't still good enough because the DSP had to take care some other stuff like ide drives, dma, fat, dac, usb , display drivers etc, so I rewrote all critical parts in assembly taking advantage of the chip's build in dsp functionality and dropped to 100ms of decoding for 1s of data.

So it might be helpful to start from a pure python implementation then move your way down, you'll still learn a lot.