I just went back and actually found the corresponding issue on my repository. Turns out it was firefox having issues with the setTargetAtTime function. I replaced it with linearRampToValueAtTime to get rid of the clicking sounds. I think for that to work I had to shift each note a little forward and then do a very quick ramp in the beginning. It does alter the sound a little though.

Also: Thank you! The idea is definitely derived from Synthesia but that isn't free anymore and I wanted to make it work in a browser :)