A year or two ago the TeXmacs package worked pretty well.

There is also an actual Octave kernel for Jupyter, similar to the Julia one: https://github.com/Calysto/octave_kernel