For interest, this seems to be an active issue for the HTML <canvas> element in browsers. There's a proposal[1] to extend the canvas data type to include both "unorm8" (the beloved default) and a new "float16" (normalised?) format - which should meet your desire?
Typically, the proposal seems to have shipped already in Chrome/Edge browsers. Documentation around what the new functionality is for and how to make best use of it is (of course!) sparse - MDN barely mentions it. As a canvas library maintainer I find this upsetting (eg: Ignore it and it might Go Away).
(I think for now my unhelpful response is: manipulate your RGB images as much as you like; just do it in the OKLAB color space.)
[1] - https://github.com/w3c/ColorWeb-CG/blob/main/hdr_html_canvas...