There is a well-known bug[1] with non-standard resolutions. Try changing the resolution and see if it makes any difference. If not, please file a new bug :). Otherwise you can see if you can help fixing the issue. Unfortunately, it requires some re-engineering of core graphics processing and they are working on it, but it'll likely take a few more months to fix completely.

[1] I think this was the bug I thought about: https://bugzilla.mozilla.org/show_bug.cgi?id=1404042, see also https://bugzilla.mozilla.org/show_bug.cgi?id=1422090