Majority of products like that i.e. SDK/libraries, are using fixed price. I'm sure some can get away with royalty but that's rare.
Somewhat related: as someone who likes the idea of Sciter and tried to use it, I had a hard time figuring the basics out and I think the best way to improve your business would be:
1. Improve documentation, especially add step-by-step tutorial for beginners. The first article should not be "Engine architecture" but "Step-by-step guide to build your first application with Sciter".
2. The SDK (I tried both Windows and Mac) is super confusing. Multiple executables, not clear what does what.
You should have separate versions for Windows / Mac / Linux because bundling all platform executables into one increases confusion.
There are plenty of samples but no clear way of how to even launch them. I spent way too much time figuring this out and had to write an alias for the sciter executable to make this bearable (i.e. alias sciter='/Users/kjk/Downloads/sciter-sdk/bin.osx/scapp')
There's a mish-mash of files that can be run directly and those that are imported by the main code.
Ideally, if the sciter launcher in SDK is launched without arguments, it should ask to load a sample from a file (pre-positioned in the samples folder) or allow drag&drop sample files on a window or list samples and allow launching them with double-clicking.
See how rebol-view does it http://www.rebol.com/download-view.html
3. Improve stability. The app crashed on some samples. Not a good first look.
Sciter is a great technology, but the experience of evaluating it is bad. I would imagine at this time improving that would be the best way to make it more popular.