"it's fiddly front end stuff I don't like all that much as a domain"
What (briefly) is your experience or skillset so I can better target my recommendations?
"A lot of what you suggest for directly solving it strike me as hacks not to my taste, but I'll look harder at that sort of thing in the future."
The problem is that the Web, Windows, and UNIX stacks are all hacks built on hacks with tons of compromises and many bad design choices. Anything that preserves their functionality without a rewrite is going to be hackish. So, we then decide where to put hacks among libraries, source-to-source translators, libraries, OS's, hardware, etc. I tried to name off more sensible one's to avoid readers having headaches I had going through 10,000 papers of this shit. ;)
"If you're going to modularize it, the interfaces are going to be complex and error prone"
Interface errors account for vast majority. Margaret Hamilton, Dijsktra, Robert Schell, and Paul Karger all independently discovered that w/ solutions presented. The consensus is you need to document the pre-conditions, post-conditions, and properties that always hold. Type-checking, Design-by-Contract, and similar techs help. Generic functions and good design helps keep complexity down. Functional languages with strong typing are best at making this safe and concise at same time. So, it doesn't have to be bad unless your tools suck.
"much more fragile cryptosystems/cryptographic protocols "
Sort of. Those do much more than just decompose: theory, design, implementation, distributed of all that... a tall order. Understanding the positive effect of decomposition and layering will be obvious if you look at this on p8 "Layered Design:"
http://www.cse.psu.edu/~trj1/cse543-f06/papers/vax_vmm.pdf
Now, try to imagine a description of how VMware or KVM do the same thing. It would probably be much more complex with looping around and such. The decomposition makes it easier to understand w/ sensible interfaces. That's how it's supposed to be done anyway.
"trying to design an public key based authentication server; no success then"
As I said, crypto protocols are extra hard to do right. It's the kind of thing that should get intense tool support. The state-of-the-art in high assurance is NSA and Rockwell-Collins tool that goes from a flowchart & specs in (Simula?) to generated code or hardware with verification checks for a processor whose security is verified. An example for the rest of us and ideal for your case is combining CRYPTOL for algorithm, state machines for overall code, and SPARK for implementation language. The protocol logic would be implemented with this kind of tool:
http://www.dsi.unive.it/~modesti/anbx/
I call out plenty of people for their methods. Yet, it was understandably hard to pull off public key servers a few years after they were invented on 1970's computers. Even the people inventing INFOSEC sucked at security back then. ;)
"It's got to be fast; for now, ever faster as web pages get more and more complicated."
It does. Hence our hacks to work with normal, highly-optimized software. However, some old stuff was still faster. JavaScript is utter garbage. One alternative (Juice) used Oberon language for applets with ZIP'd abstract syntax trees. Let you type-check it, interface-check it, and then compile it super-fast to fast binary. Client-server apps could use efficient stacks, implementation languages, avoid rendering they didn't need, etc. So, plenty room for improvement especially if we use plugins or stand-alone apps where possible. Facebook Messenger, etc are ditching the Web that way.
"browser distribution that thoroughly pretends to be from a mobile source and gets those simpler pages. :-)"
Good thinking: I thought of that one, too. Still in backlog. Another model I worked out was ultra-fast, thin-client (phone or netbook) connected to a secure, desktop browser running on best hardware it could. Turns out that a variant of that (for efficiency) was done by mobile companies like Opera. So, it's possible.
"Objectively, I hear they're dramatically losing market share AKA $$$ for further development."
"that they're preparing to move to a multiprocess architecture which will nuke their extension ecosystem"
That is potentially bad. I don't know about their management. Can only hope for the best there. Good news is many of the best extensions are actively maintained or should be ported in event plugin architecture changes. So, here's hoping there too. This from Wikipedia bothers me:
"making Yahoo! Search the default search experience for Firefox in North America effective December 2014.[7] However, it is also to be noted that Yahoo! Search is a merely a front-end to Microsoft's search engine Bing, and Microsoft takes a 12% cut of all revenue from the search business per the deal."
Yahoo itself is in a shaky position and Bing gets a cut? That's risky esp with what they made on Google. I'd have gotten expenses in order instead. Maybe.
"starting with an Raspberry Pi seL4 port (I know, stop laughing, it's the most stable (long life promise)"
It and Beagle port. No laughing here: do what you can. Check out Genode as well as they're among more mature of L4-based architectures. Maybe better integrate it with seL4, Linux virtualization with either, or middleware for integrating components in protection domains. These are low-hanging fruit.
"Thanks a lot for your contributions to all these HN security discussions!"
Appreciate it. The few who might benefit are the only reason I post. Well, I enjoy it too but still. ;)