Hacker News Reader

Comment by Yoric | Hacker News Reader

Hi, I’m one of the Firefox developers who was in charge of Session Restore, so I’m one of the culprits of this heavy SSD I/O. To make a long story short: we are aware of the problem, but fixing it for real requires completely re-architecturing Session Restore. That’s something we haven’t done yet, as Session Restore is rather safety-critical for many users, so this would need to be done very carefully, and with plenty of manpower.

I hope we can get around to doing it someday. Of course, as usual in an open-source project, contributors welcome :)

View on HN

How about this: 29 out of 30 times, save only a diff to the previous data. 1 out of 30 times, save the complete data in compressed form.

(I'm guessing there must already be functionality to diff a bunch of JSON somewhere in the millions of lines of code).

Though I'm sure this doesn't make usually make a dent in a SSD's lifetime. But there are still people running Firefox on low end Android phones with meager flash, and Raspberry Pis with SD cards.

You also need to handle all corner cases where one of the intermediary diffs gets corrupted (you can't generally assume in a program like Firefox that data you write out is going to be readable in the future because lol common hardware). Or where the diffs are larger than a fresh snapshot. And you absolutely can't get it wrong.

It's something that sounds easy until you actually try to get it coded up and shipping.

I explained below that this thing isn't a factor for Android because the program gets suspended.

Firefox does generally assume that the filesystem is reliable (as long as you use fsync properly etc). Witness eg its reliance on SQLite for data storage.

SQLite is used exactly because it is so robust even on bad hardware. And you still need to check the data that SQLite is giving you.

Until you have a NFS homedir...

On the other hand, SQLite with concurrent local and SSHFS-remote updates (where the SSHFS server is run with caching disabled) is entirely stable.

I'm curious, could you expand on this comment? Do you mean that you run Firefox from a profile loaded over SSHFS? I can only imagine that being unbearably slow because of I/O latency. And shouldn't disabling caching in SSHFS make it much, much worse?

Does the lack of proper locking on NFS matter when you're only running a single instance of firefox?

In the general case that's a huge pain. In this case, where you're writing a single blob to disk, you can stick a checksum at the end and you're good to go.

If you read the bug discussion, you'll see this is one of the most problematic parts.

This bug? https://bugzilla.mozilla.org/show_bug.cgi?id=1304389

It doesn't talk about the difficulties in getting data safely to disk. There's just a worry that taking a hash of the entire session state is expensive. I'm skeptical that a fast hash would take long compared to the time spent serializing to JSON in the first place, let alone time spent diffing.

As Microsoft have discovered in their binary patches through Windows update. If you've ever run sfc /scannow and then dism /online /cleanup-image /scanhealth then you are almost certainly seeing corrupted binary patches.

> How about this: 29 out of 30 times, save only a diff to the previous data. 1 out of 30 times, save the complete data in compressed form. > > (I'm guessing there must already be functionality to diff a bunch of JSON somewhere in the millions of lines of code).

That is actually a good idea we haven't considered yet. A bit too brute force for my tastes, but relatively easy to implement. We would need to determine how much CPU is needed for a diff between two 300Mb JSON files, though (yes, some users have these).

Of course, we're back to the issue of manpower, but definitely worth trying out.

> Though I'm sure this doesn't make usually make a dent in a SSD's lifetime. But there are still people running Firefox on low end Android phones with meager flash, and Raspberry Pis with SD cards.

The implementation of Session Restore for Android is largely independent, so I'm not sure how it works these days.

Surely you'd diff the data structures in memory, not the serialized JSON; and would it really be faster to blindly write the 300mb each time than perform this diff?

To be benchmarked.

You could at least just compress the data with LZF or any of the really fast text compressors.

It'll compress to about 30% of the size, it's easy to do, and it shouldn't add more than a tiny CPU overhead over formatting the JSON itself.

It solves half the problem with like 15 minutes of work.

Actually, we had a prototype doing this. In the end, we didn't because it broke some add-ons, but it might be time to try again.

Broke some add-ons that read or, worse, write data behind the browser's back, you mean?

Also, what do these add-ons do? The only use case I can think of is figuring out whether the user has a tab open to a given site, and that's going around the browser's security model, so breaking that would be a good thing.

15 minutes of coding, perhaps.

Then an hour of writing good tests.

Then lots of manual and automated testing on four or five platforms, and fixing the weird issues you get on Windows XP SP2, or only on 64-bit Linux running in a VM, or whatever.

Then making sure you don't regress startup performance (which you probably will unless you have a really, really slow disk).

Then implementing rock solid migration so you can land this in Nightly through to Beta.

Then a rollback/backout plan, because you might find a bug, and you don't want users to lose their session between Beta 2 and Beta 3.

Large-scale software engineering is mostly not about writing code.

They already have Google's Brotli imported, they'd only need a small tweak to also include the encoder. Or use Snappy which is also in the codebase already.

Add the code that's able to load compressed session backups and leave it in for a couple versions.

Once enough versions have passed enable the code that writes compressed session backups.

It's really not that hard to do unless you want to enable it now.

> regress startup performance

No, for example, LZ4 is unbelievably fast:

https://github.com/Cyan4973/lz4

almost 2 GB per second in decompression!

I've just tried compressing some backupXX.session file (the biggest I've managed to find, just around 2 MB) and it compressed to 70% of the original, probably not enough to implement the compression -- and I suspect the reason is that the file contains too much base64 encode image files which can't be much compressed?

So the answer to having sane session files can be first to stop storing the favicons (and other images(?)) there? I still believe somebody should analyze the big files carefully to get the most relevant facts. For the start, somebody should make a tool that extracts all the pictures from the .session file (should be easy with python or Perl, for example), just that we know what's inside.

So "somebody" was me at the end, I've rolled up my sleeves and extracted the damn pictures: in my 1.7 MB session file approximately 1 MB were the base64 encodings of exactly 57 jpeg pictures that were all the result of the Google image search (probably of the single search). There were additionally a few pngs, representing "facebook, twitter, gplus" images and one "question mark sign" gif too.

Thank you for doing that. This explains much.

For a while now I have been running a cronjob to commit my profile's sessionstore-backups directory to a git repo every 5 minutes.

This is because, occasionally, when Firefox starts, it will load tabs in some kind of limbo state where the page is blank and the title is displayed in the tab--but if I refresh the page, it immediately goes to about:blank, and the original tab is completely lost.

When this happens, I can dig the tabs out of the recovery.js or previous.js files with a Python script that dumps the tabs out of the JSON, but if Firefox overwrites those files first, I can't. So the git repo lets me recover them no matter what.

What I have noticed is that the git repo grows huge very quickly. Right now my recovery.js file is 2.6 MB (which seems ridiculous for storing a list of URLs and title strings), but the git repo is 4.3 GB. If I run "git gc --aggressive" and wait a long time, it compresses down very well, to a few hundred MB.

But it's simply absurd how much data is being stored, and storing useless stuff like images in data URIs explains a lot.

If I understood the intention of the programmers, they simply want to store "everything that can imitate to the server the continuation of the current session" even after the new start of the Firefox (like the restart never happened). The images were sent by Google, but probably remain in the DOM tree which is then written as the "session data" or something like that.

Like you, I also observed that exactly the people who depend on the tabs to "remain" after the restart are those who are hit by the bugs in the "restoration" and as I've said, I believe the users would more prefer to have "stable" tabs and URLs than the "fully restored sessions in the tabs" when all the tabs fully randomly (for them) disappear. Maybe saving just the tabs and URLs separately from "everything session" would be a good step to the robustness (since it would be much less data and much less chance to get corrupted) and then maybe, pretty please, an option "don't save session data" can be in the about:config too)?

Once there's decision to store just the URLs of the tabs as the separate file, the file can even be organized in a way that just the URL that is changed gets rewritten, therefore making the "complete corruption" of the file impossible and also removing the necessity for Firefox to keep N older versions of the big files (which then eventually still don't help the user like you).

> Maybe saving just the tabs and URLs separately from "everything session" would be a good step to the robustness (since it would be much less data and much less chance to get corrupted)

Yes, that would be very, very useful. I can get by if the tab's scroll position and favicon and DOM-embedded images--and even formdata--are lost. But if the tab itself is lost, and it was a page I needed to do something with, I may never even realize it is gone...

Why is there so much data in the session restore anyway? If the goal is to have the URLs of the currently opened tabs, I'd expect that just the given URLs should be enough? I think I've seen some unexpected stuff there like the images base64 encoded? Maybe there's enough users that would be satisfied just with the URLs? At least for them the "rewrite" would be seldom needed.

Or, maybe to reformulate, which wild scenarios does Firefox want to support now? I can imagine that the user's experience wouldn't match the wishes. Some people that use session restore claimed they "lost everything" from time to time, and I had to fish "just the urls" from their session store files which looked strange ("full of everything"), but automatically restored to nothing.

Agreed. If my computer crashes (once in a blue moon) I would be happy for it to just open the urls again. I don't care whatsoever about it keeping the current state of every tab. If any users do care about that, it should be optional (perhaps a tickbox in the preferences window).

My FF has crashed many times on OS X and while it does restore the URLs, it has never restored the state. I know it because sometimes I am in the middle of writing a comment and it crashes. On recovery, the comment is gone.

Get Lazarus

https://addons.mozilla.org/en-US/firefox/addon/lazarus-form-...

Session store contains open tabs, windows, history for each tab, form fields, referrers (so we can re-request the page correctly), titles (so we can restore tabs without re-fetching every page), favicons, charsets, some timestamps, extension data, some kinds of site storage, scroll positions, and a few other things.

The goal of session restore is to restore your session -- your open tabs should come back, the same pages should load, scrolled to the same place, and with the right content.

I'd wish they'd also restore themselves to the proper location on the proper virtual desktop. With hundreds of windows, typically organized by task, I dread having to reorganize things every time Firefox (or the computer) restarts.

Someday the pain may motivate me to try to learn enough about Firefox's internals to do it. :)

Definitely worth filing a bug in bugzilla if they are not on the right virtual desktop... it might take years to get fixed, but they do generally get round to it.

Or generations, you know? https://news.ycombinator.com/item?id=11951413

There already is a 10 years old bug about the issue: https://bugzilla.mozilla.org/show_bug.cgi?id=372650

Storing a lot of static images as JSON base64 encoded, every 15 seconds is certainly not something that users should be blamed ("some users have 300MB JSON files"), just the poor programming.

It would be interesting if somebody would actually analyze what takes the most of the mentioned 300 MB. I see a lot of base64 encoded stuff, if they are "favicons," come-on. There are so many caches in Firefox already, JSON files certainly aren't the place for these images.

Nobody is blaming the user for the 300Mb. It's just a factor we need to take in consideration.

But yeah, storing favicon in Session Restore would be pretty bad. I didn't remember that it was the case, though.

I don't see favicons too, but I've just did one "image extraction" attempt:

https://news.ycombinator.com/item?id=12569999

700 KB of binary images in a 1.7 MB session file, which can be compressed only to the 70% of its size.

I also see a lot of things like \\u0440 which spends eight characters for one unicode character (in another file, not from me). But that file was reduced to 37% of initial size with LZ4. It seems LZ4 is still worth doing, if the content remains easily accessible with the external tools, e.g. lz4cli.

As an Fx user, I'm glad that favicons are stored though. The icons are a much easier, quicker indicator of what the tab is, when I'm scrolling through dozens of tabs to see which one to click and load.

There's no need to store favicons in the session JSON. They're stored in the browser cache. If the cache gets cleared in the meantime, they can be redownloaded.

Long time Netscape/ firefox user here:

1. Thanks for your work.

2. How do I really contribute? Could you link to what I need to do to start working on this now? I'm affected by this problem ame want to figure out if I can fix it.

http://whatcanidoformozilla.org is a good starting point.

"Please enable JavaScript in your browser!"

That advice was a lot simpler than I thought it would be.

Why writing to disk if firefox is idle and its state doesn't change ? Is it repeatedly rewriting the same data over and over in a storage working like a log database or journal file ?

Unless there is a bug, Session Restore is not written if Firefox is idle. Of course, many applications are not idle even if you're not using them, and with the current architecture, just one tab using, say, DOM Storage, or updating a session cookie or a hidden form field, is enough to force a rewrite.

As an heavy user of session restore and tons of open tabs, I would be OK ignoring updates to tabs that have not been active for a while.

For example, let's say I have 10 tabs open but I have been using only the latest tab during the last 5 minutes. In this scenario I don't care about the cookies (eg due to ongoing ajax) nor the state of the other 9 tabs. If the browser crashes I'm OK with those 9 tabs being restored with a 5 minutes old snapshot.

So for instance FF should be smart to save the state of new/recent tabs, and should slow down progressively on old/inactive tabs.

don't generalize your personal use case

What would an updated version of Session Restore look like? I mean, what would be your vision for a new version?

I haven't worked on Session Restore in some time. But from the top of my head, it would be as follows.

On the back-end, I believe that Session Restore should be backed by a database, with each tab updated independently, rather than a big bunch of JSON data. The rationale being that:

- it's performance-critical;

- it's safety-critical;

- we have users with 300+ Mb of data in their Session Restore and JSON isn't meant for this scale of data;

- we wouldn't need to rewrite x Mb of data every 15 seconds, just a per-tab update;

- if we're using a relational database, it would be easier to trust the code to not screw up with the data;

- we wouldn't need to load the entire Session Restore upon startup.

(On the minus side, this might make backups a bit more complicated.)

On the front-end, we would need a high-level API for Session Restore, which would let us do things such as accessing per-tab data, (de)hibernating tabs, etc. Oh, and it would need to be accessible by WebExtensions.

In the middle, we would need to re-engineer Session Restore to make sure that we don't need to maintain this huge object representing the entire state of the session. We would also need improvements e.g. to cookie management, to avoid having to re-collect cookies so often.

How does that much data get into Session Restore? Mine are only about 3KB per tab, and Firefox starts melting at more than a thousand tabs.

I seem to remember that DOM Session Storage and Session Cookies are pretty big offenders. Other websites (e.g. Google Docs) use lots and lots and lots of hidden forms to remember lots and lots and lots of data.

Also, people who keep Session Restore tabs open (you know, the tab that lets you restore your session, when you have crashed) and continue browsing – Firefox needs to store several nested Session Restore JSON files.

I've always been impressed by that, because it's essentially bullet-proof.

I wish it were, but every now and then, I restart Firefox and get empty, title-only tabs that go to about:blank when I reload them, completely losing the tab. I have to use a Python script to dump the session data (which still appears in the JSON file, even though Firefox won't use it), eyeball the list to figure out which tab was broken, and reopen that URL manually.

Using a SQLite database for Session Restore seems to make sense. However, it would make it even more difficult to get data out of it when necessary. As ugly as a mess of JSON is, at least it's text, and I don't have to write a SQL query to grep for URLs.

Why not just use the filesystem? Imagine a layout like this:

    + profilename.default
    |--+ sessionrestore
       |--+ <timestamp>
          |--+ window1
          |  |--+ tab1
          |  |  |--- cookies
          |  |  |--- formdata
          |  |  |--- title
          |  |  |--- url
          |  ---+ tab2   
          |     |--- cookies
          |     |--- formdata
          |     |--- title
          |     |--- url
          ---+ window2
             |--+ tab1
             |  |--- cookies
             |  |--- formdata
             |  |--- title
             |  |--- url
             ---+ tab2   
                |--- cookies
                |--- formdata
                |--- title
                |--- url

One directory per timestamp, containing one directory per window, containing one directory per tab, containing one file per data type. All plain-text. Change a tab, just write to that tab's files, and update the timestamp directory name. Every x minutes, make a new timestamp tree, keeping y old ones as backups.

No serializing and writing entire sessions to disk at once. Only small writes to small files. All plain-text, easy to read outside of the browser. Easy to copy, backup, modify, troubleshoot. No complex JSON, serializing, or database code. Use the write-to-temp-file-then-rename-over-existing-files paradigm to get atomic updates.

Need to know if part of a session on disk is stale? Check the mtime for that file, see if it's older than the last time that data was changed in memory.

Simple stuff. Straightforward. No overengineering. No bloat.

Why not do this?

Not cross platform, without a tonne of extra work that, surprise surprise, things like sqlite already take care of. Overengineering is ignoring the off-the-shelf part that solves your problem because you "know" you could do it better yourself, in a simpler fashion.

It's a non-issue anyways, I have yet to see anyone show any actual evidence of an ssd dying prematurely or suffering degraded performance from this behaviour.

what operating system does not have filesystem support? :(

Have you considered minimum FS Cluser/Blocksize? Older versions of IE used to write cookies in plaintext files. So now you're wasting 4096 bytes for a few bytes of storage.

OTOH SSD lifetime is rather safety-critical for many users...

_Safety critical_? I doubt that...

How can I disable it completely?

http://kb.mozillazine.org/Browser.sessionstore.enabled

I think you are safe to rewrite it. It is not very reliable anyway as I have lost my tabs after crashes more than once. It might have been possible to save them but after a crash first thing I would do is restart Firefox and it would sometimes just completely overwrite the recovery.js.

To mitigate this I have created a daemon to copy the file to a separate directory every few hours. I then delete those old versions manually every year or so.

I have had the same problem. I finally made a git repo in the sessionrestore directory, setup a cronjob to commit it every 5 minutes, and then use this script to dump the session contents when necessary:

https://gist.github.com/alphapapa/bc7f3b25025fb99dad56

The git repo grows in size rapidly, but can be compressed way down. (Today it was 4.3 GB, but "git gc" compressed it to 230 MB). Every now and then you can blow away the repo or filter-branch to get rid of outdated sessions.

Why not leave the existing code (SR1) as is and rebuild Session Restore 2 (SR2); keep using SR1 until it has nothing to restore and then use SR2 moving forward.

That sounds pretty hard to pull off. What's the expected benefit?

That's a pretty standard M.O. for server-side software development. But in the case of desktop software such as Firefox I see no gain.

> Session Restore is rather safety-critical for many users

Maybe you could give users the option? (until it can be fixed fully)

That's what the interval config setting is for?

I think brador is referring to the options page.

about:config shows a big warning message "This may void your warranty".

Which warranty?

Have you looked at about:config recently? :)

Woosh :-)

That's not SSD-only problem. HDD affected too. FF UI is just freezing when using HDD (linux; tried WD RE4, 7200rpm or WD blue 7200 rpm), several years ago I wrote script for myself to keep profile data on RAM disk and sync back to HDD, and problem gone.

Session Restore is already a little buggy. Better than it used to be certainly, but it occasionally and apparently irreversably turns the odd window into a window full of empty New Tabs rather than their original contents.

link to bugzilla?

Reply on news.ycombinator.com