Aren't you afraid that you are going to forget a passphrase that you don't regularly use? That's my main concern with your approach (I have thought about it and didn't do it because of this fear)
This is obviously not perfect security for that reason and a few others, but it's good enough for me.