The SAM L11 doesn't have a USB peripheral... are you bit-banging USB or using a second IC or something? I was considering using an L10/L11 for another project, but this omission is what led me elsewhere.
Edit:
Thinking about it some more, it may be advantageous to rely on a second, physically separate chip to handle USB communication, as it's next to impossible to verify that there isn't a backdoor in the secure chip. Chips that are likely to be used as secure elements are probably juicy targets for backdoors, and a USB peripheral would be an excellent place to hide one (e.g. a special "knock" code that dumps the contents of secure memory and resets the chip and is unlikely to be found via fuzz testing).
I really like the concept, though. I think we need more efforts like this, aimed at making our security tools simpler, more open, and easier to verify.