I wholeheartedly agree. Methods associated with mutation should raise exceptions. If errors aren't eagerly generated, assumptions about behaviour might be made. Not to mention this will make testing harder.
Readability is impacted as well. Building a mutable copy from an immutable should be explicit. These calls should have a very prominent call signature that is easily read, not skimmed over.
Very cool library, but I can sense lots of confusion and debugging resulting from accidental misuse.
Guava's immutables are pretty solid and serve as a great reference.