This is pretty much the same dilemma as all content creation packages - whether 3D software, audio, CAD, video editing etc. Internal formats need to be fast and robust to frequent revisions, and will inevitably support procedural transformations on data that are unique to that software (eg a particular mesh modifier, particle system, audio compressor or video effect).
So external formats become quite difficult to specify, and generally require that everything gets baked/bounced down to the lowest common denominator - eg per-frame animation channels, or EDL-level edits in a video.
The .blend format isn't actually that bad, if you really need to access it. I've seen a number of external libraries written in C or Python that do this, and it's one of those formats where you can skip over the unknowns (as indeed Blender itself does when loading files from the future).
If you want to do 3D interchange, this will depend much on your particular application. We have FBX for mesh, scene and animation data which is pretty good. OBJ is always there for the simplest of things. ABC (Alembic) support is coming to Blender for more complex geometry, and there's talk of OpenVDB for voxel data.
Other roads out include numerous exporters for various game platforms - THREE.js, the new Khronos gITF format or OpenGEX.