If you opt for making your declarative language a subset of JSON or YAML, presumably it will be a proper subset, with some further expectations and constraints. It makes total sense to use a custom file extension — say, `.mdl`, "my declarative language", for the sake of discussion. Users will be able to easily find their files written in your language without having to wade through all possible files of the superset format. A `.mdl` file really is its own thing, it's not just a .json or .yml file. It may be the case that a `.mdl` file is a just `.json` or `.yml` file, but the converse will not be true.
The only advantage I can think of that accrues from using the more common extension is editor support: a user's favorite editor may well provide syntax highlighting and checking, auto-indenting, etc. for `.json` or `.yml` files, but not for `.mdl` files out of the box. You might want to develop `.mdl` profiles (sic) for popular editors.
Any decent parser for the 'true' format will accept either a string or a full filename (including extension) and shouldn't expect a fixed file extension.
Regrettably I can't think of a single example right now, but I know I've encountered many programs that use custom file extensions which have turned out to be just some familiar format after all.
That said, it's another question whether either JSON or YAML is a great choice. I agree with your reservations about both. In fact I've had the very same problem in a couple of development projects, one recently. JSON is friendlier than XML, true, but it's not really a language to think in, even declaratively — too much clutter. YAML is certainly versatile enough, but it provides probably more than you need, and it is... yes, obtuse. Of course, it depends on your intended users; I found it was too "programmerly" for mine. I ended up developing a custom parser for a language with more syntactic sugar than YAML, which has its advantages and disadvantages.