I've developed an in-memory database, naturally with it's own programming language.
I disagree that you have to have an application in mind to design your language (sorry GP). But, you definitely need use-cases in mind as every decision has trade-offs..
Some are discoveries I've made along the way: 1. High-level - I remember having a for-each loop zoomed in on my TV, I may have looked at the for a couple of hours as I looked at it and thought "how can I make it better", and it did pay off!
2. Parsers/ Lexers - I couldn't find one that worked the way I needed to (allowed me to remove lexemes that would normally be considered essential). So I started writing my own..
3. Abstract Syntax Trees (ASTs) - I don't think there's a guide on compilers that doesn't mention ASTs. I found this to be more of a hindrance than help. So I removed it, yes the parser generates the program code!!! All that is needed is a small stack of the starting points. This actually works better than anticipated. It's still a single pass compiler, and you rarely need to know to what is next (before delegating). There's checks at every step, the program can also generate sourcecode (reverse engineer itself). So moving away from ASTs doesn't mean that the program is harder to test. A unexpected result of going down this path means that the compiler 'understands' the code better than than most, able to supply actual suggestions rather than cryptic error messages. In-fact it will reject code that won't compile, or breaks dependencies.
The lesson here is don't be afraid to experiment. Yes it has taken some time, with improvements in each iteration (I've done some house keeping in the recently, swapping out ~30% of the parser and it hasn't been as daunting as you might imaging).