I was thinking about this recently, the way to do is to define a radius, and then imagine rolling a circle of that radius around the outside of the coastline (or around the inside! Define that as well) and then take the length of the equivalent track that never leaves contact with the circle.

So you get a different length depending on the radius you choose, but at least you get an answer.

You could define the radius in a scale-invariant way (proportional to the perimeter of the convex hull of the land mass for example) so that scaling the land mass up/down would also scale our declared coastline length proportionally.