Maybe you'd want to compute the convex hull on a per-connected-component basis, though, to avoid allowing small islands to have undue influence.
Either that, or say that any area of sea within its convex hull gets treated as part of the state's area. This approach differs in that it special-cases sea over other kinds of land.
So...we've got at least 3 ways so far to assign a measure of convexity/concavity to a state, and they give quite different results.
For instance, your method and the method in the comment you responded to both would assign a low concavity to a state that is almost a square, except that the boundary has a high frequency, low amplitude sawtooth pattern imposed on it. Mine would score that has a very high concavity.
http://www.ncbi.nlm.nih.gov/pubmed/7945702
Although I don't have PubMed access so I can't be sure.
This is an interesting problem.
I chose the "random points" method because it seemed easier to me to get an approximate answer for a generic shape rather than try to figure out areas. Lengths didn't occur to me though and seems interesting, although the shape you describe doesn't "feel" concave to me.
I wonder if you could prove this "probability of a random line segment violating convexity" definition equivalent to something given in terms of a ratio of different areas like the area to convex hull area suggestion below.
But I don't really think of Hawaii as "concave". So I'm going to refine my definition to only allow the points to be chosen in connected areas.
Are two areas considered connected if they cross a river, once?