'I'm going to the library', said Hermione.

You put the punctuation where it makes sense. If it belongs to the quote, it goes inside. If it belongs to the sentence, it goes outside.