I think my hardest thing to debug wasn't really a bug at all. It was having to clean up some legacy code I'd inherited and one function which was a bottleneck had a few hundred lines of horribly complicated code with lots of calculations which didn't make sense to me. It took a lot of time to figure out that the calculations were wholly unnecessary. The results of the calculations either got discarded or unconditionally overwritten. In the end, the refactoring consisted of a single press of the "delete" key, but building up the confidence to press it was what cost a lot of time and effort.