Maybe olive oil, without doing much research, might be an exception where it is not harmful, as long as you know where it is coming from and has not been adulterated or diluted.

One of the problem with processed food is that it allows large quantities, which often means lower quality. Also, you don't see what it's actually being made of.

For example, in the olives article, it cited an instance of sub-standard, damaged olives being used to make olive oil

Even beyond that, it has lots of omega-6s. This is supposed to be competitive in cell chemistry with omega-3s, and the ratio is massively tilted in the omega-6 direction today.

And there's always the anti-fat faction. It's also a high calorie-density food. Like many of these topics, some people will argue that calorie density is irrelevant, but other people will disagree with them.