Artists, who use these tools with clear vision and intent to achieve specific goals, strangely never have this problem.
For example, I took a picture of my dog at the dog park the other day. I didn’t notice when framing the picture but on review at home, right smack in the middle of the lower 3rd of the photo and conveniently positioned to have your eyes led there by my dog’s pose and snout direction, was a giant, old, crusty turd. Once you noticed it, it was very hard to not see it anymore. So I broke out the photo editing tools and used some auto retouching tool to remove the turd. And lucky for me since the ground was mulch, the tool did a fantastic job of blending it out, and if I didn’t tell you it had been retouched, you wouldn’t know.
Is that a fake image? The subject of the photo was my dog. The purpose of the photo was to capture my dog doing something entertaining. When I was watching the scene with my own human eyes I didn’t see the turd. Nor was capturing the turd in the photo intended or essential to capturing what I wanted to capture. But I did use some generative tool (algorithmic or AI I couldn’t say) to convincingly replace the turd with more mulch. So does doing that make the image fake? I would argue no. If you ask me what the photo is, I say it’s a photo of my dog. The edit does not change my dog, nor change the surrounding to make the dog appear somewhere else or to make the dog appear to be doing something they weren’t doing were you there to witness it yourself. I do not intend the photo to be used as a demonstration of how clean that particular dog park is or was on that day, or even to be a photo representing that dog park at all. My dog happened to be in that locale when they did something I wanted a picture of. So to me that picture is no more fake than any other picture in my library. But a pure “differentiate on the tools” analysis says it is a fake image, content that wasn’t captured by the sensor is now in the image and content that was captured no longer is. Fake image then right?
I think the OP has it right, the intent of your use of the tool (and its effect) matters more than what specific tool you used.
And if shot composition doesn't make it fake, what if I cropped the photo after the fact? I'm removing something the camera captured to better make the picture what I "felt like it should be" just using the removal tool. That's functionally no different from framing the shot differently, but it's modifying the actual captured image.
If we decide that removal, whether by framing or by post-hoc cropping is still "real" and it's the use of a tool that adds something that wasn't there, would the same apply to just cutting a square out of the photo without cropping the rest of the frame? A transparent square would be an interesting artistic choice for sure, but does that then get into the realm of "fake"? What if the square is black or white? Is adding a clearly "post process censor bar" crossing a line into making the photo "fake"?
If those are fine, it's the "adding content that looks like it should be there" is the problem, does that mean that dust or hair removal to remove something that was on the lens make it fake since that would also have to generate what the computer thinks is behind that hair or dust speck?
For what it's worth, I don't think there is a hard line here, like I said intent matters. But I do think that figuring out where ones personal and general lines are and when they might move that line or not is an interesting thought experiment.