Factories are a pattern that is not unique to Java. They're used prolifically in C++ at least, and I'm sure in other languages.
The factory pattern is tightly coupled with polymorphism. If you're not using polymorphism, then you may not typically need factories. But if you are, then at some point you're going to have some conditional logic about what concrete type of class you're going to create of a particular interface.
Modern Java is lending itself more towards data oriented programming, and coders are trending in that direction. This may or may not last, like every trend. But even then, at the moment this tends to still involve using sealed interfaces and records, so you still have some form of instance creation and polymorphism.
How do you avoid factories? Doesn't you code end up littered with worse code?