Logfiles are necessary to operate a service securely and guarantee quality of service. This is one form of implicit consent that users are giving you without you having to ask them for it.

But Storing IP addresses for each access indefinitely (> some days) is the problem.

If you rotate the files into a version where IP addresses are without the last part after a few days, then this is considered pseudonymous data and GDPR has no problem with you keeping this for a long time anymore.