Also, what if the business cost for slow development causes a negative impact? Sure the code and you resume look good, but your startup just became vaporware and ran out of money.
I wasted time before cleaning up code only for it to be dropped or for the code not needing any additional updates for over 5 years.
You can't always anticipate tech debt in advance so it often better to leave the quick and dirty solution and revisit if it becomes painful to update in the future.
It can become debt (or a liability in a more general sense) when you realize, a year or two later, that it's what you need but it's no longer compatible with the base system. So now it has to be cleaned up to be reintegrated properly, is it worth the trouble of the fix or should it be scrapped and restarted?
Or, worse, if it is in the system but disabled by something like a feature flag, it helps to ossify the underlying system because you now have something which depends on capabilities you want to change, but can't because it's in the system.
In a statically typed language this means care is taken with your core system to protect a feature that's not in use, but prevents compilation if you change its dependencies. And then you have to determine if the dependency change should be kept or not, if the unused feature should be kept or not, and if kept how to update it to use the changed dependency.
In many dynamically typed and interpreted language this means your core system is not stuck, but when you finally flip that feature flag on you'll realize you pulled the rug out from under yourself. Unless your tests are sufficiently comprehensive to trigger the error after a breaking change in the dependencies.
Yes, code that is not being used should be deleted as soon as possible. Even if it was the cleanest code ever written - delete it.
It is adding massive mental overhead for future work, especially as a new developer on the project - how do I know its not being used?