If you remove DNS servers from the equation, you need to write down records for other domains, too. This means you have to chase every domain for changes in CDN configuration, hosting provider or ISP migrations, IPv4 to v6 migrations and so on.
You don't have PTR records, which means you can't find out a name from its IP address.
You also miss other features of DNS, like SRV, MX and so on.
More subtly, you lose the ability to control DNS resolution over systems you can't control. If a DNS server says host.example.com is 192.168.0.4, a Windows desktop, a Linux server and your toaster will agree on that (especially if no local cache is enabled, but even then TTLs apply). If for some reason you cannot control a particular machine, you will never get it to consider that new DNS record. This can happen for a lot of reasons.
And I explicitly argue within the section about egress filtering that allowing systems access to public DNS is a security risk.
What happens if you need to scale up (either a lot or a little) and you need to hire new people?
People are often the most vulnerable chain in your infrastructure.