You still need to make sure that no one except the pilot can take over the controls and that nobody is secretly loading up the VTOL with bombs in their luggage with the intention to crash it into the side of a glass skyscraper.

Then there is the safety of the passengers. If you and the pilot are the only ones in the aircraft, there is not much risk to yourself, but if the pilot dies, then the aircraft must land autonomously. If there is a group of passengers, then a terrorist could take the passengers hostage. So you still need to check for weapons and explosives.

The only case where you could get rid of security is with a fully autonomous aircraft that carries no luggage and only a single person.