The minimum cores you can license on MSSQL, whether on bare metal, or a VM, is 4. Wherever you run it, you must license all cores available to the OS. Had I chosen to run on bare metal, I would be forced to license 12 cores (6-core Xeon CPU x 2). This app requires SQL Server 2008 R2, so the 4GB limit applies. But I can't buy 2008 R2 anymore directly. I have to license 2012, and use my downgrade rights.

You are wrong on the Windows Authentication requirement. This has nothing to do with anything.

For MSSQL, you must license the database per core if you are using it in a public web application, because you have an arbitrary number of users. And you must license all cores in the machine. The VM is the only way to license only some cores, namely, as many cores as are in your VM.

If you are using MSSQL in a private web application, you may choose CALs. However, if you do so, it doesn't matter how users authenticate. Each user or device needs a license. Microsoft is very clear about this. User count is just that: the number of users using the database, either directly, or through a web application which does everything through a single set of credentials. It's not system accounts. It's users or devices.

The same applies for Windows Server licenses. It doesn't matter how the user authenticates. If they are using a web-app hosted on a Windows Server, and that web-app is not publicly accessible (a login page does not count), then you need a CAL for each user or device.