This question relates to physical, network, application, and operations security, as well as business risk management.
Thanks for your feedback!
For example, I am trying right now to build my first web app. I know certain 'things' eg don't store user passwords in plain text, don't try to build my own crypto solutions etc. But I have no idea of what possible attack vectors exist, which ones are critical, which ones are covered by using best practices, good frameworks etc. In other words, I have no idea where to start, literally because I have no idea! I am currently going through the matasano challenges, but I can't (at the moment as I have just gotten the first set of challenges) for the life of me see how I should apply this new knowledge to building a web app.
Regarding your question, I see you have maybe already outlined the sections of your high level overview presentation: 1: physical 2: network 3: application 4: operations 5: business risk management
But in all honesty, I really don't even know enough to answer you: What do I wish I knew? Because I didn't even think about those 5 areas even.
First, I'd like to simply mention that privacy and security are two different but very important things.
The difference is that while your severs may be "secure" from hackers, if you do not take care of data or user privacy people within may have access to sensitive data. It is important to maintain a high threshold for potentially sensitive data.
Some applications that pose privacy risks are Dropbox or Google Drive. Neither encrypts your data, but it is secured and kept private over-the-wire via SSL 256-bit encryption. It is true that both Dropbox and Google have access to your files, but they maintain a privacy policy and hopefully internal policies and systems to prevent just anyone from gaining access to your documents.
Network security in applications is fairly standard, nowadays the Internet can handle SSL pretty well, and with SPDY picking up adoption there is no reason to not enforce SSL for all connections to an applications.
Application security comes in a few ways. As a developer, you want to protect your IP, give the user good performance, but make it hard for jerks (yes, just jerks) from removing any DRM/licencing/registration processes you may have on applications. These types of things are common on mobile apps, it is easy to find re-released mobile apps for Android and iOS devices online if you know what to search or a good repository for it. Application security comes in the shape here that you need to enforce a licencing system that works for you. Many pieces of software out there assist with with, so you may not want to reinvent the wheel, but adding some custom processes here usually results in a good solution until someone with some knowledge has some free time and doesn't want to spend money.
Application privacy is important on mobile and desktop applications as well. The user data you store usually is not as-at-risk as you think, and many developers ignore proper processes as "it's up to the user to keep their device safe." Banks obviously do not feel that way, luckily. Games probably shouldn't bother. Social apps might want to step it up though. If you lose your device a lot of data is available from social apps and applications should simply should have processes to fix this. Twitter and Facebook allow users to do this remotely by rejecting/revoking an application's access to your data, but applications often store data on devices for offline viewing. Therefore this data should would be focus for a developer's concerns.
Encrypting user is a good idea, but is hard to implement unless the team is good with security. Encryption also comes in many levels. It's a very good idea to encrypt your server hard drives or databases, this prevents a lot of low-level attacks onto your system. Key management is therefore also important, and is is where things will get complicated. Encryption is also important for user-entered data however, beyond just encrypting your databases. Because database-encryption is slow, it usually isn't a good idea anyway. Therefore instead of doing that, you should just encrypt the user-entered data that is sensitive.
Network security is unknown to most web developers, who believe that throwing an SSL certificate up makes their applications secure. Obviously, as I mentioned, this helps. However network security comes in many other ways. Applications hosted on shared servers, switch servers. There are so many security risks in shared servers that attempting to control it is not worth the time. This does not include VPS systems however, assuming you are the only root/sudo users authorized. On whatever system you are on, you need to enforce a whitelisting firewall. Web servers should never use a blacklist for this. Most web servers only need a few ports open, so only open those. It's also common practice to leave SSH open, and a lot of people believe that by switching the default port it's magically okay to do so. It's not. Port scanners exist, and are easy to use. Your SSH should only allow connections from authorized networks, ESPECIALLY ON PRODUCTION machines. Get a VPN, remote desktop to your home and then SSH, or tunnel through a development machine. This goes the same for a lot of other services, FTP is another major one people will leave open, which on some instances allows anonymous by default. I've also heard of companies leaving MySQL remote ports open and that makes it exceptionally easy to bruteforce as many people use rather small passwords for databases. So another point there, make sure your database password is not so simple.
Operations/company policies should hold user privacy to the level decided by the company. I wish it was the highest priority for every company, but honestly, facing the through is that it's not. Every company should however figure out where user privacy stands and how important it is to them. This is generally done when setting up a privacy policy. In general, you should never share (or sell [personal opinion]) user data, give access to all colleagues, keep admin systems open for hackers to attempt to gain access to, allow all colleagues to have admin access to admin consoles, share admin logins with colleagues. A lot of small companies, especially startups, give full access to everyone to make it more open. This is extremely dumb. That is just the truth about it. Admin consoles are dangerous tools that should only be accessed lightly. This is of course different from reporting tools, which are used by business operations, marketing, etc. Admin tools can see, add, remove, modify user data and system settings. These should always be restricted on production machines. While it is often important to run SQL or database queries against production data, it is also bad practice to do so on production. Therefore you should setup a staging machine to keep a copy of production data allowing for proper testing. Assuming you keep user data encrypted properly, there is no risk, assuming it's like most software and there is no encryption beyond a secure hash to a password you should also consider proper privacy policies for your company. Sometimes removing certain data from a few tables is the best way to go, other times it's to have all production-staged queries run by a DBA who is aware of the company's privacy policies.
There is a lot more worth adding here, but your best bet is to figure out the team's stance on privacy and security and their knowledge about it. A lot of web developers assume the only aspect to security is cross site scripting (XSS), Clickjacking, sql injections, and cross-site request forgery (CSRF). This is extremely far from the truth. Most security advisers suggest that the biggest security risks are from within a company. It is extremely important to fight against outside hackers of course, but it's important to realize that security is more than just stranger-danger.