Most large SoC vendors only sell their chips to qualified partners who enter into contract agreements. They only want partners with significant technical resources and commitments to significant minimum order quantities (6 figures).
> My first non-trivial PCB in fact, and it seems like it's going to work.
Building a PCB to support a modern SoC is a serious undertaking. Layout and routing for high speed traces (DDR RAM) and power supply distribution is a very challenging topic. Many of these SoCs are in dense BGA packages, requiring 4, 6, or 8 layer boards with advanced features like via-in-pad as the BGA pitch gets smaller and smaller. This is within the reach of dedicated hobbyists, but it's something you'd want to work your way up to over several years.
> The Pi computer module has been the most accessible way I can find to get my hands on a Broadcom processor, but still, this unit adds $35 to my BOM, from which I estimate 15-20$ to be unnecessary parts (for my application).
The Raspberry Pi compute module is a great solution to your problem. You'd be surprise at how expensive it can be to design, develop, source, and manufacture an equivalent solution on your own.
Sourcing a pre-built module from an SoC partner is the standard way to do this, even for companies operating at small (<1000pc) volume. The R-Pi compute module is particularly large, but other SoCs can be found in significantly smaller modules. Search for "Linux system on module" to find other providers. http://linuxgizmos.com/ has a news feed of module announcements.
If you absolutely must design your own system, there are some SoC vendors who sell to retail customers and provide open access to their documents. They are the exception rather than the rule, so you must explicitly target these chips from the start.
Perhaps the most well-known vendor in this category is NXP with their i.MX series of chips. You can buy them on DigiKey and download all of the documentation from NXP's website. If you want to get started, take a look at their i.MX6 series: https://www.nxp.com/products/processors-and-microcontrollers... .
However, even with the i.MX parts it's still a better choice to use a pre-built iMX module for your prototype phase and even early shipping units. Prove the product and design first, cost reduce with a custom PCB later.