The second route is the "trade-in." Most dealers almost never pay cash for the amount of money that they offer for trade-in. It's purely a negotiating tactic meant to make a buyer feel better about a (relatively arbitrary) price on a car ("$14k for a 2015 Ford Focus is kind of high, but we really like that we're getting $4k for our 2007 Pontiac Aztek!"). Whenever you work with a dealer, you're almost always getting fleeced. Again, some people enjoy the process as a sort of game or hobby, but you're likely not making a profit unless you're extremely good at it.
EDIT: I read this over and it sounded a little fanboyish. I want to make clear that I've been gone from CarMax for almost 5 years and I don't own stock or have any other financial stake in their success other than some friends who are still employed there. I just think they're a good company (although I didn't love working in IT).
Also, never do the 4-box and have your financing sorted out up-front if you're not doing cash. Any chance they get to adjust more than one number is when they'll adjust the terms in their favor.
There's more than just minutes spent in dealing with private-party sales.
Forgive me for inferring too much from your statement if I am, but this is exactly the kind of situation in which the zero arbitrage principle doesn't apply. if I make a bunch of offers to buy a car for less than it's worth, then all I need to do is find one person who is selling that car and not shopping around. There's no reason to expect any market forces to govern that transaction at all.
There is no reason to believe that selling your car for whatever the first person you find offers is likely to give you a fair price.
I'm basically speaking from personal, anecdotal experience selling (and buying) a few low-end cars via Craigslist and equivalents.
I have other things I'd rather do with my time than either handle a bunch of prospective buyers inspecting, test driving, and low-balling my car (and dealing with potential fraud), or myself inspecting, test-driving, and low-balling other people's cars, even though that may be a way to save/earn some money.
From an economics point of view, the transaction costs are significant (even if not dollar-denominated), so the arbitrage opportunity exists for someone who can minimize or simply disregard those costs.
goatherders and I both described working with multiple buyers, and that finding the right buyer (that is, the one who will pay you $1,000 more than some other buyer) requires time and effort. When considering whether to accept an offer on the table or hold out for a better offer, the seller implicitly or explicitly considers the transaction costs (search, delay, processing, risk, privacy, etc.).
This implies non-zero arbitrage, multiple offers, and market inefficiency.
Let's say it's an extra 12 hours to sell via private party (posting pics, meeting people, haggling, etc). That's $80/hr. You may not actually be earning that whole time, but it's also opportunity cost for doing other things you may value more (time with family/friends/etc).
I also just changed the oil and am considering changing the differential fluid.
I've already put multiple hours into listing, uploading pictures, meeting the guy to test drive, responding to texts, phone calls, etc.
Because of the relative value of the car, I intend to meet the buyer at the BUYER'S bank to do the paperwork and see the cashier's check drawn in person (there are lots of cashier's check scams these days).
Assuming I sell it to the interested buyer, it's a fairly conservative estimate to say it'll take me 6-8 hours with the one buyer. If I list it more widely on, say, craigslist, I get to deal with finding time to meet a bunch of tire kickers, THEN finding the time off work or on a saturday morning to go to the bank to do the transaction.
In California, if the vehicle is 5-6 years old or more, the seller is also responsible for getting the car smogged; likely if you took it to a CarMax type place they'd take care of all of that stuff for you (just a guess though).
In the end, I am LIKELY to make, say, $5k over what a dealer would offer me, but there's no way of being sure that I'll get 1 cent over what a dealer would offer.
Ultimately it depends on the market. I only buy fairly unusual cars; my previous car took me 4-6 months to find, the car I'm selling took 4 months to order. I once took 6 months to sell a purple M3.
Done right, enthusiast cars hold their value well, but the target demographic is smaller so they take longer to sell.
All this being said I likely won’t buy a used car again. My 3 other cars were all used.