It's very strange. In the UK you often pay a fortune to sit on a nearly empty train. This doesn't even seem like pricing optimized for profit, let alone general carbon reduction across all modes of transport.
The pricing, timetables and services were basically all centrally controlled. The train operators put in requests to change them, then the government (or Network Rail or whoever they delegated the power to) would approve/deny them. Since they're sharing access to the underlying infrastructure, they can't do what they want.
Additionally, there are many trains that are unprofitable but the operators are required by law to run them for public interest reasons. When they designed the franchising system, they tried to "balance" each franchise so it had roughly the same number of profitable and unprofitable routes.
The idea that our existing franchising system was anything remotely resembling free market is absurd. Then on top of that, because decision making and control was distributed amongst 40 or so different corporate entities, trying to make a change to the ticketing system was damn near impossible.
If anything, this new concession-like system will likely be more competitive, even if the government is acting as an intermediary.
As was the idea that this was even possible.
It was never set up to impose market discipline on the rail providers. It was set up to be a profitable investment. So Richard Branson could (as he put it), print money.
Under the rail franchise agreement during the bid process, the terms might mandate that journeys from A to B cost £10, and that journeys from B to C cost £5. There might be no obligation to price travel from A to D via B & C at a particular cost, so a rail operator might make that more expensive.
It’s silly from the consumer perspective, but it makes sense for the rail operators to do this in order to maximise profits.
There's no negotiations between the companies here, though they could ask the regulator (the ORR) to intervene if they believed the algorithm gave unfair results.
In real terms the main reason is to protect commuters and capacity - as a theoretical made-up example during the bid process they might be aware that lots of people travel from Milton Keynes to London so the council wants to put a cap on increases for that route. They might also still have lots of capacity on the route from Banbury to Milton Keynes on a morning, so that is priced much lower than Banbury to London. So if you go Banbury -> Milton Keynes you might get a “discount” for not a lot of people using the line, and travelling from Milton Keynes -> London you get the rate that’s protected for commuters, but if you go between Banbury and London then part of the route is already at capacity and there is less discount on the rate.