Whether those changes would have been made free if the original code was GPL-licensed is highly debatable, and there are plenty of examples on both sides. Notably, there was more or less no way MacOSX would have been built on GPL code. Conversely, many router projects have been forced to release source code due to being based on GPL-licensed Linux code.
In that regard I think even GPL advocates would be quite happy to use your example about Darwin, at least as a cautionary tale (though perhaps there was more benefit for the BSD upstreams than I am aware of; we're still waiting for an open release of ARM Darwin...). Likewise Apple's adoption of KJS/KHTML didn't go as well for KDE as we had hoped.
When a library is under BSD, for example, and is compiled into a program statically, it does not matter at all whether the developer using it was kind enough to provide the source code to all of their modifications: from the perspective of the user of the binary the freedom to change the software they are using has been lost, even if they are a developer. This is what the LGPL guarantees: not directly that the source code is available (though it does), but that any binary distribution that uses the library also provides the means required to replace the binary of the code with a modified version based on the source.
Moving to the full GPL, what is gained is the "viral" guarantee that this ability to get the source of a binary is maintained as people attempt to use the benefits of the licensed work in a larger binary. The license, philosophically and legally, is not about a property of the source code: it is a property of the binary being distributed; the result is that people who rely on the works of others in their binaries have to make the entire binary open and modifiable to end users. That's what makes GPL so interesting: it is a license about binaries and what the rules are surrounding how people can modify those binaries, and is only tangentially related to source code as the means by which these binaries will be modified.
Of course, as TiVo taught mainstream, there is a loophole: you can comply with GPL2 fully and yet users are able to lose these guaranteed freedoms if the binary software is verified by some kind of hardware lock: Android being open source is academically interesting, but provides no value to the owner of a device with a locked bootloader; GPL2 required the binaries to relate to source code in ways that failed to take into account cryptographic verification by unmodifiable firmware. This is why the GPL3 exists: to provide this freedom of modification to the binaries sitting on my hardware devices, and effectively moves the freedoms being guaranteed from the "binaries" all the way to the product itself.
At the same time, users of the fork still benefit from the original code that exists in the new project, as there is no need to rewrite it. That effort can then be made to make the new project better. There may not be as much of a benefit as there would be if the fork was also open source, but there's clearly still some benefit if users find it useful enough to use it.
Now, an argument can be made that the ability to fork the code may prevent people from contributing to the original project. In practice, maintaining the diffs for any reasonably active project is a lot of work, and it makes sense to contribute as many changes back as possible, leaving out only the 'secret sauce' bits, just to reduce the amount of work necessary to maintain the fork. Those proprietary bits might have made it into the original project if it were using a copyleft license, but they also might not have been made at all.
To the user of this binary, your arguments about potential benefits to an upstream open source project are thereby off-topic (especially if the source code is fully available and no one is hiding any modifications) and your discussion of "forks" is downright nonsensical: given 100% of the source code of the software running on a TiVo, you still have no freedom at all to actually modify that software. The system is "open" in some weird, mostly academic fashion, in that you can easily find out how it works, and maybe you could even build a TiVo competitor based on it, but to the owner of a TiVo--an actual user of the distributed binaries--the source code is worthless. The GPL's purpose is to guarantee that a user of a binary will always have the freedom to change the functionality of that binary in the context of their use.
This is a fundamental freedom of the user that requires the developer to give up some freedoms with regards to how they deploy their binaries and has nothing to do with projects and forks or even source code in any direct way. This is a trade off related to a subjective understanding of "freedom" as it applies to different actors in the system, and so to understand why the GPL is interesting you have to be willing to temporarily take off the hat of a project maintainer and put on the hat of a user, as the GPL's purpose (and yes, this is somewhat sadly ironic) is to protect the rights of a different player in the ecosystem than the maintainer whose job it is to choose a license for their project.
The GPL doesn't guarantee that a potential derivative work would be open source. The developer that might have based their work on open source might choose not to use the code of the original work in their project so as to avoid having to make their own work open source as well. This is a loss to the world, as the quality of their work would most likely not be as good, either because the code wouldn't be community reviewed or simply because time was spent writing code with functionality that already exists as open source taking away time from other features.
GPL or BSD, somewhere along the line, the developer has to willingly make their work open source. The only case the GPL can force someone to make their work open source is if someone has used GPL code with the intention of violating the license, but are later exposed to have used GPL code and are shamed into open sourcing it.
At the end of the day, the user only has freedoms to the code that the developer chooses to grant. Whether this is by choosing to use GPL code in their project and making it GPL as well, using BSD code in their project and making it BSD as well, or not using GPL code (but possibly BSD code) and not making their work open source.