a few concepts you need to start: copyright is a right to exclude. copyright does not arise from your agreement. the copyright attaches to the protected expression.
analogies are dangerous, but these concepts are intuitive if you think in terms of real property.
right to exclude: you have the right to exclude most people, under most circumstances, from your house.
doesn't arise from agreement: you can let joe into your house without an agreement. you can also kick him out whenever you want, unless you and joe made an agreement that limits your right to exclude him (or a legal exception applies).
in our analogy, the agreement might be a lease. (btw, the lease will limit the owner's right to exclude, and it will probably give both sides some non-property--i.e., contractual--rights. telling the two kinds of rights apart is the subject of the article.)
there are exceptions, just like there are exceptions in copyright. you know the names of some of them, like fair use. similarly, there are rights for people who don't have a formal lease, and there are rights that exist even if a lease purports to reject them.
but what you call the "default position" is the same: the property owner has the right to exclude others.
exists wherever the property exists: the right to exclude someone from real property attaches to the land and stays with it. similarly, copyright attaches to the protected expression; the copyright owner's right to exclude is already attached to the copy you have.
now you can see why you don't get to do whatever you want with a copy if you don't have an agreement: the copyright already governs your copy. it lets the copyright owner exclude you. if you want protection against that exclusion, then you need to agree to a license.
that is, of course, a gross oversimplification. for example, you emphasize "copy", but the copyright owner has other rights, such as the right to prepare derivative works, the right to distribute the work, and the right of public display. the act of removing "the 'agree to the gpl' part of the installer" may have created a derivative work, and that in itself would been a copyright violation. on the other hand, just having a copy isn't a copyright violation because "having" isn't a protected right. details like that rarely matter though. i only point them out to illustrate that it's dangerous to make assumptions about what words mean or to think of the law as a battle of semantics.