PCIe signals are generated by transceivers -- devices within chips that are specialized in signal conditioning e.g echo cancelling, emphasis/de-emphasis, dynamic impedance matching. These transceivers and the analog and digital techniques they implement get better with time. This is easily measurable by looking at the Bit Error Rate of data or by looking at eye diagrams (see slide 15). As data rates increase things like drive strengths, impedance mismatches, and a number of other properties of silicon will "close the eye" meaning the transmitted "0"s and "1"s are not different enough for them to be distinguished by a receiver enough of the time to successfully decode a packet. (PCIe is packet based, it's surprisingly somewhat similar to Ethernet). But essentially as our understanding and processes for manufacturing semiconductor devices increase, we're able to "open the eye" more, at which point the industry decides to increase data rates.
It helps that this isn't happening just for PCIe; there's lots of breakthroughs that benefit (and may have originated with) other high speed links.
Optical PCIe would be hugely handicapped by lack of a standard optical PCB construction method. You'd have to print waveguides onto the PCB. And then it stops working if you get dust in the socket.
There are advantages of optics including that light moves faster than electrons (important for HPC where the figure of merit is latency in us between nodes, etc) and typically has higher fidelity. But the size of these structures is orders of magnitude larger than conventional semiconductors.
Multimedia is the driving force behind increased data usage, and I think we'll continue to need more throughput until we no longer get any benefits from higher resolutions (aka when we have substantially more pixels than rods and cones in our eyes). At the moment a phone with a 4K display saturates your eyes at any distance greater than 2 feet from your face. I think a 16x PCIe 4.0 link will likely provide more than enough bandwidth to generate fully immersive VR experiences, so the question then becomes... why and when will we need optical PCIe 5.0 to quadruple the datarate of PCIe 4.0...
Both for power consumption and EMC reasons you want to minimize the maximum slew rate of the signal on the link (and thus the voltage) while on the same hand you need the voltage to stay large enough so that the receiver (which is for all purposes an analog design) can be implemented in widespread digital CMOS processes.
On the other hand both conventional parallel PCI and conventional (<=2.0) USB is limited by physical factors, which is in both cases the physical length of the link/bus and propagation velocity of the used wires (ie. speed of light divided by some small-ish constant). In both these standards this limit was intentionally introduced by Intel as cost reducing measure (in PCI's case this means that motherboard does not have to contain about 60 or so discrete resistors, real impact on cost of USB's implementation is somewhat questionable).
The electrical level isn't too magic, but there are still a lot of things that have to be tuned (the chapter on tuning in the Mindshare book on PCIe is about 100pp). For a commodity consumer bus, the relative reliability and speed of PCIe is kind of a miracle.
i'd guess the speed improvements come from much more precise timing and voltages, so you can get better guarantees about interference. if voltage is +/-10%, that field will be bigger, and interfere more. If the timings are +/-10% the field will be there when you don't want it to for the next signal.
Anyway, i'm sure there are much more knowledgeable people who can give you much better insight, but i think that's the physics 101 kinda answer.
Imagine people going to the shopping mall. Once in a while someone goes in and an hour or two later they come out the other door 10m away. But they travelled a lot more than 10m. You just didn't notice from outside.
My memory may be off but as I recall the fermi speed in copper is only around 0.5% of c, rather far off. What does propagate at speeds on the order of c is the EM field, which is what most people are actually talking about when they think of electricity moving down a wire. But, it's still something around ~60% of c in a copper transmission line.
What goes at nearly the speed of light is the "message" that electrons should move a certain way. If you want an analogy, if you blow into a flute, even though the air is moving slowly, the sound travels fast.
I'm not sure I understand what you're saying it almost sounds like you're referring to impedance.
signals propagate over the transmission line as a wave by alternatively transferring energy from electric to the magnetic fields i.e. between L & C.
which is where the delay comes from...
it is fairly trivial to derive the wave propagation equation for the above model (assuming ofcourse that leakage conductance is zero). when considering lossy transmission lines though, things get quite complicated, but you can always (almost) get away with numerical techniques...
Somewhat related, I've noticed comm tech tends to follow a fairly consistent evolution: new enabling material/process, improvements to interconnect, algorithm optimizations, repeat.
problems are how to take care of the electromagnetic noise, that's why PCIe uses differential pairs wiring (both positive and negative wires are next to each other on board) instead of single-ended (single wire with common ground, negative) which were used in original PCI and PCI-X so the if some noise hits the first wire, same noise hits the second differential wire.
improvements in coding and decoding the signals with error correction also help recovering any errors caused by the electromagnetic noise.