I kinda get what you're coming from - back in college i had a lab where we had to make a NES-style game console from scratch on an FPGA - including peripherals and video output - which used VGA, that's how I got to familiarize myself with the technology. It was super simple, you just output the correct voltages to the RGB pins and sent the proper HSYNC and VSYNC signals, and that was it, worked like a charm as long as you didn't mess up too bad, the monitor figured out the rest.
Recently I dusted off this project and bought myself a new FPGA board, with a HDMI output. Getting that thing to work was a nightmare. Not only did I have to configure the HDMI transciever, I had to build proper EDID support, and get I2C to cooperate, and get the timing of the signals exactly right. And even then, out of the 5 screens I could get my hands on, only 2 worked, one of which would inexplicably go to sleep despite displaying video on the screen.
Getting HDMI working was more painful and took about 10x as long as the rest of the original project did, and even then, it was a partial success.