Yes, the color of every pixel is ultimately determined by a shader program, but as you might expect it's more complicated then that.

There is what is referred to as a graphics pipeline consisting of a mix of fixed-function hardware stages and programmable stages. At a high level, it does the following: 1) the GPU accepts a set of 3D triangles from the CPU, 2) a 'vertex' shader program transforms (flattens) the 3D triangle vertices into 2D triangle vertices with pixel coordinates, 3) the GPU rasterizes the 2D triangles to determine exactly which pixels the triangles cover, 4) a 'pixel' shader program is run for each covered pixel to determine the color of the pixel, 5) the resulting pixel color is stored in a frame buffer (which may involve blending it with the existing color). This 'pipeline' is then repeated many times (with different triangle meshes and shaders) until the whole frame is drawn.

Hope that helps!