Both. MiSTer can output the original signal through the VGA output or through the HDMI digital output (e.g. and using a HDMI to VGA converter), and also it is posible to configure the HDMI output as a scaler (with different modes, from the lowest latency modes using one line as buffer, to the more complex changing the frequency and/or adding low-latency CRT/LCD/etc filters). Effects can be added to the analog output, e.g. scan doubler. Please consider checking the documentation, it is an amazing project.