It's a lot less sophisticated than that. They take images in multiple filters. In the context of JWST of order 10 filters (sometimes more sometimes less). Source extraction is then performed on the images by essentially identifying bright spots and dropping an aperture (separating ones that are nearby and blended if possible). The standard tool for this is called source extractor. They then have catalogs of tens of thousands of sources per image and the next step is to figure out redshift. There is a lot of code to do this but the simplest methods require fitting templates of what we think galaxies look like to these catalogs. High redshift sources tend to "drop" out of filters at shorter wavelengths. This is because neutral hydrogen in the early universe essentially absorbs almost all of the light at shorter wavelengths than 1216 angstroms. So if a galaxy is at redshift 10, the flux should essentially be zero at all filters that cover wavelengths shorter than 1.33 microns. JWST has filters both bluer and redder than this wavelength so we see the source appear in the redder filters and not the bluer ones. This technique was pioneered in the mid 1990s. This gives an approximate redshift called a "photometric redshift". There are other features in a galaxy spectrum that can mimic this "dropout" so not all photometric redshifts are robust. Therefore one has to take a spectrum of the galaxy which was what was done in this paper to confirm that the dropout is in fact the absorption feature we think it is. In this particular case, the authors were skeptical early on because there is a source right next to the object that is at a redshift where one of these other spectral features can mimic absorption by neutral hydrogen (this feature is the Balmer break). In any case, it's really an impressive demonstration of the power of JWST.