The OSIRIS Data Factory shall allow processing, reduction, archiving and quality assessment of scientific data during the observations and shall provide reduced data ready for analysis and further specific processing.
The main purposes of the reduced data originated by this system are to asses data quality and to provide with a preliminary view of astronomical data. It is not intended to be a final data reduction pipeline and the products should not be considered as scientific data. In this sense, the system will not provide with flux calibrated data, as this is a specific task to be performed by the astronomer. It will provide, however, with flat-field corrected frames and, in most observing modes, with wavelength calibrated spectra or images. The system will perform some instrument specific tasks such as TF calibrations and tests which will be used mostly during observations. Other scientific tasks, specific to this instrument, are the multi-slit mask designer and the exposure time calculator. These are not covered in this document. The OSIRIS Data Factory is integrated in the GCS Data Factory.
Definitions
Bias:
Pedestal level of (typically) several hundred ADUs (counts) which is added to the output signal. It may have spatial structure along the detector. A ZERO frame is obtained by just reading out the detector. The level of the ZERO frame is the BIAS. The noise in this frame is the READOUT NOISE.
Dark current:
On some CCD´s there is a non-negligible amount of background added, mainly due to thermal effects. This is an additive signal, normally proportional to exposure time. Nowadays CCD´s have almost no dark current (less than 1 e- /pixel/hour). The behaviour of the dark current can be measured by taking a series of frames at different exposures times and without light.
Flat-field frame:
Flat-field frame is an image of a uniformly illuminated field. The purpose of this frame is to correct from the different quantum efficiencies of the pixels across the detector. There are two main types of flat-fields: sky flats, which are obtained by imaging the sky at twilight, and dome flats, which are obtained by imaging a uniformly illuminated part of the dome (by a dedicated lamp). Quantum efficiency depends on wavelength, therefore flats should be taken for all the observing bands. Sky flats are preferred to dome flats since the spectrum of the lamps used to illuminate the dome is (normally) very different from the sky spectrum. In TF narrow-band mode, however, dome flats are needed, since sky flatfields contain Fraunhofer lines which generate fringe structure not present in science data.
In spectroscopic modes there are two types of flat-field frames that are needed to correct the different quantum efficiency of the pixel and the differential illumination across the slit (in long-slit modes). To correct from pixel to pixel variations, a calibration lamp is used. This lamp should have a continuum, free of emission lines, spectrum. To correct from possible illumination gradients along the slit, a sky flat-field is recommended to guarantee that light is passing through the same optical elements as the target frames. It is important to have high signal-to-noise flats in order to minimise the extra noise added during flat-field correction.
Free Spectral Range:
Distance in angstroms between consecutive passbands or inter-order spacing in a Fabry-Perot etalon, such as a tunable filter.
Overscan:
Region of the detector of several columns (about 30) which are not exposed to light but are read out. It contains information about bias level and dark current.
Phase effect:
Position-dependent change in wavelength across the field, present in all Fabry-Perot interferometers. The centre of the interference pattern is coincident with the optical axis and the distance of an object from this point is its optical radius.
Piezo-electric transducer (PZT):
These permit the TF to scan through a range of spacings. There are three PZTs, each located around the plate edge separated by 120°.
Target or Science frame:
Frame containing science observations, including calibration stars.
Whitelight flatfield:
Flatfield image or set of images obtained by observing a white light source over the same range of etalon spacings used in the actual observations. The whitelight flatfield maps the response of the filter as a function of position and etalon spacing.
Scope
The OSIRIS Data Factory is part of the GTC Data Factory which is a GCS subsystem. This section provides an overview of the OSIRIS Data Factory, his environment and delimitation.
An automatic set of procedures should be available to process the data acquired from standard OSIRIS observing modes. At least (a) standard imaging, (b) long-slit spectroscopy, (c) charge shuffling imaging and (d) fast photometry observing modes should be possible to process using the pipeline. The processing is done in two phases: a pre-process and a post-process. The pre-processing procedures include standard CCD operations: bias subtractiondark current correctionflat-field correctioncosmic-ray events removalPost-processing of data depends of the observing mode and will include wavelength calibration and geometric distortion correction. Required calibration frames will be taken either from existing databases or acquired frames during observations.
During observations, images will be displayed after acquisition. It will be possible to use basic tools to analyse the images and to perform statistics and profile measurements. Users may need to perform a quick reduction for fast scientific assessment of the data. The OSIRIS Data Factory is completely included in the GCS Data Factory subsystem at the Operations Co-ordination level. The GCS hardware and software standards will be used in the design and implementation of the OSIRIS Data Factory.
Related Use Cases
Some calibration data are needed to process science frames. . All frames, both science and calibration, are processed in the right way to finally obtain reduced scientific data.
There are two types of processing: Calibration frames processing:
Calibration frames (zero, flat-fields, arc lamps, TF arcs, etc.) are processed before they could be used to process science (target) frames. These calibration data are combined to produce flat-field frames adequate for each observing mode. Other necessary parameters are also obtained after this processing such as wavelength calibration solutions and TF parameters. The processed calibration frames are then validated and are ready to be archived by the "Operation Repository".
Science (target) frames processing: Using the calibration frames and parameters already processed, target frames are pre and post processed. These processes include bias subtraction, flat-field correction, cleaning of cosmic-ray events and cosmetics (pre-processing), and geometric distortion correction and wavelength calibration. Post processing tasks are observation-type dependent.
Description of the Use Cases:
Process Zero Frame This use case allows to obtain the zero level and the readout noise of the detector. All the zero frames are combined in such a way that high-sigma points are rejected (cosmic-rays or defects). Then the mean over the resulting frame is computed.
Build Imaging Flat. This use case allows to built a flat-field frame suitable for imaging observing mode.
Build Spectrum Lamp Flat. This use case allows to build a flat-field frame suitable for spectroscopic observing mode. To correct from pixel-to-pixel variations of the efficiency a uniformly illuminated image is needed. A continuum lamp provides an intense source of light. However, the spectrum of the lamp is not flat, therefore, to build a suitable flat-field the spectrum of the lamp should be eliminated. The procedure is as follows: assume the intensity in the image is I(x, y) where x is the spatial direction (along slit) and y is the spectral direction. to remove the lamp spectrum compress into the y direction: v(y) = å x I(x, y) and fit a polynomial V(y), then obtain F(x,y) = I(x,y)/V(y)to remove possible differential illumination along the slit, compress into x direction: w(x) = åy F(x, y) and fit a polynomial W(x), then obtain Flat(x, y)=F(x, y)/W(x)Flat(x, y) contains only pixel-to-pixel response.A high signal-to-noise is needed to ensure that noise is not dominating Flat(x, y). This resulting flat-field frame is used by use case Apply Flat Field.
Build Spectrum Sky Flat. This use case allows to build a flat-field frame suitable for spectroscopic observing mode. The purpose of this flat-field frame is to correct from possible illumination gradients along the slit in long-slit observations. Since we want to correct the science frames, a sky flat using the same slit is recommended. Procedure is similar to that described in use case Build Spectrum Lamp Flat for the illumination part only. The resulting flat-field frame is used by use case Apply Flat Field.
Subtract Bias. This use case allows the subtraction of the zero level and, eventually, of the dark current from a frame. It also trims the frame. There are two alternative ways for subtracting the zero level from a frame. The user may be able to select which procedure to follow. Using the overscan region of the frame. There is no need of zero frames here. The procedure consists in subtracting the zero level (or bias level) by estimating it from the overscan region. To do that, the overscan columns (we assume that the overscan region consists of a number of columns, say 30) are averaged. This produces a one dimensional vector which is fitted to a zero order polynomial (i.e. a constant) but allowing for rejection of high-sigma points, which may appear due to cosmic rays or to cosmetic defects. This procedure provides the zero level to be subtracted to the frame. This is also the recommended procedure in case the detector has a high dark current.Using the averaged zero frame obtained from the zero frames processing. It is important to remark here that, in general, it is not advisable this method since this operation has the effect of increase the noise in the science frame. This is particularly important in spectroscopic and narrow-band readout-noise-dominated images.
Combine Frames. This use case allows to combine frames in different ways. This is an important task used in the process of calibrating data. The output is a single frame. Frames are combined to obtain an average or median frame. Original frames may be scaled or not to the same level. The scaling may be either multiplicative (i.e. individual frames are multiplied by the corresponding factor to that level) or additive (i.e. the corresponding constant is added to the individual frames to get the desired common level). When computing the average or median a clipping algorithm can be applied in order to reject outsiders from this average or median. This is useful to remove cosmic rays, defects, and even real sources.
The procedure is as follows:
Let mean be one of average or median, i.e. the type of final combined image. Let factor be one of scale or zero, i.e. the type of scaling to be applied to the individual frames before computing the final image. scale corresponds to a multiplicative factor and zero to an additive factor. Let rsigma the sigma clipping factor (above and below mean) for rejecting pixels. compute the average of each frame using a specified region of the frame: Iicompute the common level as the mean of Ii : levelcompute the factors to apply:if factor = scale then factori = level/Iiif factor = zero then factori = level - Iiapply offsets:if scale imagei (new)= imagei(old)´ factoriif zero imagei (new)= imagei(old) + factorifor each pixel compute mean and sigmacompute final image, using mean, rejecting pixels above and below ± rsigma times sigmaThis is a standard procedure to produce zero frames and flats. It can be also implemented weighting the individual frames by using for instance the exposure time or the means.
Determine TF Parameters. This use case allows to calculate the TF parameters needed for wavelength calibration in narrow-band imaging mode. A narrow-band arc image is needed. This arc frame is obtained by taking a long-slit charge-shuffling exposure of an arc lamp while changing the etalon gap scanning through 2 FSR (generated by use case Take TF Arc Frame from OSIRIS Sequencer, see [R.8]). See Description of use case Take TF Arc Frame.
The resulting image is a long-slit spectrum of the arc lamp where the spectral direction corresponds to gap spacing (z). The calculation of the TF parameters is as follows: Obtain pairs (l ,z) of (line centres, gap spacing) for at least 3 or 4 known lines. When obtaining the line centres, line widths are also obtained (dz). As the scanning is through 2 FSR, arc lines appear twice in the spectrum. The distance in units of z of repeated lines gives the FSRz.Fit to a line: z(l ) = a + blFrom this we obtain the TF parameters at wavelength lo:
| Order of interference | m = l ob/FSRz | (integer) |
| Plate spacing | Lo = mlo/2 | (Å) |
| FSR | FSR = l o/m | (Å) |
| Spectral band-pass | d l = d z FSR/FSRz | (Å) |
| Resolving power | R = l o/dl | |
| Effective finesse | Neff = FSR/dl |
Process Arc Frame. This use case allows to obtain the wavelength scale for long-slit spectroscopic observations. An arc frame is needed. This arc frame is obtained by taking exposures of a known set of emission lines (from an arc lamp). Wavelength calibration consists in obtaining a relation between pixels and wavelength, which is usually a low order polynomial (around 5). In long-slit observations it is usual to have spatial distortions in the sense that, perpendicular to the spectral direction, there are shifts in wavelength. This effect is corrected by this use case using the emission lines from the arc. The wavelength solution consists of coordinate maps of wavelength (l ) and slit position (S) as functions of original pixel coordinates X,Y: l (X,Y), S(X,Y).
Process MOS Arc Frame. This use case allows to obtain the wavelength scale for multi-slit spectroscopic observations. An arc frame is needed. This arc frame is obtained by taking exposures of a known set of emission lines (from an arc lamp). Wavelength calibration consists in obtaining a relation between pixels and wavelength, which is usually a low order polynomial (around 5).
In MOS observations, the arc frame is taken using the same mask as the targets. The resulting arc frame is just an image containing the same arc spectrum in each slit. The procedure of calibration is similar as in long-slit data, although here the spatial distortion is not relevant as the slits are usually small. The only additional step is the extraction of the individual arc spectra. To do that the information of the slits positions in the mask is used. This information has been generated to design and build the MOS mask.
Compute TF parallelism corrections. This use case allows to compute the necessary piezo-electric movements to ensure that etalon plates are parallel. A dedicated frame (generated by use case Test TF Parallelism from OSIRIS Sequencer, see [R.8]) is used. See Description of use case Test TF Parallelism. The image contains three charge-shuffled spectra corresponding to three quadrants of the CCD. One of the quadrants is taken as Reference and the other two are labelled X and Y according to their position relative to the Reference quadrant. Figure 1 shows an example of a frame obtained by use case Test TF Parallelism.
The
amount of tilt between plates is measured by the size of the offset between
the line through the X or Y and the Reference quadrant. The plates are
parallel when no offsets exist in either direction and when the emission
line appears simultaneously in all three. Figure 1 shows identical central
strips taken from a full charge-shuffle image. Upper panel shows a small
offset in Y only, implying that the plates are tilted only in the y-direction.
This offset is proportional to the compensatory offsets input to the piezo-electric
transducers (PZT).
It can be shown that the measured offset DZY is proportional to the applied offset DZY' which is the amount by which the plate needs correction. The proportionality factor is a function of the radius of the pupil-plane beam and the radius at which the PZTs are located. For the x-direction the relationship is the same. The process is iterative until the wavelength offsets between all quadrants are zero.
Compute Telescope Focus. This use case computes the best telescope focus from a frame containing a sequence of a star observed at different telescope focus positions. The stars are fitted to a gaussian in order to compute their FWHM. Since stars have been observed at different focus values, this provides with a set of pairs (focus, FWHM) which is fitted to a parabola. The minimum of this parabola is taken as the best telescope focus. The frame to perform this task is obtained from use case Focus Telescope (from OSIRIS Sequencer, see [R.8]).
Compute Camera Focus. This use case computes the camera focus from a frame obtained from use case Focus Camera. The method used is TBD.
Process Science Frame This use case executes a processing recipe for science frames. This is done in two phases: a pre-process and a post-process. Pre-processing:This consists in bias subtraction, flat-field correction and cleaning. This pre-processing applies to science frames from all the OSIRIS observing modes. The task only works when all the calibration data are available and have been previously processed by use case Process Calibration Data.
PreProcess Science Frame. This use case allows to pre-process a target frame. Pre-processing consists of (1) bias subtraction, (2) flat-field correction, and (3) cosmic-rays cleaning and cosmetics. A target frame is a frame containing images or spectra of astronomical objects.
Apply Flat Field. This use case allows flat-field correction for target frames.
Clean Frame. This use case allows to clean a target frame from cosmic-ray hits, dark pixels, hot pixels, and other cosmetic defects. Cosmic-ray detection is an automatic standard task and fine tuning of parameters depends on the CCD characteristics. Cosmetics defects are the same for a given detector. A frame containing information about the position of defects should be available. This mask frame can be created during commissioning.
PostProcess Science Frame. This use case allows for the final basic reduction, i.e. corrects from geometric distortions along the image and performs wavelength calibration for spectroscopic and narrow-band data.
Correct Distortion. This use case allows to correct a frame from eventual geometric distortions. A map of geometric distortions should be measured with high accuracy (during instrument commissioning and/or several times during the year) in order to perform the correction. This is to provide a direct image (broad or narrow band) with the correct angular scale and orientation. Basically it consists of applying a previously measured relation between CCD coordinates (x,y) and relative sky coordinates (Right Ascension and Declination). It is not an absolute astrometry task.
The geometric distortions map can be obtained in two ways. One is using a mask with evenly distributed holes and obtaining a direct image using a lamp. From the measured hole positions a relation between real and imaged positions can be obtained. Another procedure is using a field of stars with known accurate positions. A direct image of the field will provide the relation between real and imaged positions.
Remove Sky Rings. The phase change over the field gives rise to night-sky rings, which are broad diffuse circles of atmospheric OH emission lines on some images, centred on the optical axis (see Figure 2). They should be circular because the phase change makes for a specific off-axis angle where the interference equation is satisfied for a specific wavelength. In fact, they are slightly elliptical due to the misalignment of the CCD plane and to optical distortions within the camera. These sky rings are not corrected through flat-field division.
Figure 2.- Example of night-sky rings. Rings are off-centred since the optical centre is not at the image centre.
At low spectral resolution night-sky rings are large and diffuse and, in general, larger than the objects of interest. In such case, the method which provides best results is that described in Jones, Shopbell & Bland-Hawthorn (2001): A background map is created by median-filtering copies of the original frame, each one offset in a regular grid pattern from the other by a few pixels. The result is then smoothed and subtracted from the original, leaving little or no night-sky residuals.
Calibrate Longslit. This use case allows to perform wavelength calibration for a longslit spectrum. The coordinate maps l (X,Y), S(X,Y) created by use case Process Arc Frame, represent the desired output coordinates for the calibrated frame. These maps are inverted to obtain X(l ,S) and Y(l ,S) on an even sampled grid of l and S over the desired output image coordinates. When mapping to l , S coordinates, it is necessary to interpolate from the original (X,Y) coordinates since these are discretely sampled. There are several different ways of interpolating that can be selected by the user: nearest neighbour, bilinear, bicubic polynomial, splines, etc. Flux conservation must be taken into account by multiplying the interpolated output pixel value by the Jacobean of the l , S X,Y transformation (which is basically the ratio of pixel areas between output and input images).
Calibrate MOS. This use case allows to perform wavelength calibration for MOS observations and extract individual 1D spectra. Application of the wavelength solution is similar as in use case Calibrate Long Slit but in one dimension only. Extraction of 1D spectra is done using the information of the slits positions generated in the mask building process. For each slit position, the spectrum is projected onto the spatial direction. Over this projection, local maxima (which correspond to sources) and sky regions are found. Then, sky is subtracted and source spectra are extracted (by using an optimal extraction algorithm).
Analyse Data. This use case allows to interact with an image displayed on an image display device. Several operations are provided in order to inspect images and asses data quality, either of raw data or reduced data.
These operations include: Image statistics: mean, dispersion, mode, median, maximum, minimum and histogram computation, all computed on image sections.Magnitude calculation: selection of sources to compute instrumental magnitude, sky, noise and signal-to-noise ratio.Fitting: ability to perform two-dimensional fitting over selected sources; ability to perform one-dimensional fitting over selected sections (e.g. emission lines). Fitting functions may be: 1D and 2D gaussian, 1D and 2D lorentzian, polynomials.Offset calculation: distance and angle between two selected points over the image.
Last update July 20, 2005, by Héctor Castañeda