Reducing AOT SWS02 Data in IA³

NOTE: This document is subject to change due to ongoing IA/ISAP software development.

To begin: Start your IA session by typing ia at the system prompt. Then follow the instructions below.

1) cal_select (help: IA UTILITIES)

This tool is used to select calibration files. See Appendix B.4 of the IA User's Manual for descriptions of each calibration file. The cal_select command opens a window which lists all current calibration files, and there are two possible settings for each CAL file: 'BASE' or 'TEST'. The choice will depend on the current status of the calibration files, so ask an IA guru.

It is possible to load a cal_select.dat file (a binary file saved out of a previous run of cal_select) by clicking the 'load' button. NOTE: if using the 'load' option, the 'path' box must be filled in or cal_select will crash.

After either setting the appropriate CAL files manually or loading your calibration file, click 'OK' to exit the cal_select tool.

2) erd=read_ferd('xxxYYYzz') (help: IA INPUT OUTPUT)

This command reads in the ERD from the raw data files. The 8 digits in parentheses refer to the revolution number (xxx), target number (YYY), and observation sequence number (zz) from the archive listing. You may also use the command "read_fspd" to read in the SPD directly and skip to step 5; however, the pipeline reduction is ever-changing, and starting with the ERD allows you to take advantage of the most recent version of the reduction software.

3) spd=dspd(erd) (help: IA DERIVE SPD)

This command converts the ERD to an SPD, using the following CAL files: (1; 2; 2a,b; 3; 4; 5; 6; 12; 16a,b,c,e; 18; 23; 24).

4) spd_orig=read_fspd('SWSPxxxYYYzz.FITS')  (help: IA INPUT OUTPUT)

		 spd.header=spd_orig.header 
		 save,spd,file='spd.save',/xdr 

Unfortunately, the full header is not properly saved by dspd; the two commands in this step correct this problem by reading in the SPD (spd_orig) directly and copying its header into your recently derived SPD (spd). The filename given to read_fspd is in the format of files coming directly from the archive, where 'xxxYYYzz' are the same digits entered in step 2 above. The third command saves the SPD into an IDL save set, which can be restored in later sessions of IA if necessary. In that case, typing

restore,'spd.save' (help: ROUTINES)

will restore the IDL structure spd to your IA session.

NOTE: An SPD is rather large, and the use of the option /xdr makes the saving process significantly faster. The /xdr (for eXternal Data Representation) option is not necessary to save smaller quantities such as the sctbl or aotlr alone (e.g., step 6), but it is needed if you wish to export any save sets between VMS and UNIX machines (see the IDL help file on "save" for more information.)

5) list_header,spd    (help: IA UTILITIES)

		 set_plot,'ps 
		 device,file='aper_plot.ps',/land

		 aper_plot,spd,'1A',60.   (help:IA UTILITIES)

		 aper_plot,spd,'3A',60. 
		 aper_plot,spd,'3E',60. 
		 aper_plot,spd,'4',60. 
		 device,/close 
		 set_plot,'x

These tasks give you more information about your dataset. The list_header command lists the FITS header, and in particular, you should note the position angle of your observations: the real position angle is (roll angle) + 90 degrees. The remaining commands in this step create a postscript file containing plots of the four SWS apertures on the sky (for bands 1 & 2; 3ACD; 3E; and 4). The value of 60. above is the declination scale of the plot in arcseconds. Consult your local system manager for printer options to determine how to print the postscript file locally.

6) sctbl=make_sctbl(spd) (help: IA UTILITIES) 
		 print_sctbl,sctbl,file='sctbl.txt',/spawn 
		 save,sctbl,file='sctbl.save' 

These commands create, print out, and save the scan table, which gives such information as the sequence of observations (darks, photometric checks, and science scans) and wavelengths covered by each band. The print_sctbl command prints the scan table to the screen as well as to the specified file name. The /spawn option is a local MPE command to print the specified text file to the default printer; again, consult your local system manager for printer options.

7) spd_mem=antimem(spd) (help:IA AUTO ANALYSIS)

This routine is designed for reducing memory effects, as is done in the standard pipeline. The use of antimem is mostly historical, but should be kept in the reduction chain because this task may in the future include a more complicated method for dealing with memory effects.

8) spd_D=dark_inter(spd,sctbl,/multia)  (help:IA UTILITIES)

		 save,spd_D,file='spd_D.save',/xdr

NOTE: Usually the server or window system is configured to retain plot window after scrolling or the start of a screen saver. The /multia option is only needed if the draw window is not retained (as on a Multia terminal); it causes IDL to handle the plot storage directly.

The dark_inter tool facilitates dark current correction. It can be the most time-consuming of all data reduction steps, but is also one of the most important in producing reliable spectra. Make sure to save the SPD after completing this task. Before you begin, you should note the sequence of darks and other scans in the scan table. Note that the reset time ("tr") and gain columns in the scan table must be the same for a dark and the scan from which it is being subtracted; dark_inter automatically makes available all darks with matching tr and gain. Also check the scan table to determine which (band+scan) combinations hold actual data: those which do not will show "0.000" for both wave_min and wave_max (this information is also shown in the message box in dark_inter as "w_range"). It is not necessary to dark subtract these (band+scan) combinations. For each of the four bands (starting at detectors 1 [SW1:], 13 [SW2:], 25 [LW1:], and 37 [LW2:]), the following sequence should be followed:

A.Select a detector. This detector will represent the whole band.

B.Revise Darks.  Click "Revise Darks". In the "Revise Darks" window which pops up:

 
		$\parbox{5.7in}{$\rightarrow$ ~Select the first set of darks. This ma...
...t'' button first and then clicking the dark you want in the middle left panel.}$

 
		 $\rightarrow$ choose clipping options-here are the most general suggestions: 
 
		 		$\parbox{5.1in}{$\triangleright$\space first 1 point and clipping(mean) with kappa=3 (i.e., 3$\sigma$\space clipping around the mean).}$

 




				$\parbox{5.1in}{$\triangleright$ After strong photometric checks, you...
...emory effects (i.e., jump in signal followedby a decline in signal over time).}$

  
		     $\rightarrow$ refine dark to be subtracted for each detector of the band, for each scan: 
 




				$\parbox{5.1in}{$\triangleright$ ''clip'' shows the points to be clip...
...mits youhave specified, and all points above or below these lines are clipped.}$

 




				$\parbox{5.1in}{$\triangleright$ ''apply to whole band'' shows the effect of the current choice for all detectors inthe band.}$

    




				$\parbox{5.1in}{$\triangleright$ many iterations may be tried (use ''...
...''undo'' button in between, results in the first detector being clipped twice.}$







		$\parbox{5.7in}{$\rightarrow$ ~Click on ''next'' to select the next d...
... dark, youwill get an error message in the text window reminding you to do so.}$

 
		$\parbox{5.7in}{$\rightarrow$ ~Click ''done'' in the ''Revise Darks'' window when all darks in the band have been revised.}$

 







C. Subtract darks from photometric checks and science scans: 
 




		$\parbox{5.7in}{$\rightarrow$ ~Select the photometric check or scienc...
...en duringthe entire use of dark\_inter, so that you may readjust as necessary.}$

 




 		      $\rightarrow$ Choose the dark subtraction method: click on "Method". 
 




				$\parbox{5.1in}{$\triangleright$ In the ''method'' menu, choose which...
...us'' and ''next'' buttons, and click ''linear interpolation'' for the method. }$

  
				$\parbox{5.1in}{$\triangleright$ When you have successfully marked th...
...and toggle your dark choices until they appear in redin the left plot windows.}$

 




				$\parbox{5.1in}{$\triangleright$\space Choosing other darks: If you w...
...the dark points are colored yellow-orange again to besure you were successful.}$

 




				$\parbox{5.1in}{$\triangleright$ If the available darks do not provid...
...tector individually, and is much more time consuming; use only whennecessary. }$

 




				$\triangleright$
Click "Done" in the Method menu.

 
		$\rightarrow$ Subtract fitted dark current. 
 
				$\parbox{5.1in}{$\triangleright$ Click on ''fit'', which willshow the fit for the selected detector in the bottom left plot.}$

 
				$\parbox{5.1in}{$\triangleright$ If the method selected is ''mean'' o...
...ll detectors in this band/scan combination,you have completed the subtraction.}$

  
				$\parbox{5.1in}{$\triangleright$ However, often ''jumps'' are seen in...
...rites a new solution for that detector, and doesn't affect theother detectors.}$

   




				$\parbox{5.1in}{$\triangleright$ If the method selected is ''spline''...
...r left plot. Repeat the ''fit'' step for each individual detector in the band.}$

 
				$\parbox{5.1in}{$\triangleright$ Additional notes for spline fits: Pr...
...ed special plot parameters to get the selected scan to fit in the leftwindows.}$

  
		$\parbox{5.7in}{$\rightarrow$ ~Repeat Step C for each photometric check and science scan in this band.}$

D. Set slider on top to the first detector number of the next band and repeat the process, starting with Step B, Revise Darks. Complete Steps B and C for all four bands. It is a good idea to save the data set periodically during the use of dark_inter, to prevent loss of efforts in the event that the tool crashes. The easiest way to do this is to use the "save current" button in the dark_inter tool, which creates a file called "dark_inter_save.xdr" containing the current data file. This file is overwritten in successive uses of this button, so you may make incremental saves without using additional disk space.

9) spd_DM=mask_inter(spd_D,sctbl)  (help: IA UTILITIES)

		 save,spd_DM,file='spd_DM.save',/xdr 

This command puts you into the mask_inter tool, which can be used to inspect your SPD and mask out bad data. Sometimes a whole detector is significantly noisier than the others in the band, or there may be significant memory effects after a strong glitch. In addition, you can mask out the jump segments first detected with dark_inter (see step 8C). See the on-line help for this tool for a detailed account of all the utilities of this tool.

Enter a band number in the "band" box, click on the scan number you wish to see, and click "disp_spd" to display a subset of the data. Note that to examine a different scan, you must click on the previous scan number again to "unset" it; otherwise there will be two scans selected and the tool will complain (multiple scans can be selected for producing an AAR within mask_inter, but only one scan can be displayed at a time). If you click the ``plotmode'' button, a background window will pop up in which you can set the plot scale; we recommend that you choose a fixed y-scale, as this is a good way of finding detectors for which dark subtraction didn't work properly, or for finding noisy detectors or other strange features. Each time you change the plot scale, click "disp_spd" again to display the data with the new plot values.

To mask out a specific section of a detector, click "select" and the detector of interest, which will cause the data from that detector to show in the bottom left window. Then click "mask" and use the left mouse button to select the region to be masked out (e.g., in the case of a memory effect). If you see the same glitch in all 12 detectors of a band, you can use the "do to all" button after masking out the glitch in one of the detectors. If you judge the input from an entire detector to be bad, you can click on the detector number in the list to the right-hand side of the display, and then click "faulty" to set it all bad.

IMPORTANT NOTE: Using the "faulty" button to set a detector "bad" is NOT reversible at the current time, and sets the specified detector bad for all scans. Setting the detector bad by masking all points out using the "mask" button IS reversible (using "unmask"), and only sets the detector bad for the specified scan.

Saving the masked SPD should be considered if a significant amount of masking has been done.

10) spd_U=updown_inter(spd_DM,sctbl)    (help: IA UTILITIES)

		 save,spd_U,file='spd_U.save',/xdr

This tool corrects for differences between the 'up' and 'down' scans of the instrument. Observations are taken in pairs of "up" and "down" scans, where wavelength decreases with increasing time in the "up" scan, and increases with time in the "down" scan. Due to memory effects, the shapes of the ``up'' and ``down'' scan may be different. The updown_inter tool takes a smoothed version of each of the up and down scans separately, and determines a "gain" or "offset" correction to make the up and down scans consistent with each other.

When you start the tool and select a detector and scan number, you will see to the left side four plots: the section of the stripchart that holds your scan, the up and down scans overplotted, the difference between the up and down scans, and the stdev of the difference. It is recommended to do the updown correction on a detector in the middle of the band, for if there is strong signal it will show up in the middle of this detector, and the effects of the updown correction will be easier to interpret. The recommended settings are the defaults: use the "fit" method, correcting "both" sections, using the "offset" correction mode, and "exclude" bad points. Choose kappa=2-3, and use the default "linear ramp" function for fitting. The "gain" option is really only useful for observations with very strong signal, where the uncertainties due to dark subtraction can be neglected with respect to gain effects. In addition, other fit functions may be used, though this is generally not advised. For example, ``Fouks (SPIE)''<sup>2</sup>and ``PHT-IST26'' are functions which are found to fit transients of infrared detectors, but they are not tuned to the SWS detectors and make no claim on reproducing a physical model of them.

At this point, you only need to click "Do Fit" to see the fit; the third plot >from the top will then overplot the fit on the "difference up-down" plot. You may iterate on choosing the correction mode, kappa, etc. and doing the fit if the solution is not satisfactory. After finding an acceptable updown correction, click on "Update full array" to perform the up-down correction on all detectors in the band for that scan. The results for all 12 detectors in the band are shown to the right of the menu panel.

Repeat this process for all scans in each band, and for all bands where you have data. NOTE: updown_inter should not be used to correct band 4 (LW2:) for very faint sources, where glitch tails can dominate the solution. Consult your IA guru for more information.

11) plot_phot,spd_U,sctbl,[1],13,/all   (help:IA UTILITIES)

		 plot_phot,spd_U,sctbl,[1],25,/all

		 plot_phot,spd_U,sctbl,[1],37,/all

The routine plot_phot can be used to look at photometric check data. The command lines given above request IA to plot the first photometric check for all 12 detectors of bands 2, 3, and 4 (which start with detector numbers 13, 25, and 37); the band 1 photometric check is not used. If you have more than one photometric check, you can replace the [1] above with a range of values (e.g. [1,3]) and compare them on the same plot. By specifying a particular detector and removing the /all flag, you can look at the photometric checks for that detector.

Data points already flagged as bad by the system will be marked (see help file for color code). If these are the only bad points and the photometric checks look sensible, the standard routines will handle the bad points appropriately. Note that the default is to always use the first photometric check, so if it is corrupted it should be replaced with another. If you suspect your photometric data has problems which are severe enough to affect the ultimate calibration, such as a corrupted first phot. check or large spikes not already flagged, consult an IA guru.

12) spd_U=respcal(spd_U)  (help:IAAUTO ANALYSIS)

		 spd_U=fluxcon(spd_U) 
		 spd_U=velcor(spd_U) 
 

The routines respcal and fluxcon apply various calibration files; respcal uses CAL files 13, 19, and 25; fluxcon uses CAL file 13. The output of fluxcon looks like the following:

Warning(SPCP): some phot. check data might be doubtful       Det:26 Uv: 2
Warning(SPCP): some phot. check data might be doubtful       Det:39 Uv: 2
Warning(SPCP): some phot. check data might be doubtful       Det:46 Uv: 3
Warning(SPCP): some phot. check data might be doubtful       Det:48 Uv: 2

If the number following "Uv:" (the number of outliers) is greater than 3, the photometric check data for that detector indicated may be suspect. The task velcor performs a correction to the heliocentric frame based on the velocity of the satellite.

13) spd_F=flat_inter(spd_U,sctbl)  (help: IA UTILITIES)

		 save,spd_F,file='spd_F.save',/xdr 

The first command starts up the flatfield correction tool flat_inter. This task corrects the flatfield of each detector in the band with respect to the others. As with "updown_inter", the correction can be a "gain" or "offset" correction, and is made on the basis of smoothed data. It is generally recommended to use the "median" filter, and to exclude poor points. As with updown_inter, you should make sure that the fit you have chosen does not introduce spurious features, and vary the smoothing or correction parameters accordingly. The default values are Res1=500 and Res2=100; try lower values, especially for band 4 (LW2:) for very faint sources, and larger values may be useful for very bright sources. In general, it is a good idea to keep the relation Res1$\sim$5 Res2. As before, the 'gain' option should only be used with strong signal. NOTE: be sure to press "return" after changing each entry; otherwise, strange effects may result if new values are not registered by the tool.

14) spd_FM=mask_inter(spd_FM,sctbl)  (help: IA UTILITIES)

		 save,spd_FM,file='spd_FM.save',/xdr 

Now that all calibrations have been applied, you can go back into the interactive masking tool used in step 9 to mask out noise spikes or outliers; for instance, a signal which is clearly spurious, such as a strong spike detected only in the up or down scan of all 12 detectors, will bias the resulting spectrum. This masking step may be best done after following the rest of the steps described below and looking at an unmasked version of the output spectrum to see where lines have been detected, and returning to this step to produce a refined dataset.

Click the "w-mode" button to plot in wavelength units, and specify the wavelength range of interest in the ``plotmode'' pop-up window. You can use the ``fluxclip'' button to mask out outlying data by hand. When you click this button, the data from all detectors is plotted in the lower left hand window, and you can define a series of boxes with the left mouse button; the points inside these boxes are then clipped. Clicking the right mouse button returns you to the main mask_inter tool. If you want to remove an unwanted glitch (glitches are marked with yellow asterisks), click "select" to specify the detector, and if needed click "zoom" and use the cursor to select the section of the scan holding the offending point (both up and down scans will be shown, with wavelength units along the upper x axis of the plot). Then click "unglitch", and click the mouse cursor on either side of the glitched point. The unglitched point will show up as a white cross, like the rest of the good data.

You can see the effects of your masking by using the "extract_aar" and "aar_f" buttons, which will produce a filtered AAR (with the resolution specified in the "resol" window) of the specific section of spectrum that you have selected with band, scan, and x_range. It is recommended to use kappa=2.5-3 and resol>2000, especially for faint sources. Further, you can output your final, masked AAR result to a file by clicking on the "save_aar" button. This action will pop up a window where you can specify the name of the file you wish to write, and what format: XDR, IDL SAVE, or FITS. The first two options both write IDL save files which can be restored in an IA session (the first using the XDR format described in step 4). Repeat for each line.

15) spd_FM=killwait(spd_FM)  (help: IA UTILITIES)

		 aar=extract_aar(spd_FM) (help: IA AUTO ANALYSIS) 
		 save,aar,file='aar.save',/xdr 
		 status=sap_wfits('aar.fits',aar)  (help: ISAP)

		 aar=shiftscans(aar)  (help: IA UTILITIES)

		 line=aarfilt(aar,[7.29,7.32],1200.,kappa=3.0,/noglitch)  

The first task flags "wait" points as "bad"; it is useful on SWS01, SWS02, and SWS06 data. The next task extracts an AAR from the reduced SPD, and the following two commands save the AAR as an IDL save set and a FITS file. Note that all the masked and glitched points are still in the AAR, but are ignored by shiftscans and aarfilt. If you have an SWS02 dataset which has multiple scans on the same line, you should use the routine ``shiftscans''. This command does an additional flatfielding of common scans with respect to each other; it is an ``offset'' mode correction and is particularly valuable for faint sources. This task is not necessary if the wavelength ranges of your scans do not overlap.

The last task produces a final, filtered AAR for a particular SWS02 scan; you should repeat this step for each individual line scan. In this example, the wavelength range is 7.29-7.32 microns, the resolution is 1200, and kappa is set to 3 (3$\sigma$). Save the individual filtered files as the whole AAR was saved above. If you have faulty detectors that were not masked out before, you can do so with this command as well-see the help file.

After this step, the basic reduction of SWS02 data is complete, but there is likely significant fringing, especially for lines lying in bands 2 and 3. These can be difficult to deal with for SWS02 data, as there are only very short baselines over which to measure the fringes. There are different tools you can use to try to deal with fringes, either by trying to remove the fringes or fitting a line shape; these include linefit (in ISAP), aarfringe, or multifit. Consult your IA guru for strategies for recovering as much information as possible.

Finishing up:

Type ``help'' to look at the list of IDL structures you have generated in this data reduction section. Make sure to save all files which were produced from the most time-consuming steps (i.e., any of the interactive tools) as IDL save sets and FITS files before exiting from IA. Then you can restore the SPD or AAR at any stage and redo some of the reduction if you find it necessary at a later date.