Reducing AOT SWS01 Data in IA³

¹

NOTE: This document is subject to change due to ongoing IA/ISAP software development.

To begin: Start your IA session by typing ia at the system prompt. Then follow the numbered instructions in this guide.

1) cal_select (help: IA UTILITIES)

This tool is used to select calibration files. See Appendix B.4 of the IA User's Manual for descriptions of each calibration file. The cal_select command opens a window which lists all current calibration files, and there are two possible settings for each CAL file: 'BASE' or 'TEST'. The choice will depend on the current status of the calibration files, so ask an IA guru.

It is possible to load a cal_select.dat file (a binary file saved out of a previous run of cal_select) by clicking the 'load' button. NOTE: if using the 'load' option, the 'path' box must be filled in or cal_select will crash.

After either setting the appropriate CAL files manually or loading your calibration file, click 'OK' to exit the cal_select tool.

2) erd=read_ferd('xxxYYYzz') (help: IA INPUT OUTPUT)

This command reads in the ERD from the raw data files. The 8 digits in parentheses refer to the revolution number (xxx), target number (YYY), and observation sequence number (zz) from the archive listing. You may also use the command "read_fspd" to read in the SPD directly and skip to step 5; however, the pipeline reduction is ever-changing, and starting with the ERD allows you to take advantage of the most recent version of the reduction software.

3) spd=dspd(erd,aotlr=aotlr) (help: IA DERIVE SPD)

This command converts the ERD to an SPD, and uses the following CAL files: (1; 2; 2a,b; 3; 4; 5; 6; 12; 16a,b,c,e; 18; 23; 24). The aotlr (grating position data) will be needed later in updown1_inter.

4) spd_orig=read_fspd('SWSPxxxYYYzz.FITS  ')   (help: IA INPUT OUTPUT)

		 spd.header=spd_orig.header

		 save,spd,aotlr,file='spd.save',/xdr 

Unfortunately, the full header is not properly saved by dspd; the two commands in this step correct this problem by reading in the SPD (spd_orig) and copying its header into your recently derived SPD (spd). The filename given to read_fspd is in the format of files coming directly from the archive, where 'xxxYYYzz' are the same digits entered in step 2 above. The third command saves the SPD and the aotlr into an IDL save set, which can be restored in later sessions of IA if necessary. In that case, typing

restore,'spd.save' (help: ROUTINES)

will restore the IDL structures spd and aotlr to your IA session.

NOTE: An SPD is rather large, and the use of the option /xdr makes the saving process significantly faster. The /xdr (for eXternal Data Representation) option is not necessary to save smaller quantities such as the sctbl or aotlr alone (e.g., step 6), but it is needed if you wish to export any save sets between VMS and UNIX machines (see the IDL help file on "save" for more information.)

5) list_header,spd    (help: IA UTILITIES)

 
		 set_plot,'ps 
		 device,file='aper_plot.ps',/land

		 aper_plot,spd,'1A',60.   (help:IA UTILITIES)

		 aper_plot,spd,'3A',60. 
		 aper_plot,spd,'3E',60. 
		 aper_plot,spd,'4',60.  
		 device,/close 
		 set_plot,'x

These tasks give you more information about your dataset. The list_header command lists the FITS header, and in particular, you should note the position angle of your observations: the real position angle is (roll angle) + 90 degrees. The remaining commands in this step create a postscript file containing plots of the four SWS apertures on the sky (for bands 1 & 2; 3ACD; 3E; and 4). The value of 60. above is the declination scale of the plot in arcseconds. Consult your local system manager for printer options to determine how to print the postscript file locally.

6) sctbl=make_sctbl(spd)    (help: IA UTILITIES)

		 print_sctbl,sctbl,file='sctbl.txt',/spawn 
		 save,sctbl,file='sctbl.save' 

These commands create, print out, and save the scan table, which gives such information as the sequence of observations (darks, photometric checks, and science scans) and wavelengths covered by each band. The print_sctbl command prints the scan table to the screen as well as to the specified file name. The /spawn option is a local MPE command to print the specified text file to the default printer; again, consult your local system manager for printer options.

7) spd=antimem(spd) (help:IA AUTO ANALYSIS)

This routine is designed for reducing memory effects, as is done in the standard pipeline. The use of antimem is mostly historical, but should be kept in the reduction chain because this task may in the future include a more complicated method for dealing with memory effects.

8) spd_D=dark_inter(spd,sctbl,/multia) (help:IA UTILITIES)

		 save,spd_D,file='spd_D.save     ',/xdr

NOTE: Usually the server or window system is configured to retain plot window after scrolling or the start of a screen saver. The /multia option is only needed if the draw window is not retained (as on a Multia terminal); it causes IDL to handle the plot storage directly.

The dark_inter tool facilitates dark current correction. It can be the most time-consuming of all data reduction steps, but is also one of the most important in producing reliable spectra. Make sure to save the SPD after completing this task. Before you begin, you should note the sequence of darks and other scans in the scan table. Note that the reset time ("tr") and gain columns in the scan table must be the same for a dark and the scan from which it is being subtracted; dark_inter automatically makes available all darks with matching tr and gain. Also check the scan table to determine which (band+scan) combinations hold actual data: those which do not will show "0.000" for both wave_min and wave_max (this information is also shown in the message box in dark_inter as "w_range"). It is not necessary to dark subtract these (band+scan) combinations. For each of the four bands (starting at detectors 1 [SW1:], 13 [SW2:], 25 [LW1:], and 37 [LW2:]), the following sequence should be followed:

A.Select a detector. This detector will represent the whole band.

B.Revise Darks.  Click "Revise Darks". In the "Revise Darks" window which pops up:

 
		$\parbox{5.7in}{$\rightarrow$ ~Select the first set of darks. This ma...
...t'' button first and then clicking the dark you want in the middle left panel.}$

 
		 $\rightarrow$ choose clipping options-here are the most general suggestions: 
 
		 		$\parbox{5.1in}{$\triangleright$\space first 1 point and clipping(mean) with kappa=3 (i.e., 3$\sigma$\space clipping around the mean).}$

 




				$\parbox{5.1in}{$\triangleright$ After strong photometric checks, you...
...emory effects (i.e., jump in signal followedby a decline in signal over time).}$

  
		     $\rightarrow$ refine dark to be subtracted for each detector of the band, for each scan: 
 




				$\parbox{5.1in}{$\triangleright$ ''clip'' shows the points to be clip...
...mits youhave specified, and all points above or below these lines are clipped.}$

 




				$\parbox{5.1in}{$\triangleright$ ''apply to whole band'' shows the effect of the current choice for all detectors inthe band.}$

    




				$\parbox{5.1in}{$\triangleright$ many iterations may be tried (use ''...
...''undo'' button in between, results in the first detector being clipped twice.}$







		$\parbox{5.7in}{$\rightarrow$ ~Click on ''next'' to select the next d...
... dark, youwill get an error message in the text window reminding you to do so.}$

 
		$\parbox{5.7in}{$\rightarrow$ ~Click ''done'' in the ''Revise Darks'' window when all darks in the band have been revised.}$

 







C. Subtract darks from photometric checks and science scans: 
 




		$\parbox{5.7in}{$\rightarrow$ ~Select the photometric check or scienc...
...en duringthe entire use of dark\_inter, so that you may readjust as necessary.}$

 




 		      $\rightarrow$ Choose the dark subtraction method: click on "Method". 
 




				$\parbox{5.1in}{$\triangleright$ In the ''method'' menu, choose which...
...us'' and ''next'' buttons, and click ''linear interpolation'' for the method. }$

  
				$\parbox{5.1in}{$\triangleright$ When you have successfully marked th...
...and toggle your dark choices until they appear in redin the left plot windows.}$

 




				$\parbox{5.1in}{$\triangleright$\space Choosing other darks: If you w...
...the dark points are colored yellow-orange again to besure you were successful.}$

 




				$\parbox{5.1in}{$\triangleright$ If the available darks do not provid...
...tector individually, and is much more time consuming; use only whennecessary. }$

 




				$\triangleright$
Click "Done" in the Method menu.

 
		$\rightarrow$ Subtract fitted dark current. 
 
				$\parbox{5.1in}{$\triangleright$ Click on ''fit'', which willshow the fit for the selected detector in the bottom left plot.}$

 
				$\parbox{5.1in}{$\triangleright$ If the method selected is ''mean'' o...
...ll detectors in this band/scan combination,you have completed the subtraction.}$

  
				$\parbox{5.1in}{$\triangleright$ However, often ''jumps'' are seen in...
...rites a new solution for that detector, and doesn't affect theother detectors.}$

   




				$\parbox{5.1in}{$\triangleright$ If the method selected is ''spline''...
...r left plot. Repeat the ''fit'' step for each individual detector in the band.}$

 
				$\parbox{5.1in}{$\triangleright$ Additional notes for spline fits: Pr...
...ed special plot parameters to get the selected scan to fit in the leftwindows.}$

  
		$\parbox{5.7in}{$\rightarrow$ ~Repeat Step C for each photometric check and science scan in this band.}$

D. Set slider on top to the first detector number of the next band and repeat the process, starting with Step B, Revise Darks. Complete Steps B and C for all four bands. It is a good idea to save the data set periodically during the use of dark_inter, to prevent loss of efforts in the event that the tool crashes. The easiest way to do this is to use the "save current" button in the dark_inter tool, which creates a file called "dark_inter_save.xdr" containing the current data file. This file is overwritten in successive uses of this button, so you may make incremental saves without using additional disk space.

9) spd_DM=mask_inter(spd_D,sctbl)    (help: IA UTILITIES)

		 save,spd_DM,file='spd_DM.save',/xdr 

This command puts you into the mask_inter tool, which can be used to inspect your SPD and mask out bad data. Sometimes a whole detector is significantly noisier than the others in the band, or there may be significant memory effects after a strong glitch. In addition, you can mask out the jump segments first detected with dark_inter (see step 8C). See the on-line help for this tool for a detailed account of all the utilities of this tool.

Enter a band number in the "band" box, click on the scan number you wish to see, and click "disp_spd" to display a subset of the data. Note that to examine a different scan, you must click on the previous scan number again to "unset" it; otherwise there will be two scans selected and the tool will complain (multiple scans can be selected for producing an AAR within mask_inter, but only one scan can be displayed at a time). If you click the ``plotmode'' button, a background window will pop up in which you can set the plot scale; we recommend that you choose a fixed y-scale, as this is a good way of finding detectors for which dark subtraction didn't work properly, or for finding noisy detectors or other strange features. Each time you change the plot scale, click "disp_spd" again to display the data with the new plot values.

To mask out a specific section of a detector, click "select" and the detector of interest, which will cause the data from that detector to show in the bottom left window. Then click "mask" and use the left mouse button to select the region to be masked out (e.g., in the case of a memory effect). If you see the same glitch in all 12 detectors of a band, you can use the "do to all" button after masking out the glitch in one of the detectors. If you judge the input from an entire detector to be bad, you can click on the detector number in the list to the right-hand side of the display, and then click "faulty" to set it all bad.

IMPORTANT NOTE: Using the "faulty" button to set a detector "bad" is NOT reversible at the current time, and sets the specified detector bad for all scans. Setting the detector bad by masking all points out using the "mask" button IS reversible (using "unmask"), and only sets the detector bad for the specified scan.

For now, set 'order' to zero, thus showing all the data in that band; the order is used to separate different AOT bands, which are of interest in later steps. Saving the masked SPD should be considered if a significant amount of masking has been done.

10) spd_U=updown1_inter(spd_DM,sctbl,aotlr   )  (help: IA UTILITIES)

		 save,spd_U,file='spd_U.save',/xdr

This tool corrects for differences between the 'up' and 'down' scans of the instrument. Observations are taken in pairs of "up" and "down" scans, where wavelength decreases with increasing time in the "up" scan, and increases with time in the "down" scan. Due to memory effects, the shapes of the ``up'' and ``down'' scan may be different. The updown1_inter tool takes a smoothed version of each of the up and down scans separately, and determines a "gain" or "offset" correction to make the up and down scans consistent with each other. The top plot of the tool shows a section of the scan including the region you are correcting; you will see darks or photometric checks on either end of this plot. The center plot shows the up and down scan laid over each other, and the bottom of the three long plots shows either the ratio or difference between the up and down scans (depending on whether you have chosen the "gain" or "offset" option). For updown1_inter (as opposed to updown_inter, the version used for other observing modes), the middle and bottom plots are in wavelength-like units, but due to the overlapping AOT bands they are not in actual wavelengths. Note that in these plots, wavelength decreases to the right.

The "gain" option is really only useful for observations with very strong signal, where the uncertainties due to dark subtraction can be neglected with respect to gain effects. In addition, the choice of the smoothing radii should depend on speed, choosing smaller values for higher speeds; it is also dependent on resolution. Table 1 summarizes the general recommendations for median and smoothing radii as well as correction method for the different bands. The key is to look at the resultant correction that the tool wants to make and be sure that it doesn't follow the noise too closely or introduce artifacts into the time spectrum.

NOTE: Ignore "Floating divide by zero" errors upon starting updown1_inter, as they are spurious.

The general procedure:

 


		 $\parbox{5.5in}{$\rightarrow$ Set ''det'' to the first detector number of the band you want to correct, and select the scan.}$

 




 		$\rightarrow$Set the correction mode and median and smoothing radii          (see Table 1).

 




 		$\parbox{5.5in}{$\rightarrow$ Click ''Do Adjustment''; the bottom of ...
...s the correction determined from the smoothed data overlaid as a dotted curve.}$

  




 		$\parbox{5.5in}{$\rightarrow$ You may iterate on choosing the correct...
... to perform the up-down correction on all detectors in the band for that scan.}$

Repeat this process for all scans in each band, and for all four bands. After completing the updown correction, save the SPD.

Table 1: Recommended Up-Down Correction Settings

Band	Scan	R(median)	R(smooth)	method
1	1	50-100	50-200	offset
1	2	"	"	"
2	1	"	"	"
2	2	"	"	"
3	1	"	"	offset or (gain)
3	2	100-200	200-400	"
3	3	"	"	"
4	1	"	"	"

11) plot_phot,spd_U,sctbl,[1],13,/all     (help:IA UTILITIES)

		 plot_phot,spd_U,sctbl,[1],25,/all

		 plot_phot,spd_U,sctbl,[1],37,/all

The routine plot_phot can be used to look at photometric check data. The command lines given above request IA to plot the first photometric check for all 12 detectors of bands 2, 3, and 4 (which start with detector numbers 13, 25, and 37); the band 1 photometric check is not used. If you have more than one photometric check, you can replace the [1] above with a range of values (e.g. [1,3]) and compare them on the same plot. By specifying a particular detector and removing the /all flag, you can look at the photometric checks for that detector.

Data points already flagged as bad by the system will be marked (see help file for color code). If these are the only bad points and the photometric checks look sensible, the standard routines will handle the bad points appropriately. Note that the default is to always use the first photometric check, so if it is corrupted it should be replaced with another. If you suspect your photometric data has problems which are severe enough to affect the ultimate calibration, such as a corrupted first phot. check or large spikes not already flagged, consult an IA guru.

12) spd_U=respcal(spd_U)  (help: IA AUTO ANALYSIS)

		 spd_U=fluxcon(spd_U) 
		 spd_U=velcor(spd_U) 

The routines respcal and fluxcon apply various calibration files: respcal uses CAL files 13, 19, and 25; fluxcon uses CAL file 13. The output of fluxcon looks like the following:

Warning(SPCP): some phot. check data might be doubtful       Det:26 Uv: 2
Warning(SPCP): some phot. check data might be doubtful       Det:39 Uv: 2
Warning(SPCP): some phot. check data might be doubtful       Det:46 Uv: 3
Warning(SPCP): some phot. check data might be doubtful       Det:48 Uv: 2

If the number following "Uv:" (the number of outliers) is greater than 3, the photometric check data for that detector indicated may be suspect.

The velcor task performs a correction to the heliocentric frame based on the velocity of the satellite.

13) spd_F=flat_inter(spd_U,sctbl)     (help: IA UTILITIES)

		 save,spd_F,file='spd_F.save',/xdr 

The first command starts up the flatfield correction tool flat_inter. This task corrects the flatfield of each detector in the band with respect to the others. As with "updown1_inter", the correction can be a "gain" or "offset" correction, and is made on the basis of smoothed data. It is generally recommended to use the "median" filter, and to exclude poor points. As with updown1_inter, you should make sure that the fit you have chosen does not introduce spurious features, and vary the smoothing or correction parameters accordingly, and the 'gain' option should only be used with strong signal. You should apply flatfields to each AOT separately by choosing the appropriate combination of band, scan, and order. Table 2 shows the suggested settings for correction mode and smoothing radii for each AOT band. NOTE: be sure to press "return" after changing each entry; otherwise, strange effects may result if new values are not registered by the tool.

Table 2: Flatfield correction settings

AOT	Band	scan	order	mode	res1	res2
1A	1	1	4	offset	50-500	50-200
1B	1	1	3	"	"	"
1D	1	2	3	"	"	"
1E	1	2	2	"	"	"
2A	2	1	2	"	"	"
2B	2	1	1	"	"	"
2C	2	2	1	"	"	"
3A	3	1	2	offset or (gain)	25-400	50-100
3C	3	2	2	"	"	"
3D	3	2	1	"	"	"
3E	3	3	1	"	"	"
4	4	1	1	"	"	"

14) spd_FM=mask_inter(spd_F,sctbl)  (help: IA UTILITIES)

		 save,spd_FM,file='spd_FM.save',/xdr 

Now that all calibrations have been applied, you can go back into the interactive masking tool used in step 9 to check for lines which have been marked as glitches (in low-resolution SWS01 data, lines can be confused as glitches). There can be a significant number of such glitches, even in the slowest-speed SWS01 observations, and they may mark the line wings as well as the line peak. You may also wish to mask out noise spikes or outliers; for instance, a signal which is clearly spurious, such as a strong spike detected only in the up or down scan of all 12 detectors, will bias the resulting spectrum. This masking step may be best done after following the rest of the steps described below and looking at an unmasked version of the output spectrum to see where lines have been detected, and returning to this step to produce a refined dataset.

See Table 3 below for the wavelength ranges covered by each AOT. Click the "w-mode" button to plot in wavelength units, and specify the wavelength range of interest in the ``plotmode'' pop-up window. Glitched points are marked by yellow asterisks. To remove an unwanted glitch, click "select" to specify the detector, and if needed click "zoom" and use the cursor to select the section of the scan holding the offending point (both up and down scans will be shown, with wavelength units along the upper x axis of the plot). Then click "unglitch", and click the mouse cursor on either side of the glitched point. The unglitched point will then appear as a white cross, like the rest of the good data. You can also use the ``fluxclip'' button to mask out data by hand. When you click this button, the data from all detectors is plotted in the lower left hand window, and you can define a series of boxes with the left mouse button; the points inside these boxes are then clipped. Clicking the right mouse button returns you to the main mask_inter tool.

You can see the effects of your masking by using the "extract_aar" and "aar_f" buttons, which will produce a filtered AAR (with the resolution specified in the "resol" window) of the specific section of spectrum that you have selected with band, scan, order, and x_range. Further, you can output your final, masked AAR result to a file by clicking on the "save_aar" button. This action will pop up a window where you can specify the name of the file you wish to write, and what format: XDR, IDL SAVE, or FITS. The first two options both write IDL save files which can be restored in an IA session (the first using the XDR format described in step 4).

If you choose this method of producing AARs, then step 15 below is not necessary; skip to step 16 to complete your reduction.

15) spd_FM=killwait(spd_FM)     (help: IA UTILITIES)

		 aar=extract_aar(spd_FM)     (help: IA AUTO ANALYSIS)

		 save,aar,file='aar.save',/xdr 
		 status=sap_wfits('aar.fits',aar)   (help: ISAP)

		 aar_filt=aarfilt1(aar,1,/noglitch,/noover)   (help: IA UTILITIES)

The first task flags "wait" points as "bad"; it is useful on SWS01, SWS02, and SWS06 data. Note that running the killwait task is not necessary if AARs are produced from the mask_inter tool (see step 14). The next task extracts an AAR from the reduced SPD, and the following two commands save the AAR as an IDL save set and a FITS file. Note that all the masked and glitched points are still in the AAR, but are ignored by aarfilt1.

The last task produces a final, filtered AAR (bad points removed, order overlaps resolved, each AOT set to a particular resolution). The second parameter of aarfilt1 is the speed factor, which determines the effective resolution. A speed factor of 1 should be used only for the slowest of the 4 AOT1 modes (S01 S=4; see the scan table for scan_type). Be advised that aarfilt1 was optimized for the slowest AOT1 mode, and that the optimum resolution will depend on the source distribution as well as the AOT Band. For S=1 (fastest) data, a speed factor of $\sim$0.3 is appropriate, with intermediate values for S=2 and S=3 data. Consult an IA guru for more advice. Note that glitches are not automatically neglected; you must use the /noglitch option to do so. This is most significant in bands 3 and especially 4, where there are many points marked as glitches. Table 3 shows the band width and resolution settings in the current version of aarfilt1 (version 1.1, 20 May 1996).

Table 3: AOT Bandwidths set by AARFILT1

AOT	(Band,Scan,Order)	($\lambda_1$,$\lambda_2$)	R($\lambda$/ $\delta\lambda$) a
1A	(1,1,4)	2.3,2.6	1700
1B	(1,1,3)	2.6,3.02	1400
1D	(1,2,3)	3.02,3.52	1650
1E	(1,2,2)	3.52,4.08	1200
2A	(2,1,2)	4.08,5.30	(0.8)*1600
2B	(2,1,1)	5.30,7.03	(0.8)*1000
2C	(2,2,1)	7.03,12.35	(0.8)*1600
3A	(3,1,2)	12.35,16.4	1500
3C	(3,2,2)	16.4,19.5	1900
3D	(3,2,1)	19.5,27.5	1200
3E	(3,3,1)	27.5,29.5	1450
4	(4,1,1)	29.5,45.2	1400

^a Resolution assumes a speed factor of 1.

16) aar_fringe=aarfringe(aar_filt) (help:IA UTILITIES)

After this step, the basic reduction of SWS01 data is complete, but there is likely significant fringing, especially for lines lying in band 3 (and to some extent in band 2). These may be dealt with using the interactive tool "aarfringe".

The documentation for aarfringe is quite detailed, and we recommend you read it thoroughly to determine your options; only a brief description of tool usage is given here. When you start the tool, you will see buttons across the top of the tool to select the data and method of fringe fitting. Select each band, one at a time, from the pull-down AOT Band menu at the top left. Then you must choose the fringe fitting method: Fast Fourier Transform (FFT), sinusoidal (SIN) or Do Kester's method (DOK) of iteratively fitting sine components. You may also choose gain or offset; gain is the default for this tool, as you usually see fringes most clearly in stronger sources.

Before attempting a fringe fit, you should enter the radial velocity of your source as well as the wavelengths of strong lines in the bottom center panel. These lines will be excluded >from the fit; choose the resolution such that the entire line gets masked out. The bottom right panel is where you specify parameters specific to each model; consult the help file for how best to fill in this block. Do Kester's method has been seen to produce the best results in general, but certainly try multiple methods to see what works best for your data.

Clicking the ``Do it'' button at top right performs a fit; you can change parameters/methods and click this button again. When you think you have a good fit that you want to keep, click ``Apply''. Repeat for each AOT band that you want to correct. When you are done with all, click ``Done'' to return to the IA prompt. At this point, you should save your AAR again if you did any significant de-fringing.

Finishing up:

Type ``help'' to look at the list of IDL structures you have generated in this data reduction section. Make sure to save all files which were produced from the most time-consuming steps (i.e., any of the interactive tools) as IDL save sets and FITS files before exiting from IA. Then you can restore the SPD or AAR at any stage and redo some of the reduction if you find it necessary at a later date.