To: 2MASS Science Team
From: Roc Cutri
Re: Protocamera Run Data Processing Update
As you know, the initial processing of the April/May 1995 Protocamera run data with the operational (ops) pipeline was completed on 20 July 1995. Position reconstruction failed for approximately 300 of the 2000 scans processed in this initial run. The majority (2/3) of the position reconstruction errors were incurred in very high density and/or high obscuration fields. The minority of failures occurred for low density fields due to a processing oversight that failed to pass successful astrometric solutions to the final processor.
Position reconstruction has now been recovered for approximately 80 of the 300 failed scans in predominantly low density fields. The position reconstructed source lists and the headers of the coadded images for these scans now contain the nominally correct positional information. The WWW 2MASS Protocamera Processing Status page has been updated to reflect these corrected scans.
As discussed at the August Science Team Meeting, the point source photometry extracted from the April/May Prototype Camera run data is seriously effected by poor seeing and seeing variations. In particular, seeing variations lead to artificial photometry zero-point offsets between scans on very short timescales and systematic offsets between aperture and psf-fit photometry. Left uncorrected, these offsets make photometric calibration effectively impossible.
Examples were presented at the Team meeting that showed how relatively simple aperture corrections can be applied to the aperture photometry to reduce the effect of seeing variations. The aperture-corrected aperture photometry can then be used to renormalize the psf-fit photometry output to the "infinite-aperture" photometry scale. The results of these corrections lower the dispersion in photometry from repeated scans of the same field taken in poor seeing conditions close to the levels observed in scans taken in very good seeing. It should not be surprising that these corrections are essentially identical to those standardly used for photometry from all 2-d detectors.
As proposed and approved at the Team meeting, a post-processor has been developed to generate and apply aperture corrections and renormalization to the existing source lists. This post-processing was begun on 17 August and should be complete in approximately 2 weeks.
For each scan taken during a night, the steps taken by the post-processor are as follows:
m'(ap) = m(ap) + AC
m'(ap_1) = m(ap_1) + AC
m'(psf) = m(psf) + ZC
The galaxy processor has "matured" considerably in the three months since the delivery of operational pipeline. Therefore, we will re-run galaxy extractions for all scans that had successful position reconstruction. The improvements to the extractor include more robust star/galaxy discrimination and better optimized photometry methods. The new galaxy data will replace the existing files for each scan as the processing takes place.
The photometric corrections will be applied only to those scans that had nominally successful position reconstruction since consistent position information is required for the aphot/psf-fit normalization and persistance and merge processing. Please refer to the WWW Processing Page to verify those scans that have successful position reconstruction.
The coadded images, point and extended source lists with photometry and ancillary files are available on-line to Team members. The location of data for each scan is listed on the WWW 2MASS Protocamera Processing Status page which can be reached via the IPAC/2MASS information page at:
http://www.ipac.caltech.edu/2mass/
Files have been rearranged recently to accommodate the post-processing, so please refer back to the WWW page for current locations. All Team members are welcome and encouraged to review the data products. You will need an active ipac computer account to view or retrieve files, so please contact Jack Lampley at IPAC (818-397-9551) to set up an account if necessary.
To assist you in your navigation of the processed data, a "roadmap" of the data file content and organization has been included below. This is intended to provide you with a brief overview of some of the more useful of the many files present. Please contact us if you would like to find out about the location or content of other files.
"Roadmap" to 2MASS Prototype Camera Data NOTE: wherever necessary, an example file name is given on the right to help make the description on the left more understandable.TOP LEVEL DIRECTORY STRUCTURE Main Data Directory: For example: ==================== ==================== PARTITION/prod/DATE /l4/prod/95-05-04 Beneath the main data directory are the following files and directories: DATE.log 95-05-04.log (log file containing information on all scans taken for that night) bigjob_NNM - bigjob_NNN bigjob_002, bigjob_003, etc. (processing log file for each scan processed for that night) corners (directory containing the positional corners information for each scan processed) sNNM - sNNN s002, s003 s004, etc (directories containing processed data for each scan taken that night) ridgestats_B ridgestats_h, j, k (Files containing ridge statistics for the three bands) ******************************************************************************** WHERE THE DATA LIVES Beneath each scan directory: For example: ============================ ==================== sNNN s043 bigjob_NNN bigjob_43 (processing log file for each scan processed for that night) extended (directory containing extended source processor output) fits.log (a log file written by the position reconstruction program showing RA & DEC) flat (directory containing flat fields, individual flattened image frames, and other miscellaneous files from the flattening processor.) Of particular interest may be: Flat Field Images kc0430013.fits Individual Flattened Frames kf0430001.fits.Z Frame Offset Information k043.of "of" (subdirectory of "flat" directory which contains the aperture photometry and positions of sources detected in individual frames. Used for creating coadds.) Aperture photometry files kf0430001.of image (directory containing coadded images and aperture photometry output) Coadded Images ki0430009.fits.Z Multiaperture photometry for sources in coadded image ki0430009.aps DAOphot summary out for each coadded image ki0430009.ph X, Y info on coadds k043.coadd jobmrg (processing log file from the merge processing) jobr1 (processing log file from the READ-1 processing) jobr2 (processing log file from the READ2-1 processing) mrg (directory containing input parameters for merge processing) plots (directory containing PostScript plots from the processing) File Name Description of Plot ########### ################### apmpsfm.ps ap_m - psf_m plot blend.ps blend # histogram from position reconstructed kamphot source extractions chi.ps histogram of chi^2 values of kamphot extractions delm.ps 2 plots on one page: delta magnitude vs average magnitude from confirmation processor and delta magnitude histogram from confirmation processor delmsep.ps delta magnitude separation from confirmation processor logmag.ps log magnitude histogram from position reconstructed kamphot extractions phnomrgmag.ps magnitude histogram (from confirmation processor) for unmatched DAOphot source extractions phskypssky.ps kamphot vs. daophot sky plot from confirmation processor pipe_out.ps 4 plots printed after pipeline runs as first quality check. Contains: RA-DEC plot of kamphot extractions chi^2 vs dec of all kamphot extractions sky background vs dec of kamphot extractions log plot of magnitudes of kamphot extractions psnomrgmag.ps magnitude histogram for unmatched KAMphot sources from confirmation processor rchimag.ps reduced chi^2 vs. magnitude sep.ps separation histogram from confirmation processor sigmmag.ps sigma magnitude vs. magnitude (kamphot extractions) pos (directory containing output from the position reconstruction processor) Position reconstructed DAOphot extractions k043.p.ph Position reconstructed KAMphot extractions k043.p.ps POSREC output plot for DAOphot reconstruction posrec_ph.ps POSREC output plot for KAMphot reconstruction posrec_ps.ps Any file in this directory with the word "posrec" in its name is a diagnostic file from POSREC, and is used for quality analysis. sdata (directory containing namelist and parameter files used in pipeline processing) src (directory containing source lists before & after position reconstruction) "Extended" sources detected by KAMphot k043.es (contains many noise detections) Point sources detected by DAOphot k043.ph Position reconstructed DAOphot sources k043.p.ph Point sources detected by KAMphot k043.ps Position reconstructed KAMphot sources k043.p.ps Sources rejected by persistence processor k043.p.ps_r Sources accepted and processed for persistence k043.p.ps_s Confirmed sources (between DAOphot and KAMphot) k043phps.conf2.dat DAOphot detections with no KAMphot match k043.ph.nomerge.dat KAMphot detections with no DAOphot match k043.ps.nomerge.dat Read-1 point sources detected by DAOphot k043_1.ph Position reconstructed Read-1 DAOphot sources k043_1.p.ph / tk043.p.ph Table files made from position reconstructed /\ tk043.p.ps source lists (used by other programs \t.k043.p.ps