From jwf@ipac.caltech.eduWed Sep 10 14:43:26 1997
Date: Fri, 25 Jul 1997 12:41:53 -0700 (PDT)
From: jwf@ipac.caltech.edu
To: 2mass@ipac.caltech.edu
Cc: chas@ipac.caltech.edu, sstrom@donald.phast.umass.edu,
stiening@ipac.caltech.edu
Subject: WG Mtg #127 Minutes
IPAC 2MASS Working Group Meeting #127 Minutes
7/22/97
Attendees: R. Beck, R. Cutri, T. Evans, J. Fowler, L. Fullmer, T. Jarrett,
G. Kopan, B. Light, H. McCallon, B. Wheaton, S. Wheelock, J. White
AGENDA
1.) Compiler Options
2.) Camera Anomalies
3.) Data Compression
4.) Engineering Data Bases
5.) Data Base Updates
6.) Images on CACR
7.) ERB Meeting
8.) Saturation Analysis
DISCUSSION
1.) Compiler Options
Some of the more CPU-intensive subsystems have not yet been compiled with
the options that take maximum advantage of the ultra sparc capabilities. It
was decided that all relevant subsystems, especially DARKS, PIXCAL, POSFRM,
and GALWORKS, would be redelivered by the end of Wednesday, July 23, with
makefiles updated to use the options described in a recent email from G.
Kopan, namely:
-fast -depend -xarch=v8plusa -xchip=ultra
After all deliveries have been made, the night of 970521 will be run with the
new software, and run times and products will be examined. The products should
be identical, but the runtimes should be significantly improved. Since the
goal is to determine whether runtimes are acceptable for production, debug
output generation should be temporarily turned off.
2.) Camera Anomalies
Some serious anomalies not seen in protocamera data have been observed in
all bands at this time. Each band has a different affliction. As noted
previously, Ks suffers from "hot pixels" which suddenly turn on at high DN
levels for several frames; these tend to look like cosmic ray glitches and are
now being treated as such, since they do not correlate with sources on the
sky. H has a problem with pixels dying each time a thermal cycle is performed.
The intent is not to have any more thermal cycles than absolutely necessary,
but two unexpected warmups have occurred recently despite efforts to the
contrary.
What appears to be the worst problem is occurring in J: bias fluctuations
have started appearing in dark and flat sequences and in calibration and
science scans. A statistical description of this phenomenon is just now being
developed, so it is not yet known whether scans and dark/flat sequences can
simply be rejected once reliable diagnostics are developed.
The fluctuations appear to occur somewhat independently in different
quadrants and to have the nature of a rescaling of the bias-reset-decay
pattern. On one night (970717), the mask logic in DARKS turned off over 58,000
pixels in J as a result of high standard deviations about the trimmed-average
dark values (because the darks and responsivity for J were rejected, the
draconian masks were also not used for downstream processing). The DARKS
summary for this night included the following:
Band/ReadType: J/1 H/1 K/1 J/2 H/2 K/2
Avg Dark Sigma: 6.24 7.05 5.45 19.37 1.80 2.12
St Dev of Above: 18.16 30.52 17.60 4.45 0.79 4.09
No. Masked Pix: 345 347 91 58084 10 53
The average standard deviation about the trimmed average dark value is about
ten times as large in J Read2-Read1 than in H or K.
J. Fowler reported that in addition to these symptoms, and probably to
some extent in concert with them, the new darks and responsivities computed
for each night are being systematically rejected, with the canonicals being
used for downstream processing. The rejections are occurring in all bands, not
just J (although the K results are sometimes accepted), and the basis for
rejection is the new results being too different from the canonicals. Since
the same canonicals are being used night after night, the amounts by which new
results differ indicates the extent of instability in the darks and
responsivities. The following table shows the night-to-night variation in the
differences between the nightly dark results and the canonicals for
Read2-Read1, where is the mean difference from the canonical in DN, and
d() is the difference in between the night and previous available value,
where data are not available if the difference was within spec (these data are
taken from the DARKS printout that occurs only when results are rejected).
Night d(J2) d(H2) d(K2)
970702 43.3 20.1
970703 34.4 -8.9 22.3 +2.2
970704 38.4 +4.0 18.2 -4.1 10.1
970705 41.1 +2.7 20.1 +1.9
970706 48.5 +7.4 1.6 -18.5 19.4 +9.3
970707 45.4 -3.1 13.7 +12.1 17.8 -1.6
970708 70.0 +24.6 29.9 +16.2 11.8 -6.0
Mean differences greater than 10.0 cause rejection. The Read1 darks are also
being rejected; a similar table for Read1 follows.
Night d(J2) d(H2) d(K2)
970702 -108.0 -30.1 14.4
970703 -99.7 +8.3 -35.5 +5.4
970704 -86.8 +12.9 -36.8 -1.3
970705 -102.0 -15.2 -22.7 +14.1 12.9 -1.5
970706 11.5 +113.5 68.2 +90.9
970707 28.3 -39.9
970708 51.2 +22.9 21.1 +8.2
Responsivity rejections are occurring due to out-of-spec mean differences
between new and canonical values and due to the number of individual pixels
that differ by more than 0.01 in responsivity. To be acceptable, the mean
responsivity difference ("" below) must be less than 0.003 with a
standard deviation ("Sig()" below) of less than 0.005, and the number of
individual pixels that differ by more than 0.01 ("NDR()" below) must be less
than 25. The following table gives the values of these parameters for cases
where rejections occurred in the same nights as those above.
Night Sig(J) NDR(J) Sig(H) NDR(H) Sig(K) NDR(K)
970702 -0.0007 0.0206 40302 0.0084 0.0267 53725 -0.0006 0.0074 8588
970703 -0.0008 0.0206 40459 0.0086 0.0267 53839 0.0006 0.0103 10070
970704 -0.0013 0.0209 40492 0.0084 0.0268 53235 0.0007 0.0073 9007
970705 -0.0008 0.0206 40492 0.0085 0.0268 53551 0.0016 0.0074 10631
970706 -0.0032 0.0222 45228 0.0016 0.0177 28147 0.0003 0.0056 4402
970707 0.0003 0.0166 30589 0.0020 0.0122 16577 0.0002 0.0050 2931
970708 (no flat sequences available)
The responsivities appear to be returning to the canonical values after some
excursion. It seems advisable to consider loosening up the thresholds to
permit the nightly responsivities to be used. Currently no automated creation
of new canonicals is done; human intervention is required to appoint a set of
masks, darks, and responsivities to canonical status. The current policy is
also to decouple the eligibility of darks from responsivities and between
bands. New masks are used only if a corresponding dark or responsivity is
used.
3.) Data Compression
J. White reported that the data compression problem is still open. Some
progress has been made, but the goal has not been met. Many variations in
preconditioning and software compression have been tried. The best result so
far managed to write 52 GB to one tape; the goal is to write 70 GB to one
tape, which is the advertised spec for DLT hardware compression.
4.) Engineering Data Bases
T. Evans reported that engineering data bases (see the minutes of meeting
number 125, 6/17/97) do not require data dictionaries after all, as had been
previously believed. The absence of a data dictionary makes it necessary to
access the data by column names or numbers, but when this is done, plotting
and statistical analyses can be done via the data base's built-in
capabilities.
5.) Data Base Updates
T. Evans reported that a change in data base structure concurrent with a
redelivery of the DBMAN subsystem will be coming up in a few weeks. A brief
period of data base unavailability is expected, and further advance notice
will be given.
6.) Images on CACR
R. Beck reported that images are now being shipped to the CACR storage
facility. A script is being written that will permit accessing of the data,
and this will be distributed when ready.
7.) ERB Meeting
R. Cutri reported that the 2MASS Executive Review Board will meet in
Boston next Tuesday. He, C. Beichman, and T. Chester will attend. Roc
requested that members review the items to be discussed; information is
available at the following web addresses.
http://spider.ipac.caltech.edu/staff/tchester/2mass/erb_issues.html
http://pegasus.phast.umass.edu/2mass/erb/
8.) Saturation Analysis
B. Wheaton reported that progress has been made in the saturation
analysis. The average J "soft saturation" threshold appears to be very close
to 43,000 DN, which is the single value currently being used for all bands.
The values for H and K look as though significantly different means will be
found, however, and the dispersions in values for a single band appear large
enough to warrant separate values for individual pixels.