Returning to work as an astronomer was quite exciting for me following my stint with BDM and NASA. I was hired by the Applied Research Corporation to serve as their COBE Program Manager. My duties would be to 'manage' 10 astronomers and scientific programmers, with over 90 % of my time to be served as an astronomer working with the COBE DIRBE program. This means that I would be conducting my own research as well as joining in on the validation of software and the persuit of research programs already in place by the various Principle Investigators, most notably Dr. Mike Hauser - the PI for the DIRBE project.
On the first day I spent the whole time filling out forms, and getting briefings from the ARC staff and COBE staff about how things work at my new home: the Cosmology Data Analysis Center. For the first time in my professional life I had my own office, which I promptly cluttered up with overstuffed bookcases, file cabinets, and boxes of 'old' data and magnetic tapes. My colorful posters went up on the walls. Like a student with a fresh pad of paper on the first day of school, I gleefully opened my new account on the computer system and created my directory and sub-directory tree into which I would eventually crowd my data bases.
By the time I joined the group on February 16, 1993, the COBE program was nearly finished with obtaining data, and was busily processing all the data for public release. The COBE Team had originally met in March 1976 to design the satellite and its science mission. NASA approved the mission in July 1982, and this 4857 pound, $350 million satellite was flown on a Delta rocket out of Vandenberg Air Force Base "Western Space and Missile Center" on November, 1989 The staff already included some 300 people whose names are listed in George Smoot's "Wrinkles in Time" book.
The first release of COBE data to the NSSDC would occur on June 1993, with a second and final release one year later. I was to work with the DIRBE team, who were behind schedule due to the complexity of the calibration process for the 10 bands of infrared data.
I spent my entire first week reading manuals describing the computer system, its software, and locations of critical directories and data archives. It is a hellaciously complicated system of directories, sub_directories and sub-sub-directories into which raw data, software modules in various stages of development, and a rabbits warren of private data archives exist.
The COBE satellite returned to earth an enormous amount of data describing both the spacecrafts health, and actual scientific data. This data was stored for future use in analyzing the scientific data. For instance, the sensitivity of a detector is affected by the orientation of the spacecraft with respect to the earth's magnetic field, the temperature of the focal plane on which the detectors are located, certain voltages which the detectors are set at, and a number of other properties. In designing the satellite, the scientists had to anticipate all of the variables that might influence the operation of the satellite's sensors, and monitor the changes in these parameters as religeously as the scientific data itself. About 300 parameters were recorded 6 times every second during the entire 4 year life of the satellite, resulting in over 67.2 Giga bytes of information.
During a typical day, COBE would accumulate 24 hours of data which was stored on a pair of on-board tape recorders which had a total capacity of 112 Megabytes. The data is then telemetered to the TDRSS satellite during a 9 minute 22 second download window through which 46 MBy are returned to the ground station at Wallops Island. From there the data is transmitted via a microwave link to GSFC where it is archived and split into its various individual components; one each for the three instruments, along with spacecraft 'housekeeping' data. Provision is also made for 'quick look' analysis of the data to make sure nothing unexpected has compromized data quality.
The data and software storage requirements for COBE were enormous by astronomical standards. DIRBE would need 170 GBy ,DMR 110 GBy and FIRAS would require 30 GBy to store all of the data and data products.
The data from each of the three instruments, DIRBE,FIRAS and DMR, are stripped out of the telemetry data stream and processed into final calibrated maps by software facilities which reside on the COBE 'CLUSTER' which is a collection of 17 VAX workstations and mainframes linked together by a network. The CDAC diskfarm has a capacity of 110 gigabytes of data for on-line data storage. Each one of us at CDAC has his/her own desk terminal on which we can manipulate data, modify programs and display graphics as needed.
For the next few weeks I served as a guinne pig for their Guest Investigator User's handbook and provided several pages of comments about things that could be included or explained better. My first few weeks were pretty exciting as I explored, and became familiar with, the COBE software and data.
By March, the first flush of excitement had waned a bit, and I was now introduced to the second aspect of my job. Although a great deal of software had already been written, it had to be checked for accuracy; a process called Validation.
The processing of all the COBE data is a complex undertaking which has required the development of large quantities of computer code broken into 44 major facilities. Each of these facilities, in turn, is built-up from a collection of subroutines. For example, for the DIRBE data, the facility called BPW reads a weeks worth of calibrated data, removes contaminated observations ( cosmic ray spikes, scattered light from moon etc), averages each detector's useful data for each sky pixel covered, determines the polarization, and applies a beam offset correction. It consists of a total of 64 subroutines and about 21,500 lines of code, not including the contents of 279 additional files which are 'included' with the main program code and can add an additional 10,000 lines to the above sum! The end result is a collection of all-sky maps in each of the 10 wavelength bands, and a pair of maps showing the degree of polarization measured in the 3 short-wavelength bands.
With a total of 11,156 subroutines, and 654,000 lines of computer code, there is a lot that could accidently go wrong. A single typographical error such as typing a '1' instead of 'I' could lead to a final product that may be in error in any of a hundred different, and maddeningly frustrating, ways. That's why a staff of 'validators' who are trained in astronomy and data analysis techniques have to go over this software to check that the answers it produces make sense.
The functions that the code was to perform was decided in a set of facility recommendations and outlines which were produced by the Science Team during the years prior to COBE's launch in 1989. These documents outlined exactly how the software was to perform its many functions in transforming the raw data into something that a scientist could use to study the universe. These documents were then handed over to computer programmers at Hughes STX who developed the system archetecture and wrote and debugged the software. Once the software was written, a 'pathfinder' run was begun to process a small part of the data. At this point the validators would go over the output and make certain that the software was indeed performing the job it was designed to do. It was a tedious process. When errors were discovered by the validators, the offending software was identified, and a 'Software Processing Report' or SPR was filed against the program. Corrections were then made in the software and the result was checked by re-running the corrected module just to make certain.
There were several stages in the evolution of the COBE software, particularly in the DIRBE portion. In total about 11,744 SPRS were filed against the software, and corrected by the programmers. Some of these were relatively minor such as the accidental re-assignment of a variable from '3.0' to '2.0'. Other changes were so severe that entire subroutines had to be almost re-written from scratch. In the end, however, the number of errors found declined sharply which signaled that the process of validation was coming to its natural end. Luckly, this coincided with the deadline on April 21 when the absolutely final processing of the all the DIRBE data would begin so that the final data products could be delivered to NSSDC on the agreed upon date in August.
How long does it take the software to process the COBE data? Because of the complex I/O required to input data, create numerous intermediate scratch files, and delete files to save space which was at an absoulte premium, it took about 20 hour of CPU time to process a single weeks data. A total of 5 months would be required to process a single year of data from all three instruments!
The end of the validation process also signaled the end of many many months of tedious work and endless report filing to document for posterity all of the changes and enhancements that had been made to the pipeline software. With this done, we could now return to the exciting work of looking through the data and making new discoveries about the Milky way and the universe.
In terms of research projects, most of the spectacular results had already been written-up by the DIRBE Science team, and were in the publication pipeline. The DIRBE team had been very busy in the years before I joined, extracting many of the exciting results from the data. By the time I had arrived, only a small number of very difficult issues remained open. These were being tackled quite hard by the DIRBE team which by that time had had a long history of working together. I realized quite early in the game that if I was going to carry-out any research at all, it would not involve anything that the DIRBE Team had previously examined. It would have to be something quite a bit less spectacular than modeling the Zodiacal emission from the solar system, or obtaining better estimates for the cosmological background light in the infrared.
The other adjustment difficulty I encountered in moving from an independent researcher to a member of a world-class group, was that, unlike my postdoc days at NRL, here with COBE my science analysis time was charged to the COBE contract and any research I wished to carry-out had to be formally approved by the COBE Science Working Group. This group of astronomers consisted of the original PIs and co-PIs who designed COBE, and who determined what kind of research was appropriate for the COBE project to support. If they didn't like your research idea, you did not get to persue your interest on the COBE contract. If you went ahead and did the research on your own private time, you would not be able to get COBE to pay the page charges for its publication. This ment that many interesting lines of inquiry were never persued, in favor of what were deemed the more spectacular lines of research. This attitude of letting only a few flowers bloom was quite different than the attitude of other NASA programs such as the Extreme Ultraviolet Explorer. In the EUVE program, for example, three times as many papers were published with only a fraction of the data that COBE had.
Particularly disadvantaged by this conservative research program were the contract astronomers. The Science Working Group consisted of astronomers who had tenure either at an academic institution, or through the Government Service. The SWG contracted-out to groups like Hughes STX, ARC and GSC to supply the astronomers and programmers needed to process the data, and to polish it for analysis. Once the COBE contract ends, the contract astronomers 'go away' and have to find something else to do. On the otherhand, astronomers in the rest of the community who also have tenure, get to use the COBE data to further their research. The contract astronomers are noticeably short-changed by this and have to get as much out of the COBE data as they can while working on the COBE contract. If the research and publication policy of the SWG is conservative, and directed towards long-range goals, the contract astronomers whose goals are short-term, get screwed.
Under the terms of the COBE project that were negotiated by the PIs with NASA, following the launch of COBE, the COBE Science Team was obligated to generate a set of data bases to the National Space Science Data Center at GSFC. Each of the three teams, DMR, FIRAS and DIRBE was to provide a 'time-ordered' data set of each observation along with sky maps obtained from each channel that was observed ( 6 for DMR, 10 for DIRBE) along with calibrated spectra ( for FIRAS). In addition, a much larger collection of data sets are needed by the COBE Science Working Group to check the scientific quality of the data. This collection included 117 additional maps and plots.
So where is COBE now? Well, the cryogens that kept the DIRBE far-IR detectors operating at wavelengths longer than 3 microns, finally were exhausted on September 21, 1990 so the 7 long-wavelengths bands ceased returning scientific data. The three short-wavelength detectors covering the range from 1 to 3.5 microns, however, could still function though only at about 20% of their original sensitivity. This 'warm-era' data is now being processed to see if it tells us anything new. The DMR, which is a microwave receiver, continued to function until the COBE satellite itself was shut-off on December ... 1993. The COBE satellite, though it no longer returns scientific data, is still being used as a 'live' satellite for ground-based tracking centers around the world.
Was it really necessary to shut-off the DMR? It was, after all, still working and returning useful scientific data about the Cosmic Microwave Background Radiation! The answer depends on who you talk to. Some say that in terms of its contribution to 'beating down the noise', it had reached a point of diminishing returns for the money invested. With each doubling of the database size, one can reduce the random noise in the data by averaging it, by a factor of square-root of 2 or 70%. But now that the accumulated data spans 4 years, the only way to get a further significant reduction in the noise is to observe for 4 more years! Each year of keeping COBE active and transmitting data to earth costs...., so clearly the next doubling time will cost far more than the previous one from 2 to 4 years.