Progress in Reprocessing the Hyperion/EO-1 Data Set with a Prototype Pipeline for SBG
Explore the source record for details and available documents.
Jon M. Jenkins↗
SEARCH · Search NASA
Results for “reprocessed data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Parallelization and visual analysis of multidimensional fields: Application to ozone production, destruction, and transport in three dimensions
Atmospheric modeling is a grand challenge problem for several reasons, including its inordinate computational requirements and its generation of large amounts of data concurrent with its use of very large data sets derived from measurement instruments like satellites. In addition, atmospheric models are typically run several times, on new data sets or to reprocess existing data sets, to investigate or reinvestigate specific chemical or physical processes occurring in the earth's atmosphere, to understand model fidelity with respect to observational data, or simply to experiment with specific model parameters or components.
Schwan, Karsten↗
Combined Mesonet and Tracker
Title: Combined Mesonet and Trackers (UNL Mobile Mesonets) Authors University of Nebraska PI: Adam Houston, UNL Professor (ahouston2@unl.edu) Mailing Address: 126 Bessey Hall P.O. Box 880340 Lincoln, NE 68588-0340 CoMeT Overview The University of Nebraska-Lincoln operates three Combined Mesonet and Tracker (CoMeTs). CoMeTs are Ford Explorers (model years 2015, 2017, and 2019) with forward-mounted suites of meteorological sensors and dual moonroofs, combining the capability of a mobile mesonet to collect near-surface observations with the capability of an unmanned aircraft systems (UAS) tracker vehicle, which enables an observer in the second row of seats to see the aircraft and maintain compliance with Federal Aviation Administration policies on UAS operation. The CoMeTs collect observations of slow temperature and humidity at ~2 m above ground level (AGL) using a Vaisala HMP155A, fast temperature at ~2 m AGL using a Campbell Scientific 109SS-L thermistor, pressure at ~2.5 m AGL using a Vaisala PTB210, wind speed and direction at ~3.25 m AGL using an R.M. Young 05103 propeller anemometer, and vehicle heading using a KVH Industries C-100 fluxgate compass (Barbieri et al. 2019). The HMP155A and 109SS-L thermistor are shielded and aspirated within a U-tube (Waugh and Frederickson 2010; Houston et al. 2016). This list of sensors is also included in the CoMeT data file metadata. Manufacturer specifications for these instruments are given in Table 1 of Hanft and Houston (2018). The reported measured quantities are summarized below.CoMeT-3 was funded through an equipment allocation included in the NSF TORUS award (AGS-1824649). Instrument Description The specific sensors included on each CoMeT are summarized in the table at the end of this section. In general each CoMeT collects observations of slow temperature and humidity at ~2 m above ground level (AGL) using a Vaisala HMP155, fast temperature at ~2 m AGL using a Campbell Scientific 109SS-L thermistor, pressure at ~2.5 m AGL using a Vaisala PTB210 barometer with a Gill pressure port, wind speed and direction at ~3.25 m AGL using an R.M. Young 05103 propeller anemometer, position using a Garmin 19x HVS receiver, and vehicle heading using a KVH Industries C-100 fluxgate compass. The HMP155 and 109SS are shielded and aspirated within a U-tube (Waugh and Frederickson 2010; Houston et al. 2016). Fast temperature and corrected RH measurements (using sensors housed within the U-tube) have a time constant of 10-12 s based on data collected across a temperature and RH shock during the CLOUD-MAP 2017 calibration/validation tests on June 26, 2017. Vehicle speed was < 10 kts for this test. CoMeT-1 CoMeT-2 CoMeT-3 Slow Temperature Slow RH Vaisala HMP155A-L20-PT Part #: 22280-7 Vaisala HMP155E Part #: E1AA11A0B1A1A0A Vaisala HMP155E Part #: E1AA11A0B1A1A0A Fast temperature Campbell Scientific 109SS-L20-PT Part #: 21448-3 Campbell Scientific 109SS-L12-PW Part #: 21448-109 Campbell Scientific 109SS-L12-PW Part #: 21448-150 Pressure Vaisala PTB-210 Part #: A1A1B Gill Pressure Port Part #: 61002 Vaisala PTB-210 Part #: A1A1B Gill Pressure Port Part #: 61002 Vaisala PTB-210 Part #: A1A1B Gill Pressure Port Part #: 61002 Wind RM Young 05103-L20-PT Part #: 18435-310 RM Young 05103-L20-PW Part #: 18435-244 RM Young 05103-L20-PW Part #: 18435-244 GPS Garmin GPS 19x HVS (NMEA 0183) Part #: 010-01010-00 Garmin GPS 19x HVS (NMEA 0183) Part #: 010-01010-00 Garmin GPS 19x HVS (NMEA 0183) Part #: 010-01010-00 Compass KVH C-100 Part #: 01-0177-15 KVH C-100 Part #: 01-0177-15 KVH C-100 Part #: 01-0177-15 Logger Campbell Scientific CR6-NA-XT-SW Part #: 28385-9 Campbell Scientific CR6-WIFI-XT-SW Part #: 28385-6 Campbell Scientific CR6-WIFI-XT-SW Part #: 28385-6 Data Collection and Real-Time Processing The reported measured quantities are summarized in the table below. Quantity Units Source Epoch time Seconds GPS Latitude and longitude Degrees GPS Altitude m GPS Pressure hPa PTB210 Temperature (fast) deg C 109SS-L Temperature (slow) deg C HMP155 RH (slow) % HMP155 Vehicle speed m/s GPS Vehicle heading deg C-100 and GPS In addition to the measured variables, several derived variables are calculated. Corrected/fast relative humidity (%) Relative humidity is adjusted to the fast temperature following Richardson et al. (1998) and Houston et al. (2016). Water vapor mixing ratio (g/kg) Dew point temperature (°C) Potential temperature (Kelvin) Virtual potential temperature (Kelvin) Equivalent potential temperature (Kelvin) Regular intercomparisons between all three CoMeTs were performed during TORUS 2019. Comparisons were also conducted between CoMeT-1 and CoMeT-2 during LAPSE-RATE (2018) on 14 July. In these intercomparisons, the vehicles were parked adjacent to each other aligned perpendicular to (and facing into) the wind. To minimize engine heating effects, intercomparisons were only conducted when the wind speed was >10 kts. Data Format Original data files for each deployment are saved as text files and then converted to NetCDF. NetCDF versions have units that are CF compliant and may not match the original units in the txt files. The naming convention for the NetCDF files is as follows: UNL.CoMeT3.{deployment date YYYYMMDD}.{start time of observation collection in UTC HHMM}.L2.{post-processing codes}.cdf example: UNL.CoMeT3.20190627.1931.L2.g1.f1.cdf Post-processing codes are included to track modifications to the raw data. These codes are closely connected to error flags associated with each record. Each letter corresponds to a particular instrument: g: GPS p: Barometer tf: Fast temperature ts: Slow temperature rh: Relative humidity f: Compass w: Wind monitor a: All instruments Each number corresponds to a particular post-processing action described more below. Measured and derived variables are included in the following table. Variable Heading Standard Name Units time Time seconds since 00:00:00, 01-01-1970 Alt Altitude meters lat Latitude degrees north lon Longitude degrees east fast_temp Air Temperature kelvin slow_temp Air Temperature kelvin pressure Air Pressure pascals logger_RH Relative Humidity percent calc_corr_RH Relative Humidity percent wind_speed Wind Speed meters per second wind_dir Wind From Direction degrees vehicle_dir Vehicle Direction degrees dewpoint Dew Point Temperature kelvin mixing_ratio Humidity Mixing Ratio g/g theta Air Potential Temperature kelvin theta_v Virtual Potential Temperature kelvin theta_e Equivalent Potential Temperature Kelvin error_flag The error_flag variable is a string that matches the post-processing codes listed above. All instruments will have an associated code, but will have a “0” if the datum is unchanged from the initial processed value. Error Codes The following table summarizes the error codes for data collected before 2020: Error Code Relevant CoMeT Description g1 1,2,3 Exact correction. GPS position and time reprocessed from raw data g2 1 As far as we can tell this is an exact correction to an error in the GPS time. During the correct time periods the time suddenly went backwards ~250s and stayed at this offset for 750s when it corrected itself. The offset was applied to the “time warp” period. p1 2 Approximate correction. Hole in the pressure tube connecting the pressure port to the barometer. Resulted in erroneously low air pressure measurements when the vehicle was in motion. Derived variables recalculated (dew point temperature [e depends on qv and p], water vapor mixing ratio, potential temperature, virtual potential temperature, equivalent potential temperature) a1 3 Exact correction. Missing data reprocessed from raw data a2 1 Bug fix to bias correction for ts1, ts2, and rh1: water vapor mixing ratio was off by a factor of 10 and virtual potential temperature was wrong because of this. f1 3 No correction, missing data. Fluxgate compass inoperable. Wind speed and direction calculated using GPS-derived vehicle heading instead. rh1 1 Approximate correction. Constant bias of +1.7% removed from relative humidity. Derived variables recalculated (corrected/fast relative humidity, dew point temperature, water vapor mixing ratio, virtual potential temperature, equivalent potential temperature) ts1 1 Approximate correction. Constant bias of +0.6 K removed from slow temperature. Derived variables recalculated (corrected/fast relative humidity, dew point temperature, water vapor mixing ratio, virtual potential temperature, equivalent potential temperature) ts2 1 Approximate correction. Constant bias of +1.0 K removed. Derived variables recalculated (corrected/fast relative humidity, dew point temperature, water vapor mixing ratio, virtual potential temperature, equivalent potential temperature) References Bolton, D., 1980: The Computation of Equivalent Potential Temperature. Mon. Wea. Rev., 108, 1046–1053, https://doi.org/10.1175/1520-0493(1980)108<1046:TCOEPT>2.0.CO;2. Hanft, W., and A. L. Houston, 2018: An Observational and Modeling Study of Mesoscale Air Masses with High Theta-E. Mon. Wea. Rev., 146, 2503–2524, https://doi.org/10.1175/MWR-D-17-0389.1.Wexler Houston, A. L., R. J. Laurence III, T. W. Nichols, S. Waugh, B. Argrow, and C. L. Ziegler, 2016: Intercomparison of unmanned aircraft-borne and mobile mesonet atmospheric sensors. Journal of Atmospheric and Oceanic Technology. 33, 1569-1582, doi: 10.1175/JTECH-D-15-0178.1. Lowe, P. R., 1977: An Approximating Polynomial for the Computation of Saturation Vapor Pressure. J. Applied Meteorology, 16, 100–103. Richardson, S. J., S. E. Frederickson, F. V. Brock, and J. A. Brotzge, 1998: Combination temperature and relative humidity probes: Avoiding large air temperature errors and associated relative humidity errors. Preprints, 10th Symp. On Meteorological Observations and Instrumentation, Phoenix, AZ, Amer. Meteor. Soc., 278–283. Waugh, S., and S. E. Frederickson, 2010: An improved aspirated temperature system for mobile meteorological observations, especially in severe weather. 25th Conf. on Severe Local Storms, Denver, CO, Amer. Meteor. Soc., P5.2. [Available online at https://ams.confex.com/ams/25SLS/techprogram/paper_176205.htm.]
54 ENVIRONMENTAL SCIENCES↗
On the Reprocessing and Reanalysis of Observations for Climate
The long observational record is critical to our understanding of the Earths climate, but most observing systems were not developed with a climate objective in mind. As a result, tremendous efforts have gone into assessing and reprocessing the data records to improve their usefulness in climate studies. The purpose of this paper is to both review recent progress in reprocessing and reanalyzing observations, and to summarize the challenges that must be overcome in order to improve our understanding of climate and variability. Reprocessing improves data quality through more scrutiny and improved retrieval techniques for individual observing systems, while reanalysis merges many disparate observations with models through data assimilation, yet both aim to provide an climatology of Earth processes. Many challenges remain, such as tracking the improvement of processing algorithms and limited spatial coverage. Reanalyses have fostered significant research, yet reliable global trends in many physical fields are not yet attainable, despite significant advances in data assimilation and numerical modeling. Oceanic reanalyses have made significant advances in recent years, but will only be discussed here in terms of progress toward integrated Earth system analyses. Climate data sets are generally adequate for process studies and large-scale climate variability. Communication of the strengths, limitations and uncertainties of reprocessed observations and reanalysis data, not only among the community of developers, but also with the extended research community, including the new generations of researchers and the decision makers is crucial for further advancement of the observational data records. It must be emphasized that careful investigation of the data and processing methods are required to use the observations appropriately.
Uncertainty↗
A Comprehensive Plan for the Long-Term Calibration and Validation of Oceanic Biogeochemical Satellite Data
The primary objective of this planning document is to establish a long-term capability and validating oceanic biogeochemical satellite data. It is a pragmatic solution to a practical problem based primarily o the lessons learned from prior satellite missions. All of the plan's elements are seen to be interdependent, so a horizontal organizational scheme is anticipated wherein the overall leadership comes from the NASA Ocean Biology and Biogeochemistry (OBB) Program Manager and the entire enterprise is split into two components of equal sature: calibration and validation plus satellite data processing. The detailed elements of the activity are based on the basic tasks of the two main components plus the current objectives of the Carbon Cycle and Ecosystems Roadmap. The former is distinguished by an internal core set of responsibilities and the latter is facilitated through an external connecting-core ring of competed or contracted activities. The core elements for the calibration and validation component include a) publish protocols and performance metrics; b) verify uncertainty budgets; c) manage the development and evaluation of instrumentation; and d) coordinate international partnerships. The core elements for the satellite data processing component are e) process and reprocess multisensor data; f) acquire, distribute, and archive data products; and g) implement new data products. Both components have shared responsibilities for initializing and temporally monitoring satellite calibration. Connecting-core elements include (but are not restricted to) atmospheric correction and characterization, standards and traceability, instrument and analysis round robins, field campaigns and vicarious calibration sites, in situ database, bio-optical algorithm (and product) validation, satellite characterization and vicarious calibration, and image processing software. The plan also includes an accountability process, creating a Calibration and Validation Team (to help manage the activity), and a discussion of issues associated with the plan's scientific focus.
Hooker, Stanford B.↗
On the Reprocessing and Reanalysis of Observations for Climate
The long observational record is critical to our understanding of the Earth s climate, but most observing systems were not developed with a climate objective in mind. As a result, tremendous efforts have gone into assessing and reprocessing the data records to improve their usefulness in climate studies. Many challenges remain, such as tracking the improvement of processing algorithms and limited spatial coverage. Reanalyses have fostered significant research, yet reliable global trends in many physical fields are not yet attainable, despite significant advances in data assimilation and numerical modeling. Communication of the strengths, limitations and uncertainties of reprocessed observations and reanalysis data, not only among the community of developers, but also with the extended research community, including the new generations of researchers and the decision makers is crucial for further advancement of the observational data records. WCRP provides the means to bridge the different motivating objectives on which national efforts focus.
Bosilovich, Michael G.↗
Back to the Future: Surveying the Northern Hemisphere and Reprocessing the Southern TESS Data Set
TESS launched 18 April 2018 to conduct a two-year, near all-sky survey for at least 50 small, nearby exoplanets for which masses can be ascertained and whose atmospheres can be characterized by ground- and space-based follow-on observations. TESS has completed its survey of the southern hemisphere and begun its survey of the northern hemisphere, identifying >1000 candidate exoplanets and unveiling a plethora of exciting non-exoplanet astrophysics results, such as asteroseismology, asteroids, and supernova. The TESS Science Processing Operations Center (SPOC) processes the data downlinked every two weeks to generate a range of data products hosted at the Mikulski Archive for Space Telescopes (MAST). For each sector (~1 month) of observations, the SPOC calibrates the image data for both 30-min Full Frame Images (FFIs) and up to 20,000 pre-selected 2-min target star postage stamps. Data products for the 2-min targets include simple aperture photometry and systematic error-corrected flux time series. The SPOC also conducts searches for transiting exoplanets in the 2-min data for each sector and generates Data Validation time series and associated reports for each transit-like feature identified in the search. Multi-sector searches for exoplanets are conducted periodically to discover longer period planets, including those in the James Webb Continuous Viewing Zone (CVZ), which are observed for up to one year. Starting with Sector 8, scattered light from the Earth and Moon contaminated significant portions of the data in each orbit. We have developed algorithms for automated identification of the scattered light features at the individual target level. Previously, data for all stars on a CCD affected by scattered light were manually excluded. The automated flagging will allow us to retain significantly more data for stars that are not affected by the scattered light even though it is occurring elsewhere on the CCD. We also discuss enhancements to the SPOC pipeline and the newly available FFI light curves. The TESS Mission is funded by NASA's Science Mission Directorate as an Astrophysics Explorer Mission.
Jenkins, Jon M.↗
An Assessment of SNPP and NOAA20 VIIRS RSB Calibration Performance in NASA SIPS Reprocessed Collection-2 L1B Data Products
Two VIIRS sensors onboard the SNPP and NOAA20 satellites have been successfully operating for over 10 and 4 years, respectively, providing the worldwide user community with high-quality imagery and radiometric measurements of the land, atmosphere, cryosphere, and oceans. This study provides a temporal radiometric stability and calibration consistency assessment of the SNPP and NOAA20 VIIRS reflective solar bands using the latest NASA SIPS C2 L1B products. Several independent vicarious approaches are used to examine the stability of SNPP VIIRS and consistency of the at-sensor reflectance between the two VIIRS instruments. These approaches include observations from simultaneous nadir overpasses, the Libya-4 desert and Dome C snow/ice sites, and deep convective clouds. The impact of existing band spectral differences on the reflectance measurements is accounted for utilizing scene-specific hyperspectral observations provided by the SCIAMACHY sensor onboard the ENVISAT platform. Results indicate that both SNPP and NOAA20 VIIRS reflectances are stable within 1% over their mission periods for all bands, except for a few bands in the visible range from SNPP VIIRS that show more upward drifts at high radiances. NOAA20 VIIRS reflectances are systematically lower than SNPP by 2 to 4% for most bands, with the exception of few short wavelength bands where it is seen to be up to 7%.
VIIRS↗
The CARMENES Search for Exoplanets Around M Dwarfs: Revisiting the GJ 581 Multi-Planetary System With New Doppler Measurements From CARMENES, HARPS, and HIRES
Context.GJ 581 is a nearby M dwarf known to host a packed multiple planet system composed of two super-Earths and a Neptune-mass planet. We present new orbital analyses of the GJ 581 system, utilizing recent radial velocity (RV) data obtained from the CARMENES spectrograph combined with newly reprocessed archival data from the HARPS and HIRES spectrographs. Aims.Our aim was to analyze the post-discovery spectroscopic data of GJ 581, which were obtained with CARMENES. In addition, we used publicly available HIRES and HARPS spectroscopic data to seek evidence of the known and disputed exoplanets in this system. We aimed to investigate the stellar activity of GJ 581 and update the planetary system’s orbital parameters using state-of-the-art numerical models and techniques. Methods.We performed a periodogram analysis of the available precise CARMENES, HIRES, and HARPS RVs and of stellar activity indicators. We conducted detailed orbital analyses by testing various orbital configurations consistent with the RV data. We studied the posterior probability distribution of the parameters fit to the data and we explored the long-term stability and overall orbital dynamics of the GJ 581 system. Results. We refined the orbital parameters of the GJ 581 system using the most precise and complete set of Doppler data available. Consistent with the existing literature, our analysis confirms that the system is unequivocally composed of only three planets detectable in the present data, dismissing the putative planet GJ 581 d as an artifact of stellar activity. Our N-body fit reveals that the system’s inclination is i=47.0+14.6−13.0deg, which implies that the planets could be up to 30% more massive than their previously reported minimum masses. Furthermore, we report that the GJ 581 system exhibits long-term stability, as indicated by the posterior probability distribution, characterized by secular dynamical interactions without the involvement of mean motion resonances.
Satellites↗
GLDAS-2 Land Surface Model Data and Data Services at NASA GES DISC
The goal of the NASA Global Land Data Assimilation System (GLDAS, https://ldas.gsfc.nasa.gov/gldas(https://ldas.gsfc.nasa.gov/gldas)) is to generate optimal fields of land surface states and fluxes by ingesting satellite- and ground-based observational data products, using advanced land surface modeling and data assimilation techniques (Rodell et al., 2004).The GLDAS dataset currently archived at and distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC, https://disc.gsfc.nasa.gov/ (https://disc.gsfc.nasa.gov/)) is GLDAS Version 2 (GLDAS-2). It contains a series of output fields from the upgraded Noah-3.6, Catchment-F2.5, and VIC-4.1.2 Land Surface Models (LSMs) in the Land Information System (LIS-V7, https://lis.gsfc.nasa.gov/ (https://lis.gsfc.nasa.gov/)). GLDAS-2 has three components:GLDAS-2.0, GLDAS-2.1, and GLDAS-2.2. GLDAS-2.0 is forced entirely with the upgraded Princeton Meteorological ForcingV2.2 Dataset and provides a temporally consistent series from 1948 through 2014. GLDAS-2.1 is forced with a combination of model and observation data, with data spanning from 2000 to the present. The GLDAS-2.2 product suite uses data assimilation(DA), whereas the GLDAS-2.0 and GLDAS-2.1 products are "open-loop" (i.e., no data assimilation). The choice of forcing data, as well as DA observation source, variable, and scheme, varies for different GLDAS-2.2 products. The currently availableGLDAS-2.2 data contain a daily 0.25-degree output from the Catchment-F2.5 LSM in LIS-V7. The data are forced with the meteorological analysis fields from the operational European Centre for Medium-Range Weather Forecasts Integrated Forecasting System (ECMWF-IFS) and assimilated with GRACE and GRACE-FO data, ranging from February 1, 2003 to the present. The current GLDAS-2.0 and 2.1 Noah LSM data were reprocessed in November 2019 and January 2020 respectively and their data from Catchment and VIC LSMs are new to the GLDAS-2 collection. This presentation provides a summary of theGLDAS-2 data products, their land surface fields, and their related data services at the GES DISC; and a description of the majorGLDAS-2 climatological characteristics as well as the intercomparison with the data of the previous version.
Hydrology↗
Incorporating the TRMM Dataset into the GPM Mission Data Suite
In June 2015 the TRMM satellite came to its end. The 17 plus year of mission data that it provided has proven a valuable asset to a variety of science communities. This 17plus year data set does not, however, stagnate with the end of the mission itself. NASA/JAXA intend to integrate the TRMM data set into the data suite of the GPM mission. This will ensure the creation of a consistent, intercalibrated, accurate dataset within GPM that extends back to November of 1998. This paper describes the plans for incorporating the TRMM 17plus year data into the GPM data suite. These plans call for using GPM algorithms for both radiometer and radar to reprocess TRMM data as well as intercalibrating partner radiometers using GPM intercalibration techniques. This reprocessing will mean changes in content, logical format and physical format as well as improved geolocation, sensor corrections and retrieval techniques.
TRMM↗
Automated data processing architecture for the Gemini Planet Imager Exoplanet Survey
The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.
79 ASTRONOMY AND ASTROPHYSICS↗
The Process of Bringing Dark Data to Light: The Rescue of the Early Nimbus Satellite Data
Myriad environmental satellite missions are currently orbiting the earth. The comprehensive monitoring by these sensors provide scientists, policymakers, and the public critical information on the earths weather and climate system. The state of the art technology of our satellite monitoring system is the legacy of the first environment satellites, the Nimbus systems launched by NASA in the mid-1960s. Such early data can extend our climate record and provide important context in longer-term climate changes. However, the data was stowed away and, over the years, largely forgotten. It was nearly lost before its value was recognized and attempts to recover the data were undertaken. This paper covers what it took the authors to recover, navigate and reprocess the data into modern formats so that it could be used as a part of the satellite climate record. The procedures to recover the Nimbus data, from both film and tape, could be used by other data rescue projects, however the algorithms presented will tend to be Nimbus specific. Data rescue projects are often both difficult and time consuming but the data they bring back to the science community makes these efforts worthwhile.
dark↗
Accelerating Machine Learning Inference with GPUs in ProtoDUNE Data Processing
Abstract We study the performance of a cloud-based GPU-accelerated inference server to speed up event reconstruction in neutrino data batch jobs. Using detector data from the ProtoDUNE experiment and employing the standard DUNE grid job submission tools, we attempt to reprocess the data by running several thousand concurrent grid jobs, a rate we expect to be typical of current and future neutrino physics experiments. We process most of the dataset with the GPU version of our processing algorithm and the remainder with the CPU version for timing comparisons. We find that a 100-GPU cloud-based server is able to easily meet the processing demand, and that using the GPU version of the event processing algorithm is two times faster than processing these data with the CPU version when comparing to the newest CPUs in our sample. The amount of data transferred to the inference server during the GPU runs can overwhelm even the highest-bandwidth network switches, however, unless care is taken to observe network facility limits or otherwise distribute the jobs to multiple sites. We discuss the lessons learned from this processing campaign and several avenues for future improvements.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
First ARM Aerosol Chemical Speciation Monitor Users’ Meeting Report
The first U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility aerosol chemical speciation monitor (ACSM) users’ meeting was held on April 11-13, 2017, at Aerodyne Research, Inc. in Billerica, Massachusetts, to discuss the Southern Great Plains (SGP) atmospheric observatory ACSM data quality and establish best practices for data collection and processing. The participants examined six years of calibration and processed data. Specific issues raised by data users were addressed and case studies from two field experiments were examined. The most recent data from the ACSM installed in the newly commissioned SGP ARM Mobile Facility 7 (AMF7) were also evaluated. The participants recommended that the SGP ACSM data be reprocessed using calibration values averaged over the history of SGP ACSM calibrations. They also recommended that the data quality be evaluated by comparing (1) observed versus predicted particulate ammonium (NH4+) mass loadings, and (2) ACSM mass loadings versus mass loadings calculated from particle size and light scattering data. The contents of the datastreams from the ACSM were defined based on ARM requirements and the necessary tasks to implement these recommendations were assigned to the mentor and the instrument manufacturer. Some of the recommendations were implemented for the SGP data and are presented in this report. Data from two ARM sites located in marine environments were also analyzed. These analyses as well as instrument intercomparisons have led to further questions about data treatment. These will be addresses in the second users’ meeting scheduled for the summer of 2018.
54 ENVIRONMENTAL SCIENCES↗
First ARM Aerosol Chemical Speciation Monitor Users’ Meeting Report
The first U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility aerosol chemical speciation monitor (ACSM) users’ meeting was held on April 11-13, 2017, at Aerodyne Research, Inc. in Billerica, Massachusetts, to discuss the Southern Great Plains (SGP) atmospheric observatory ACSM data quality and establish best practices for data collection and processing. The participants examined six years of calibration and processed data. Specific issues raised by data users were addressed and case studies from two field experiments were examined. The most recent data from the ACSM installed in the newly commissioned SGP ARM Mobile Facility 7 (AMF7) were also evaluated. The participants recommended that the SGP ACSM data be reprocessed using calibration values averaged over the history of SGP ACSM calibrations. They also recommended that the data quality be evaluated by comparing (1) observed versus predicted particulate ammonium (NH 4 + ) mass loadings, and (2) ACSM mass loadings versus mass loadings calculated from particle size and light scattering data. The contents of the datastreams from the ACSM were defined based on ARM requirements and the necessary tasks to implement these recommendations were assigned to the mentor and the instrument manufacturer. Some of the recommendations were implemented for the SGP data and are presented in this report. Data from two ARM sites located in marine environments were also analyzed. These analyses as well as instrument intercomparisons have led to further questions about data treatment. These will be addresses in the second users’ meeting scheduled for the summer of 2018.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
First ARM Aerosol Chemical Speciation Monitor Users’ Meeting Report
The first U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility aerosol chemical speciation monitor (ACSM) users’ meeting was held on April 11-13, 2017, at Aerodyne Research, Inc. in Billerica, Massachusetts, to discuss the Southern Great Plains (SGP) atmospheric observatory ACSM data quality and establish best practices for data collection and processing. The participants examined six years of calibration and processed data. Specific issues raised by data users were addressed and case studies from two field experiments were examined. The most recent data from the ACSM installed in the newly commissioned SGP ARM Mobile Facility 7 (AMF7) were also evaluated. The participants recommended that the SGP ACSM data be reprocessed using calibration values averaged over the history of SGP ACSM calibrations. They also recommended that the data quality be evaluated by comparing (1) observed versus predicted particulate ammonium (NH 4 + ) mass loadings, and (2) ACSM mass loadings versus mass loadings calculated from particle size and light scattering data. The contents of the datastreams from the ACSM were defined based on ARM requirements and the necessary tasks to implement these recommendations were assigned to the mentor and the instrument manufacturer. Some of the recommendations were implemented for the SGP data and are presented in this report. Data from two ARM sites located in marine environments were also analyzed. These analyses as well as instrument intercomparisons have led to further questions about data treatment. These will be addresses in the second users’ meeting scheduled for the summer of 2018.
54 ENVIRONMENTAL SCIENCES↗
First ARM Aerosol Chemical Speciation Monitor Users’ Meeting Report
The first U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility aerosol chemical speciation monitor (ACSM) users’ meeting was held on April 11-13, 2017, at Aerodyne Research, Inc. in Billerica, Massachusetts, to discuss the Southern Great Plains (SGP) atmospheric observatory ACSM data quality and establish best practices for data collection and processing. The participants examined six years of calibration and processed data. Specific issues raised by data users were addressed and case studies from two field experiments were examined. The most recent data from the ACSM installed in the newly commissioned SGP ARM Mobile Facility 7 (AMF7) were also evaluated. The participants recommended that the SGP ACSM data be reprocessed using calibration values averaged over the history of SGP ACSM calibrations. They also recommended that the data quality be evaluated by comparing (1) observed versus predicted particulate ammonium (NH 4 + ) mass loadings, and (2) ACSM mass loadings versus mass loadings calculated from particle size and light scattering data. The contents of the datastreams from the ACSM were defined based on ARM requirements and the necessary tasks to implement these recommendations were assigned to the mentor and the instrument manufacturer. Some of the recommendations were implemented for the SGP data and are presented in this report. Data from two ARM sites located in marine environments were also analyzed. These analyses as well as instrument intercomparisons have led to further questions about data treatment. These will be addresses in the second users’ meeting scheduled for the summer of 2018.
54 ENVIRONMENTAL SCIENCES↗