Livewire Data Platform: File Standards Version 1.0
This technical document is a user guide to help users of the Livewire Data Platform understand the standards and requirements for storing and sharing data on the Livewire Data Platform.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
This technical document is a user guide to help users of the Livewire Data Platform understand the standards and requirements for storing and sharing data on the Livewire Data Platform.
Often data transfer software is designed to meet specific requirements or apply to specific environments. Frequently, this requires source code integration for added functionality. An extensible data transfer framework is needed to more easily incorporate new capabilities, in modular fashion. Using FrETT framework, functionality may be incorporated (in many cases without need of source code) to handle new platform capabilities: I/O methods (e.g., platform specific data access), network transport methods, data processing (e.g., data compression.).
The Soil Moisture Active/Passive (SMAP) microwave radiometer is a fully polarimetric L-band radiometer flown on theSMAP satellite in a 6 AM/6 PM sun-synchronous orbit at 685 km altitude. After the SMAP L1B_TB data product version 3 was released in 2016, the radiometer has been undergoing further calibration and validation with the goal of reducing both the bias in the cold-sky measurements and calibration drift in the global ocean measurements experienced during eclipse seasons in data product version 3. The post-launch calibration algorithm has been upgraded by using new estimates of the reflector emissivity as well as using multiple scenes to calibrate the radiometer internal reference sources and antenna gain simultaneously. In addition, a correction offset is applied to the ocean roughness model for horizontal polarization based on nadir observations. Test and validation results show that the goal is achieved (e.g., biases are removed and the calibration stability achieved for data release version 4 is 0.1 K(rms) over both the global ocean and CS).
Through phase 1 and phase 2 SBIR grants, s fully functional I/O Controller and communication protocol for version 4 of EPICS is completed. This new software architecture provides a flexible and extendible architecture. Version 4 is implemented fully in Java. The performance metrics look promising. The final portion of phase 2 is to optimize the communication mechanisms. Subsequent work on different aspects of this are required to provide a viable solutions in various areas. Version 3 of EPICS is able to provide a platform for implementing channel based control, because the channel and attributes for time stamping, alarm, display and control were narrow, well defined, and complete. To extend EPICS functionality beyond this, it is necessary to define attributes needed for archive data, array, image data, and directory services. The proper handling of several array types enables the development of middle layer servers such as orbit and bump control in accelerators. Phase 1 should produce a well defined, reviewed, and agreed upon definition of the metadata required for these services. A Phase 2 grant would provide tools that implemented archiving, general array, imaging, and directory applications.
The design of the NASA JSC data handling system is presented. Data acquisition parameters and computer display formats and the flow of image data through the system, with recommendations for improving system efficiency are discussed along with modifications to existing data handling procedures which will allow utilization of data duplication techniques and the accurate identification of imagery.
This data package includes hourly meteorological and soil sensor data at eight ecohydrology monitoring sites in East River Watershed, Colorado as part of the Watershed Function Scientific Focus Area (WFSFA) research led by Lawrence Berkeley National Lab (LBNL). Four field sites were located on the hillslope of East River (ER) near Pump House (PH) at Mount Crested Butte (ER-PHS1 to 4), and the other four are in the Snodgrass Mountain (SG) area (SG-EHS5 to 8). In terms of vegetation cover, three sites are in montane grasslands (ER-PHS1, ER-PHS2, and SG-EHS5), three are below evergreen conifer canopy (ER-PHS3, SG-EHS6, and SG-EHS7), and two are below deciduous aspen canopy (ER-PHS4 and SG-EHS8). The monitoring period began in October 2019 at the East River sites, in October 2020 at SG-EHS5 and SG-EHS6, and in October 2021 at SG-EHS7 and SG-EHS8. In September 2024, all four East River sites were fully retired. The four Snodgrass Mountain sites remain active. Each site is equipped with a comprehensive suite of meteorological sensors on a tripod and soil sensors that measure weather, energy fluxes, and soil variables. This data package includes measurements from ten different types of sensors and up to thirteen individual sensors per site, including (1) a weather station (measurement height ranges from 2.8~3.8 meters (m) above ground), (2) a quantum sensor for photosynthetic active radiation (PAR) (2.4~3.3m), (3) a net radiometer (1.7~2.1m), (4) an infrared radiometer (1.6~2.2m), (5) a sonic distance sensor (1.5~1.9m), (6) a soil carbon dioxide (CO2) flux chamber (0m), (7) a soil heat flux plate (-0.05m below ground), (8) a soil oxygen sensor (-0.3m), (9) a soil water potential sensor (-0.3m), and (10) soil water content sensors at 3~4 depths (-1.15 ~ -0.1m). A total of twenty-three variables is reported in this data package, including (1) atmospheric variables: air temperature (TA), atmospheric pressure (PA), vapor pressure (VP), and vapor pressure deficit (VPD), (2) precipitation variables: rain precipitation (P) and snow depth (D_SNOW), (3) energy fluxes variables: four-component net radiation (NETRAD) (shortwave/longwave incoming/outgoing radiation, SW_IN, SW_OUT, LW_IN, LW_OUT), photosynthetic photon flux density (PPFD), and soil heat flux (G), (4) soil variables: soil water content (SWC), soil water potential (SWP), soil temperature (TS), soil bulk electrical conductivity (COND_SOIL), and soil gaseous oxygen concentration (O2_SOIL), (5) wind variables: two-dimensional wind speed (WS), gust speed (WS_MAX), and wind direction (WD), and (6) surface variables: surface infrared temperature (T_CANOPY) and soil CO2 flux (CO2_SOIL). Please see the Methods section for data processing and QA/QC steps taken to generate the hourly datasets. The following files are included in this data package (notes on version: v{x}-{y}, where x is the metadata version, and y is the data version, when applicable): (1) “metadata_site_v{x}-{y}.csv” - a site metadata file that summarizes location information of all sites, including site ID, description, coordinates, timeframe, elevation, and vegetation cover, (2) “metadata_instrument_v{x}-{y}.csv” - an instrument metadata file that summarizes sensor information of all sites, including sensor manufacturer and model, measurement height, and sampling and averaging interval of all variables, (3) "data_{SITE_ID}_v{x}-{y}.csv" - eight data files that contain hourly data of each site indicated by {SITE_ID} in the filename, (4) “/figure/data_{SITE_ID}_v{x}-{y}.png" - eight figures that help visualize data of each site indicated by {SITE_ID} in the filename, (5) “/photo/*” - photos of each site indicated by {SITE_ID} in the filename, and (6) four file level metadata (flmd.csv) and data dictionary (*_dd.csv) files that summarize file, header, column, and variable information of all files. Notes: (1) Measurement height: Each variable name is followed by conventional positional qualifiers “H_V_R”, where H indicates the relative horizontal positions of that specific variable, V the vertical positions, and R the replicates. In this data package, only the vertical qualifier V varies, and V increases from the highest vertical position (V=1) to the lowest. Variables with the same qualifier are not necessarily measured by the same sensor, and the same variable with the same qualifier across different sites are not necessarily measured at the same height. Please refer to “metadata_instrument.csv” for the sensor information and measurement heights, and whether a variable is measured below the canopy. (2) Variable availability: Snow depth is not available at ER-PHS3 and SG-EHS7. SWC, soil temperature, and soil bulk EC at the deepest depth (<-1m) are not available at SG-EHS6 and SG-EHS7. The missing value code for numeric variables is -9999, except for SWP. For SWP, the missing value code is +9999, because SWP values are negative. (3) Sampling frequency: Please refer to “metadata_instrument.csv” for the increase of sampling frequency of some variables from 30-min to 1-min at ER-PHS1 to 4 in July 2020. (4) Sensors: While the methods of each sensor are not detailed, all sensors are commercially available, and their methods can be found in their manuals. Please refer to “metadata_instrument.csv” for the sensor manufacturer and model information. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.
Comparison of Total Ozone Mapping Spectrometer (TOMS) data to the network of ground-based Dobson/Brewer measurements reveals difference in the time dependence of the calibration of the two systems. We have been searching for a method to determine the time dependence of the TOMS calibrations that is independent of the Dobson/Brewer network. In a separate paper by DeLand et al., calibrations of the Solar Backscatter UV Spectrometer (SBUV) instruments have been rederived using the D-pair (306/313 nm wavelengths) data at the equator. These calibrations have been applied to the data from the Nimbus 7 SBUV and the NOAA 9 and 11 SBUV/2 data to derive a new version 7 data set for each instrument. We have used these data to do a detailed comparison to the Nimbus 7 and Earth Probe TOMS data. Assuming that the D-pair establishes the correct calibration, these comparisons reveal some small calibration drifts (approximately 1%) in the TOMS data. They also reveal an offset in the D-pair calibration with respect to the Dobson network of approximately 8 Dobson units with the Dobson being lower than the D-pair. The D-pair calibration offsets have been used to create a merged ozone data set from TOMS with a calibration that has been determined independent of the Dobson/Brewer network. Trend analyses of these data will be presented and compared to trend analyses using the ground-based data.
Analyses from SAGE I/II version 6.0 data exhibit upper stratospheric ozone trends which are not significantly different from those in version 5.96 data. Trend calculations show larger downward trends at mid-high latitudes in the Southern Hemisphere than in the Northern Hemisphere, particularly in 1980s. There are also indications of decreasing downward trends with time from 1979 to 1999. We have used a chemical box model and the UARS measurements of long lived gases, CH4, H2O, NO(x), and temperature to show that, with a constant Cl(sub y) trend, a hemispheric ozone trend asymmetry of 1%/decade at 45 deg. around 43 km is expected due to the hemispheric differences of temperature and CH4 during late winter/early. Also ozone trends should have been approximately 1%/decade more negative from 1979-1989 than from 1989-1999 because of the chemical feedbacks. The model results further indicate that both the reported decrease in CH4 and the increase in H2O in HALOE measurements will result in a larger downward ozone trend and a decrease in the hemispheric ozone trend asymmetry.
Abstract. A new weakly coupled land data assimilation (WCLDA) system based on the four-dimensional ensemble variational (4DEnVar) method is developed and applied to the fully coupled Energy Exascale Earth System Model version 2 (E3SMv2). The dimension-reduced projection four-dimensional variational (DRP-4DVar) method is employed to implement 4DVar using the ensemble technique instead of the adjoint technique. With an interest in providing initial conditions for decadal climate predictions, monthly mean anomalies of soil moisture and temperature from the Global Land Data Assimilation System (GLDAS) reanalysis from 1980 to 2016 are assimilated into the land component of E3SMv2 within the coupled modeling framework with a 1-month assimilation window. The coupled assimilation experiment is evaluated using multiple metrics, including the cost function, assimilation efficiency index, correlation, root-mean-square error (RMSE), and bias, and compared with a control simulation without land data assimilation. The WCLDA system yields improved simulation of soil moisture and temperature compared with the control simulation, with improvements found throughout the soil layers and in many regions of the global land. In terms of both soil moisture and temperature, the assimilation experiment outperforms the control simulation with reduced RMSE and higher temporal correlation in many regions, especially in South America, central Africa, Australia, and large parts of Eurasia. Furthermore, significant improvements are also found in reproducing the time evolution of the 2012 US Midwest drought, highlighting the crucial role of land surface in drought lifecycle. The WCLDA system is intended to be a foundational resource for research to investigate land-derived climate predictability.
A global in situ data set for validation of ocean colour products from the ESA Ocean Colour Climate Change Initiative (OC-CCI) is presented. This version of the compilation, starting in 1997, now extends to 2021, which is important for the validation of the most recent satellite optical sensors such as Sentinel 3B OLCI and NOAA-20 VIIRS. The data set comprises in situ observations of the following variables: spectral remote-sensing reflectance, concentration of chlorophyll-a, spectral inherent optical properties, spectral diffuse attenuation coefficient, and total suspended matter. Data were obtained from multi-project archives acquired via open internet services or from individual projects acquired directly from data providers. Methodologies were implemented for homogenization, quality control, and merging of all data. Minimal changes were made on the original data, other than conversion to a standard format, elimination of some points, after quality control and averaging of observations that were close in time and space. The result is a merged table available in text format. Overall, the size of the data set grew with 148 432 rows, with each row representing a unique station in space and time (cf. 136 250 rows in previous version; Valente et al., 2019). Observations of remote-sensing reflectance increased to 68 641 (cf. 59 781 in previous version; Valente et al., 2019). There was also a near tenfold increase in chlorophyll data since 2016. Metadata of each in situ measurement (original source, cruise or experiment, principal investigator) are included in the final table. By making the metadata available, provenance is better documented and it is also possible to analyse each set of data separately.
This Technical Note provides the design document, database specification, and database administration for the NIDAS Version 3.1 System Developed for the Naval Oceanographic Office.
This document describes the initial formulation (Version 1.0.0) of the Atmospheric Radiation Measurement (ARM) program satellite data analysis procedures. Techniques are presented for calibrating geostationary satellite data with Sun synchronous satellite radiances and for converting narrowband radiances to top-of-the-atmosphere fluxes and albedos. A methodology is documented for combining geostationary visible and infrared radiances with surface-based temperature observations to derive cloud amount, optical depth, height, thickness, temperature, and albedo. The analysis is limited to two grids centered over the ARM Southern Great Plains central facility in north-central Oklahoma. Daytime data taken during 5 Apr. - 1 May 1994, were analyzed on the 0.3 deg and 0.5 deg latitude-longitude grids that cover areas of 0.9 deg x 0.9 deg and 10 deg x 14 deg, respectively. Conditions ranging from scattered low cumulus to thin cirrus and thick cumulonimbus occurred during the study period. Detailed comparisons with hourly surface observations indicate that the mean cloudiness is within a few percent of the surface-derived sky cover. Formats of the results are also provided. The data can be accessed through the World Wide Web computer network.
The flow cytometry data file standard provides the specifications needed to completely describe flow cytometry data sets within the confines of the file containing the experimental data. In 1984, the first Flow Cytometry Standard format for data files was adopted as FCS 1.0. This standard was modified in 1990 as FCS 2.0 and again in 1997 as FCS 3.0. We report here on the next generation flow cytometry standard data file format. FCS 3.1 is a minor revision based on suggested improvements from the community. The unchanged goal of the standard is to provide a uniform file format that allows files created by one type of acquisition hardware and software to be analyzed by any other type. The FCS 3.1 standard retains the basic FCS file structure and most features of previous versions of the standard. Changes included in FCS 3.1 address potential ambiguities in the previous versions and provide a more robust standard. The major changes include simplified support for international characters and improved support for storing compensation. The major additions are support for preferred display scale, a standardized way of capturing the sample volume, information about originality of the data file, and support for plate and well identification in high throughput, plate based experiments. Please see the normative version of the FCS 3.1 specification in Supporting Information for this manuscript (or at http://www.isac-net.org/ in the Current standards section) for a complete list of changes.
We have performed an initial assessment of available data libraries for photonuclear reactions on 239 Pu. Specifically, we considered the following five evaluated data libraries: the ENDF/B-VIII.0 and the recently deployed ENDF/B.VIII.1 Evaluated Nuclear Data Files, the International Atomic Energy Agency (IAEA) Photonuclear Data Library 2019 (IAEA-2019), the Japanese evaluated nuclear data library version 5 (JENDL-5), and the 2023 version of the TALYS Evaluated Nuclear Data Library (TENDL-2023). Each of these libraries were translated form ENDF to GNDS format using FUDGE. For each of the Pu isotopes, we provide a comparison with available experimental data in the EXFOR experimental nuclear reaction database, as well as a rapid quantitative assessment of the quality of the evaluation.
This report provides background data for the Boreal Ecosystem Atmosphere Study (BOREAS) sites, including daily, seasonal, interannual, and spatial variability of shortwave (SW) radiation at the Earth's surface. This background data, from the Version 1.1 SW data set, was provided by the Surface Radiation Budget (SRB) Climatology Project established by the World Climate Research Program (WCRP).
The Stratospheric Aerosol and Gas Experiment III (SAGE III) instrument, installed on the International Space Station (ISS), has completed over a year of data collection and production of science data products, including the newly released version 5.1 data product. The SAGE III/ISS is a solar and lunar occultation instrument, scanning the light from the sun and moon, through the limb (edge) of the Earth's atmosphere. It was launched in February 2017 and provides data from June 2017 to the present. It continues SAGE's legacy of ozone, aerosol and water vapor profile measurements and extends the lengthy records for monitoring constituents important for understanding stratospheric ozone trends. This presentation shows the preliminary validation results of comparing SAGE III/ISS ozone vertical profiles with those of mission funded ozone sondes, associated lidar, and comparisons with other correlative data.
An ongoing challenge in visual exploration and analysis of large, multi-dimensional datasets is how to present useful, concise information to a user for some specific visualization tasks. Typical approaches to this problem have proposed either reduced-resolution versions of data, or projections of data, or both. These approaches still have some limitations such as consuming high computation or suffering from errors. In this work, we explore the use of a statistical metric as the basis for both projections and reduced-resolution versions of data, with a particular focus on preserving one key trait in data, namely variation. We use two different case studies to explore this idea, one that uses a synthetic dataset, and another that uses a large ensemble collection produced by an atmospheric modeling code to study long-term changes in global precipitation. The primary findings of our work are that in terms of preserving the variation signal inherent in data, that using a statistical measure more faithfully preserves this key characteristic across both multi-dimensional projections and multi-resolution representations than a methodology based upon averaging.
The Ozone Mapping and Profiler Suite Limb Profiler (OMPS-LP) on board the Suomi National Polar-orbiting Partnership (SNPP) satellite was launched on Oct. 28, 2011. Limb profilers measures the radiance scattered from the Earth's atmospheric in limb viewing mode from 290 to 1000 nm and infer ozone profiles from tropopause to 60 km. The recently released OMPS-LP Version 2 data product contains the first publicly released ozone profiles retrievals, and these are now available for the entire OMPS mission, which extends from April, 2012. The Version 2 data product retrievals incorporate several important improvements to the algorithm. One of the primary changes is to turn off the aerosol retrieval module. The aerosol profiles retrieved inside the ozone code was not helping the ozone retrieval and was adding noise and other artifacts. Aerosols including polar stratospheric cloud (PSC) and polar mesospheric clouds (PMC) have a detectable effect on OMPS-LP data. Our results show that ignoring the aerosol contribution would produce an ozone density bias of up to 10 percent in the region of maximum aerosol extinction. Therefore, aerosol correction is needed to improve the quality of the retrieved ozone concentration profile. We provide Aerosol Scattering Index (ASI) for detecting aerosols-PMC-PSC, defined as ln(Im-Ic) normalized at 45km, where Im is the measured radiance and Ic is the calculated radiance assuming no aerosols. Since ASI varies with wavelengths, latitude and altitude, we can start by assuming no aerosol profiles in calculating the ASIs and then use the aerosol profile to see if it significantly reduces the residuals. We also discuss the effect of aerosol size distribution on the ozone profile retrieval process. Finally, we present an aerosol-PMC-PSC correction scheme.