Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Massively scalable Kerr comb-driven silicon photonic link

Abstract The growth of computing needs for artificial intelligence and machine learning is critically challenging data communications in today’s data-centre systems. Data movement, dominated by energy costs and limited ‘chip-escape’ bandwidth densities, is perhaps the singular factor determining the scalability of future systems. Using light to send information between compute nodes in such systems can dramatically increase the available bandwidth while simultaneously decreasing energy consumption. Through wavelength-division multiplexing with chip-based microresonator Kerr frequency combs, independent information channels can be encoded onto many distinct colours of light in the same optical fibre for massively parallel data transmission with low energy. Although previous high-bandwidth demonstrations have relied on benchtop equipment for filtering and modulating Kerr comb wavelength channels, data-centre interconnects require a compact on-chip form factor for these operations. Here we demonstrate a massively scalable chip-based silicon photonic data link using a Kerr comb source enabled by a new link architecture and experimentally show aggregate single-fibre data transmission of 512 Gb s −1 across 32 independent wavelength channels. The demonstrated architecture is fundamentally scalable to hundreds of wavelength channels, enabling massively parallel terabit-scale optical interconnects for future green hyperscale data centres.

Rizzo, Anthony (ORCID:000000034752797X)↗
DOE OSTI2024

Performance Analysis of Data Processing in Distributed File Systems with Near Data Processing

In the era of big data, the escalating volume and velocity of data generation pose significant challenges in data processing. Traditional systems like Spark and Hadoop manage the increasing amount and velocity of data by improving data placement and processing speeds. However, they face inherent limitations due to the essential data movement required for processing. In this paper, we explore the Skyhook framework, a novel extension of the Ceph distributed system, which significantly reduces the need for data movement. We present an extensive case study using the Skyhook framework, applying it with the TPC-H and K-means clustering algorithms. More specifically, we leverage the TPC-H benchmark to distinguish between CPU-intensive and I/O-intensive tasks. We explore the integration of K-means clustering into SQL, coupled with a near-data processing system to offload the computational burden of the K-means clustering algorithm to storage nodes. We conduct a comprehensive performance evaluation of distributed data processing applications across three processing approaches: traditional layout (baseline), optimized layout, and near-data processing. Additionally, we introduce the use of the FIO tool to simulate real-world system workloads, enabling the measurement of performance metrics such as average latency and CPU utilization. Our research is a significant advance in understanding how to optimize data processing systems to meet the demands of the modern data landscape.

Hou, Shiyue↗
DOE OSTI2024

Modeling performance of data collection systems for high-energy physics

Exponential increases in scientific experimental data are outpacing silicon technology progress, necessitating heterogeneous computing systems—particularly those utilizing machine learning (ML)—to meet future scientific computing demands. The growing importance and complexity of heterogeneous computing systems require systematic modeling to understand and predict the effective roles for ML. We present a model that addresses this need by framing the key aspects of data collection pipelines and constraints and combining them with the important vectors of technology that shape alternatives, computing metrics that allow complex alternatives to be compared. For instance, a data collection pipeline may be characterized by parameters such as sensor sampling rates and the overall relevancy of retrieved samples. Alternatives to this pipeline are enabled by development vectors including ML, parallelization, advancing CMOS, and neuromorphic computing. By calculating metrics for each alternative such as overall F1 score, power, hardware cost, and energy expended per relevant sample, our model allows alternative data collection systems to be rigorously compared. We apply this model to the Compact Muon Solenoid experiment and its planned high luminosity-large hadron collider upgrade, evaluating novel technologies for the data acquisition system (DAQ), including ML-based filtering and parallelized software. The results demonstrate that improvements to early DAQ stages significantly reduce resources required later, with a power reduction of 60% and increased relevant data retrieval per unit power (from 0.065 to 0.31 samples/kJ). However, we predict that further advances will be required in order to meet overall power and cost constraints for the DAQ.

Olin-Ammentorp, Wilkie (ORCID:0000000224729862)↗
DOE OSTI2026

Constellation: The autonomous control and data acquisition system for dynamic experimental setups

The operation of instruments and detectors in laboratory or beamline environments presents a complex challenge, requiring stable operation of multiple concurrent devices, often controlled by separate hardware and software solutions. These environments frequently undergo modifications, such as the inclusion of different auxiliary devices depending on the experiment or facility, adding further complexity. The successful management of such dynamic configurations demands a flexible and robust system capable of controlling data acquisition, monitoring experimental setups, enabling seamless reconfiguration, and integrating new devices with limited effort. This paper presents Constellation, a flexible and network-distributed control and data acquisition software framework tailored to laboratory and beamline environments, that addresses the limitations of existing solutions. The framework is designed with a focus on extensibility, providing a streamlined interface for instrument integration. It supports efficient system setup via network discovery mechanisms, promotes stability through autonomous operational features, and provides comprehensive documentation and supporting tools for operators and application developers such as controllers and logging interfaces. At the core of the architectural design is the autonomy of the individual components, called satellites, which can make independent decisions about their operation and communicate these decisions to other components. This paper introduces the design principles and framework architecture of Constellation, presents the available graphical user interfaces, shares insights from initial successful deployments, and provides an outlook on future developments and applications.

Autonomy↗
DOE OSTI2021

Albedo Data Sets for Bifacial PV Systems

For use by the PV and financial communities to better estimate the performance and to reduce the risk of bifacial PV systems, data sets of ground albedo and associated meteorological data were developed by using existing measurement network data and data contributed by the PV industry. The data sets include time-series data as well as summary information of tabular monthly and yearly data and plots of monthly and hourly albedo values. Complete information is presented in a user's guide and data are available for download from NREL's DuraMAT website.

albedo↗
DOE OSTI2021

EGS-Collab Experiment 1: Time-lapse ERT Data and E4D Inversion Files for 10-24-2018 through 11-07-2018 Flow Test

This data submission includes the raw time-lapse ERT (electrical resistivity tomography) monitoring data, flow system data, operator logs, E4D (https://e4d.pnnl.gov) inversion files, and meta-data necessary to reproduce the 4D ERT inversion for the Oct. 24 through Nov. 7 2018 post-stimulation flow test in test bed 1. The tests were done at the Sanford Underground Research Facility at Homestake Mine in South Dakota.

15 GEOTHERMAL ENERGY↗
DOE OSTI2022

Comparison of scattering ratio profiles retrieved from ALADIN/Aeolus and CALIOP/CALIPSO observations and preliminary estimates of cloud fraction profiles

The space-borne active sounders have been contributing invaluable vertically resolved information of atmospheric optical properties since the launch of Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation (CALIPSO) in 2006. To build long-term records from space-borne lidars useful for climate studies, one has to understand the differences between successive space lidars operating at different wavelengths, flying on different orbits, and using different viewing geometries, receiving paths, and detectors. In this article, we compare the results of Atmospheric Laser Doppler INstrument (ALADIN) and Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) lidars for the period from 28 June to 31 December 2019. First, we build a dataset of ALADIN–CALIOP collocated profiles (Δdist<1°; Δtime<6 h). Then we convert ALADIN's 355 nm particulate backscatter and extinction profiles into the scattering ratio vertical profiles SR(z) at 532 nm using molecular density profiles from Goddard Earth Observing System Data Assimilation System, version 5 (GEOS-5 DAS). And finally, we build the CALIOP and ALADIN globally gridded cloud fraction profiles CF(z) by applying the same cloud detection threshold to the SR(z) profiles of both lidars at the same spatial resolution. Before comparing the SR(z) and CF(z) profiles retrieved from the two analyzed lidar missions, we performed a numerical experiment to estimate the best achievable cloud detection agreement CDA norm (z) considering the differences between the instruments. We define CDA norm (z) in each latitude–altitude bin as the occurrence frequency of cloud layers detected by both lidars, divided by a cloud fraction value for the same latitude–altitude bin. We simulated the SR(z) and CF(z) profiles that would be observed by these two lidars if they were flying over the same atmosphere predicted by a global model. By analyzing these simulations, we show that the theoretical limit for CDA$_{norm}^{theor}$(z) for a combination of ALADIN and CALIOP instruments is equal to 0.81±0.07 at all altitudes. In other words, 19 % of the clouds cannot be detected simultaneously by two instruments due to said differences. The analyses of the actual observed CALIOP–ALADIN collocated dataset containing ~78 000 pairs of nighttime SR(z) profiles revealed the following points: (a) the values of SR(z) agree well up to ~3 km height. (b) The CF(z) profiles show agreement below ~3 km, where ~80 % of the clouds detected by CALIOP are detected by ALADIN as expected from the numerical experiment. (c) Above this height, the CDA$_{norm}^{obs}$(z) reduces to ~50 %. (d) On average, better sensitivity to lower clouds skews ALADIN's cloud peak height in pairs of ALADIN–CALIOP profiles by ~0.5±0.6 km downwards, but this effect does not alter the heights of polar stratospheric clouds and high tropical clouds thanks to their strong backscatter signals. (e) The temporal evolution of the observed CDA$_{norm}^{obs}$(z) does not reveal any statistically significant change during the considered period. This indicates that the instrument-related issues in ALADIN L0/L1 have been mitigated, at least down to the uncertainties of the following CDA$_{norm}^{obs}$(z) values: 68±12 %, 55±14 %, 34±14 %, 39±13 %, and 42±14 % estimated at 0.75, 2.25, 6.75, 8.75, and 10.25 km, respectively.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

DOE Cold Climate Heat Pump Challenge: Development, Metrics, and Early Field Observations

Space heating in residential buildings is a major contributor of Greenhouse Gas (GHG) emissions in the United States. New, advanced electric heat pumps are poised to provide a low carbon alternative to traditional fossil-based heating, especially in colder climates. Widespread deployment of cold climate heat pumps could help address the significant portion of building emissions and primary energy used in American households, but these gains will require broader acceptance from consumers and decision makers. As part of the “Energy, Emissions, and Equity” (E3) initiative, the U.S. Department of Energy launched the Cold Climate Heat Pump Challenge (CCHP) in 2021 to accelerate the deployment of the next generation of air source heat pumps. The Challenge is currently focused on residential, centrally ducted, electric heat pumps, with a nominal cooling capacity greater than or equal to 24,000 Btu/h (2 tons) and less than or equal to 65,000 Btu/h (5 tons). The Challenge specifications represent a best-in-class heat pump product that provides high-efficiency heating performance in cold climates, employs environmentally friendly low-Global Warming Potential (GWP) refrigerants, and is designed to be grid interactive. Spearheaded by DOE in partnership with the US Environmental Protection Agency (EPA) and Natural Resources Canada (NRCan), the CCHP Challenge brings together numerous major heat pump manufacturers and key stakeholders including utilities and state agencies across the country. In addition to performance testing in a laboratory environment, a key component of this research is evaluating the in-field performance of the prototype heat pumps developed as part of the Challenge. This paper will discuss the development of the CCHP Challenge, key performance specifications, and the energy and non-energy metrics that will be evaluated through the field study. The types of data that are being collected from the units installed in homes across North America as part of the field validation will be described, including details about the sensors and data acquisition system. Data cleaning and analysis methodologies will be discussed, along with early observations from the winter 2022-2023, spring and summer performance periods, and an assessment of non-energy metrics through pre- and post-installation homeowner surveys. Challenges uncovered and lessons learned throughout the process will also be discussed. Finally, the paper will discuss next steps for the second winter assessment period, areas needing additional research, and potential applications of the data collected and analyzed through this work.

Mendon, Vrushali V.↗
DOE OSTI2020

Albedo Data Sets for Bifacial PV Systems: Preprint

For use by the PV and financial communities to better estimate the performance and to reduce the risk of bifacial PV systems, data sets of ground albedo and associated meteorological data were developed by using existing measurement network data and data contributed by the PV industry. The data sets include time-series data as well as summary information of tabular monthly and yearly data and plots of monthly and hourly albedo values. Complete information is presented in a user’s guide and data are available for download from NREL’s DuraMAT website.

41 EE - Solar Energy Technologies Office (EE-4S)↗
DOE OSTI2024

MODAQ-BB (Modular Offshore Data Acquisition System - Blackbox) [SWR-24-72]

MODAQ-BB is a compact, rapid deployment data acquisition system that can withstand water depths up to 400m. Codenamed "BlackBox" (or simply BB), since its initial purpose was to track marine assets and record vital data streams that could later be recovered in the event of a mishap - much like a traditional black box used in aviation and shipping, the name has stuck. MODAQ BlackBox is a battery-operable microcontroller platform with internal inertial sensing, GPS, satellite communications, and additional I/O (input/output) support in a depth-rated pressure enclosure. Since BB is self-contained and relatively compact, it can be quickly deployed with minimal effort. The enclosure can be clamped to a tube (such as part of a railing) or mast in a location with unobstructed view of the sky using the available clamp accessory or a common hose clamp. While BB is designed to operate unattended, users can configure what data are uploaded and the frequency of satellite transmissions. Once data are uploaded, they can be relayed to an email distribution list, the MODAQ:Web operational dashboard, or a custom destination. BB has found utility in the National Laboratory of the Rockies' (NLR's) Waterpower projects beyond its original vision and has been configured and successfully deployed in more traditional data-gathering applications where a simple, battery-operated solution was indicated. As part of the MODAQ family, BB fills a space in the spectrum of missions that can be supported that were previously impractical using the traditional MODAQ hardware architecture due to factors such as weight, size, and cost.

Raye, Robert↗
DOE OSTI2024

The Data Acquisition System for Phase-III of the BeEST Experiment

The BeEST experiment is a precision laboratory search for physics beyond the standard model that measures the electron capture decay of 7 Be implanted into superconducting tunnel junction (STJ) detectors. For Phase-III of the experiment, we constructed a continuously sampling data acquisition system to extract pulse shape and timing information from 16 STJ pixels offline. Four additional pixels are read out with a fast list-mode digitizer, and one with a nuclear MCA already used in the earlier limit-setting phases of the experiment. Here, we present the performance of the data acquisition system and discuss the relative advantages of the different digitizers.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

MetaCat - metadata catalog for data management systems

Metadata management is one of three major areas and parts of functionality of scientific data management along with replica management and workflow management. Metadata is the information describing the data stored in a data item, a file or an object. It includes the data item provenance, recording conditions, format and other attributes. MetaCat is a metadata management database designed and developed for High Energy Physics experiments. As a component of a data management system, it’s main objectives are to provide efficient metadata storage and management and fast data items selection functionality. MetaCat is supposed to work on the scale of 100 million files (or objects) and beyond. The article will discuss the functionality of MetaCat and technological solutions used to implement the product.

Mandrichenko, Igor↗
DOE OSTI2024

MODAQ 2.0 (Modular Ocean Data AcQuisition System v2.0) [SWR-24-138]

MODAQ 2.0 is the continuation of the Modular Ocean/Offshore Data AcQuisition system which offers new benefits of being based on generally available hardware and open-source development tools. The MODAQ 2.0 project is the amalgamation of ROS packages, tools, and guidance for developing data acquisition and control applications for marine energy devices using a ROS2 based architecture. The MODAQ 2.0 Reference Design includes several packages that are targeted for the marine energy sector and allow most developers with a basic background in programming to spin up a high-quality DAQ and control system. It is comprised of multiple software repositories which can be found at the MODAQ2 GitHub Organization: https://github.com/NREL-MODAQ2

Nichols, Casey↗
DOE OSTI2020

Requirements Document for a Replacement Data Acquisition System

A flight vehicle is in the developmental stages. While in development flight testing is required by both the manufacturer and customer to rate design performance. The performance of the vehicle must fulfill the agreed upon requirements. Testing of a flight vehicle during operation requires various sensors in order to record and monitor performance for evaluation. A Data Acquisition System (DAQ) is needed in order to control and capture data from various sensors placed throughout the vehicle during flight. This DAQ will receive the analog data from the suite of sensors and convert them to a digital signal that can be transmitted to a recording station where the data will later be analyzed by test experts. Some of the key features to be included in the DAQ will be a self-check operation to verify the health of all sensors including the DAQ. The DAQ will also have the capability to store sensor calibration data files loaded during assembly and transmit those files to the recording station during flight. Breakthroughs in new technologies has allowed for higher resolution recording systems and faster sampling rates. Improved sensor data using a new Data Acquisition System (DAQ) can provide important answers to questions discovered during the last few flight tests. These questions arose when the flight system expressed unexpected behaviors. Having the capability of the new DAQ in flight tests would allow for more high-fidelity data to be captured from the system during flight tests and ultimately reducing the number of required flight test. Reducing the number of flight tests also reduce developmental cost and schedule.

42 ENGINEERING↗
DOE OSTI2022

Development of an open-source regional data assimilation system in PEcAn v. 1.7.2: application to carbon cycle reanalysis across the contiguous US using SIPNET

Abstract. The ability to monitor, understand, and predict the dynamics of the terrestrial carbon cycle requires the capacity to robustly and coherently synthesize multiple streams of information that each provide partial information about different pools and fluxes. In this study, we introduce a new terrestrial carbon cycle data assimilation system, built on the PEcAn model–data eco-informatics system, and its application for the development of a proof-of-concept carbon “reanalysis” product that harmonizes carbon pools (leaf, wood, soil) and fluxes (GPP, Ra, Rh, NEE) across the contiguous United States from 1986–2019. We first calibrated this system against plant trait and flux tower net ecosystem exchange (NEE) using a novel emulated hierarchical Bayesian approach. Next, we extended the Tobit–Wishart ensemble filter (TWEnF) state data assimilation (SDA) framework, a generalization of the common ensemble Kalman filter which accounts for censored data and provides a fully Bayesian estimate of model process error, to a regional-scale system with a calibrated localization. Combined with additional workflows for propagating parameter, initial condition, and driver uncertainty, this represents the most complete and robust uncertainty accounting available for terrestrial carbon models. Our initial reanalysis was run on an irregular grid of ∼ 500 points selected using a stratified sampling method to efficiently capture environmental heterogeneity. Remotely sensed observations of aboveground biomass (Landsat LandTrendr) and leaf area index (LAI) (MODIS MOD15) were sequentially assimilated into the SIPNET model. Reanalysis soil carbon, which was indirectly constrained based on modeled covariances, showed general agreement with SoilGrids, an independent soil carbon data product. Reanalysis NEE, which was constrained based on posterior ensemble weights, also showed good agreement with eddy flux tower NEE and reduced root mean square error (RMSE) compared to the calibrated forecast. Ultimately, PEcAn's new open-source regional data assimilation framework provides a scalable workflow for harmonizing multiple data constraints and providing a uniform synthetic platform for carbon monitoring, reporting, and verification (MRV) as well as accelerating terrestrial carbon cycle research.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Recent Experience with the CMS Data Management System

The CMS[1] experiment manages a large-scale data infrastructure, currently handling over 200 PB of disk and 500 PB of tape storage and transferring more than 1 PB of data per day on average between various WLCG[2] sites. Utilizing Rucio[3] for high-level data management, FTS[4] for data transfers, and a variety of storage and network technologies at the sites, CMS confronts inevitable challenges due to the system’s growing scale and evolving nature. Key challenges include managing transfer and storage failures, optimizing data distribution across different storages based on production and analysis needs, implementing necessary technology upgrades and migrations, and efficiently handling user requests. The data management team has established comprehensive monitoring to supervise this system and has successfully addressed many of these challenges. The team’s efforts aim to ensure data availability and protection, minimize failures and manual interventions, maximize transfer throughput and resource utilization, and provide reliable user support. This paper details the operational experience of CMS with its data management system in recent years, focusing on the encountered challenges, the effective strategies employed to overcome them and the ongoing challenges as we prepare for future demands.

Öztürk, Hasan [CERN]↗
DOE OSTI2025

Development and evaluation of a new 4DEnVar-based weakly coupled ocean data assimilation system in E3SMv2

The development, implementation, and evaluation of a new weakly coupled ocean data assimilation (WCODA) system for the fully coupled Energy Exascale Earth System Model version 2 (E3SMv2) utilizing the four-dimensional ensemble variational (4DEnVar) method are presented in this study. The 4DEnVar method, based on the dimension-reduced projection four-dimensional variational (DRP-4DVar) approach, replaces the adjoint model with the ensemble technique, thereby reducing computational demands. Monthly mean ocean temperature and salinity data from the EN4.2.1 reanalysis are integrated into the ocean component of E3SMv2 from 1950 to 2021 with the goal of providing realistic initial conditions for decadal predictions and predictability studies. The performance of the WCODA system is assessed using various metrics, including the reduction rate of the cost function, root mean square error (RMSE) differences, correlation differences, and model biases. Results indicate that the WCODA system effectively assimilates the reanalysis data into the climate model, consistently achieving negative reduction rates of the cost function and notable improvements in RMSE and correlation across various ocean layers and regions. Significant enhancements are observed in the upper ocean layers across the majority of global ocean regions, particularly in the north Atlantic, north Pacific, and Indian Ocean. Model biases in sea surface temperature and salinity are also substantially reduced. For sea surface temperature, cold biases in the north Pacific and north Atlantic are diminished by about 1–2 °C, and warm biases in the Southern Ocean are corrected by approximately 1.5–2.5 °C. In terms of salinity, improvements are observed with bias reductions of about 0.5–1 psu in the north Atlantic and north Pacific and up to 1.5 psu in parts of the Southern Ocean. The ultimate goal of the WCODA system is to advance the predictive capabilities of E3SM for subseasonal to decadal climate predictions, thereby supporting research on strategic energy-sector policies and planning.

54 ENVIRONMENTAL SCIENCES↗