SEARCH · Search NASA

Results for “data-intensive systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2025

RTN-011: Rubin Observatory Plans for an Early Science Program

This document outlines Rubin Observatory's plans for a dedicated \emph{Early Science Program} to enable high-impact science prior to the first annual data release of the Legacy Survey of Space and Time (LSST). Components of the Early Science Program include releasing science-grade commissioning data products via a series of ``Data Previews,'' ramping up of the transient alert stream during commissioning, implementing a program of incremental template generation to augment alert production in the early phases of the survey, and the first LSST Data Release, DR1, based on the first 6 months of data from the LSST. A detailed breakdown of which data products can be expected when is provided. The Rubin Operations team is working closely with the science community to optimize the Early Science Program for the time-domain and solar system science achievable in the first year of operations. This is a living document; both it and the Early Science Program will continue to evolve over the course of commissioning and pre-operations in response to the state of the as-built system and to community guidance.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2024

ExaFEL: extreme-scale real-time data processing for X-ray free electron laser science

ExaFEL is an HPC-capable X-ray Free Electron Laser (XFEL) data analysis software suite for both Serial Femtosecond Crystallography (SFX) and Single Particle Imaging (SPI) developed in collaboration with the Linac Coherent Lightsource (LCLS), Lawrence Berkeley National Laboratory (LBNL) and Los Alamos National Laboratory. ExaFEL supports real-time data analysis via a cross-facility workflow spanning LCLS and HPC centers such as NERSC and OLCF. Our work therefore constitutes initial path-finding for the US Department of Energy's (DOE) Integrated Research Infrastructure (IRI) program. We present the ExaFEL team's 7 years of experience in developing real-time XFEL data analysis software for the DOE's exascale supercomputers. We present our experiences and lessons learned with the Perlmutter and Frontier supercomputers. Furthermore we outline essential data center services (and the implications for institutional policy) required for real-time data analysis. Finally we summarize our software and performance engineering approaches and our experiences with NERSC's Perlmutter and OLCF's Frontier systems. This work is intended to be a practical blueprint for similar efforts in integrating exascale compute resources into other cross-facility workflows.

59 BASIC BIOLOGICAL SCIENCES↗

NASA NTRSDate not supplied

Expanding Repository Data Available For Sharing and Knowledge Discovery

Some of the hardest space biology and space health challenges require data-intensive, bioinformatic, meta-analytical, and computer-assisted research approaches. These challenges include examining interdisciplinary space life science research across experiments and across interacting spaceflight hazards (radiation, altered gravity, confinement, hostile-closed environments, distance-duration from Earth). The approaches to confront these challenges involve mining multiple datasets simultaneously from various hierarchical organizations of biological complexity, all while concurrently evaluating how experimental design factors affect endpoints of standard assays. To enable this field, it is essential that principal investigators (PIs) submit data in a structure so it can be maximally re-used. The purpose of the NASA Ames Life Sciences Data Archive (ALSDA) is to collect, curate, and make publicly available all non-human space-relevant biological data. ALSDA must also ensure data are open-access, and maximally findable, accessible, interoperable, and reusable (FAIR). The scope of ALSDA data collected and submitted by PIs include subject and study design metadata, assay metadata parameters, raw and processed assay data, assay imagery/video, and subject-experienced mission data telemetry (radiation, temperature, humidity, acoustics, vibrations, etc.). ALSDA recently integrated into a collaborative group of Open Science projects to facilitate a suite of new tools and workflows that will improve data submission, accessibility, and reusability by implementing digital data submission agreements, and adopting the data management system originally developed by NASA GeneLab. ALSDA intends to bring current biological repository data and all future collected data into this new scientific data reuse reality. This new suite of tools will enable ALSDA to deploy a science curation system using scientific assay configurations for the data submission portal. It will capture essential assay parameters according to established standards in each sub-field within biology. The submission portal expedites data collection by enhancing ease of PI data submission, providing a user interface and specificity for which data is to be submitted. Data submissions can be brought into cutting-edge informatic analysis portals to enable mining of physiological, behavioral, biochemical, and imaging datasets in conjunction with ‘omics-level datasets. As ALSDA datasets are submitted, curated, and published (e.g., micro-computed tomography, histology, pulse oximetry, serum metabolites, magnetic resonance imaging, intraocular pressure, novel object recognition, etc.), the merging together of spaceflight data along this multi-hierarchical complexity of biology will enable informatics and data-intensive approaches resulting in knowledge discoveries across missions, space hazards, and biological disciplines.

Biology↗

NASA NTRSDate not supplied

Expanding Repository Data Available For Sharing And Knowledge Discovery

life science↗

DOE OSTI2022

A Framework for Neural Network Inference on FPGA-Centric SmartNICs

FPGA-based SmartNICs offer great potential to significantly improve the performance of high-performance computing and warehouse data processing by tightly coupling support for reconfigurable data-intensive computation with cross-node communication, thereby mitigating the von Neumann bottleneck. Existing work, however, has been generally been limited in that it assumes an accelerator model where kernels are offloaded to SmartNICs, but most control tasks are left to the CPUs. This leads to frequent waiting, inferior performance, and scaling challenges. In this work, we propose a new distributive data-centric computing framework, named FCsN, for reconfigurable SmartNIC-based systems. Through a lightweight task circulation execution model and its implementation architecture, FCsN allows the complete detaching of kernel execution, control logic, system scheduling, and network communication to the SmarNICs. This boosts performance by: (i) avoiding the control dependency with CPUs and (ii) supporting streaming kernel execution and network communication at line rate and in a very fine-grained manner. We demonstrate the efficiency and flexibility of FCsN using various types of neural network applications including graph neural networks; as these last are both irregular and data intensive they offer an especially robust demonstration. Evaluations using commonly-used neural network models and graph datasets show that a system with the support of FCsN can achieve, on average, 144 speedups over the MPI-based standard CPU baselines.

Guo, Anqi↗

DOE OSTI2026

FitCache: A Transparent Drop-In Framework for Multi-Tier Caching to Accelerate Distributed Deep Learning Workloads

Training in Deep learning (DL) remains highly compute- and data-intensive, with I/O becoming a critical bottleneck as models and datasets scale. Recent studies report that data loading can dominate training time, especially on large-scale HPC systems with shared parallel file systems (PFS). Existing caching approaches either rely on single-tier designs or require intrusive modifications to training pipelines, limiting their portability and effectiveness. In this work, we present FitCache, a transparent drop-in framework for multi-tier caching to accelerate distributed DL training by coordinating fast local memory (e.g., DRAM, Persistent Memory (PMem)) and NVMe as hierarchical caches atop PFS. Our design adapts to hardware diversity, i.e., if NVMe is missing, memory transparently acts as a caching tier, ensuring stable performance. FitCache transparently intercepts I/O requests and issues concurrent fetches across all tiers, returning data from the fastest responder without centralized metadata or static redirection paths. FitCache adapts to dynamic workloads and heterogeneous clusters while maintaining POSIX compatibility. Experiments on Frontier (2048 GPUs) and smaller research clusters show that FitCache reduces training time by up to 40% and per-batch I/O latency by up to 71.6% compared to Lustre Orion PFS, offering a drop-in solution for scalable DL training.

Hu, Guangxing [ORNL] (ORCID:0009000283203614)↗

DOE OSTI2024

Employing artificial intelligence to steer exascale workflows with colmena

Computational workflows are a common class of application on supercomputers, yet the loosely coupled and heterogeneous nature of workflows often fails to take full advantage of their capabilities. We created Colmena to leverage the massive parallelism of a supercomputer by using Artificial Intelligence (AI) to learn from and adapt a workflow as it executes. Colmena allows scientists to define how their application should respond to events (e.g., task completion) as a series of cooperative agents. In this paper, we describe the design of Colmena, the challenges we overcame while deploying applications on exascale systems, and the science workflows we have enhanced through interweaving AI. The scaling challenges we discuss include developing steering strategies that maximize node utilization, introducing data fabrics that reduce communication overhead of data-intensive tasks, and implementing workflow tasks that cache costly operations between invocations. These innovations coupled with a variety of application patterns accessible through our agent-based steering model have enabled science advances in chemistry, biophysics, and materials science using different types of AI. In conclusion, our vision is that Colmena will spur creative solutions that harness AI across many domains of scientific computing.

Workflows↗

DOE OSTI2025

Pushing the limits of NAND technology scaling with ferroelectrics

Artificial intelligence (AI) continues to drive transformative advancements across various industries. The data-intensive nature of AI training (and inferencing) has resulted in the generation of unprecedented volumes of data with machine-generated content surpassing human-generated data by more than 100-fold in 2025. Efficiently managing this data influx necessitates advanced digital storage technologies. However, traditional NAND flash memory, which is critical for supporting data flows in AI systems—alongside high-bandwidth memory, for AI training—faces fundamental scaling limitations as it approaches the 1000-layer milestone, encompassing more than 40 trillion transistors. This article delves into the potential of hafnia-based ferroelectric materials as a breakthrough solution to these challenges. Recent advancements indicate that the intrinsic limitations of ferroelectric field-effect transistors (FEFETs) can be mitigated through material and device-level engineering. These advancements enable FEFETs to meet the stringent density, reliability, and scalability requirements of future three-dimensional NAND technology. The role of ferroelectrics in addressing NAND scaling challenges and expanding storage capabilities presents a promising avenue for meeting the storage demands of the AI-driven era.

3D NAND↗

DOE OSTI2025

2020 Budget Request for the DOE Computational Science Graduate Fellowship (CSGF) Grant

The Department of Energy Computational Science Graduate Fellowship (DOE CSGF) is essential for addressing the increasingly complex national workforce demands stemming from the growth of computational science and engineering challenges. Computational science and engineering (CSE) takes a multidisciplinary approach that utilizes scientific computing to tackle practical problems and provide technical tools across the spectrum of scientific discovery. The DOE CSGF specifically highlights high-performance computing (HPC) as a critical enabling technology in CSE, driving advancements in science and engineering that are vital to both the DOE and the broader economy. Over the past half-century, HPC has been an essential tool for DOE’s success. During this period, important missions, such as nuclear stockpile stewardship, have turned to HPC as an essential technology. Entire science disciplines have been transformed through the augmentation of scientific observation via HPC. At government laboratories, academic institutions, and in industry, DOE CSGF alumni are helping push traditional HPC boundaries while contributing to discoveries in high-energy physics, quantum information systems, fusion-reactor design, machine learning, additive manufacturing, nano materials for next-generation batteries and transistors, and advanced nuclear reactor modeling. In addition, HPC is used to address national health needs that will eventually point to cures both by helping cancer researchers manage and analyze huge troves of data, by simulating biological mechanisms, and by accelerating drug development. A 2023 report from the ASCAC Subcommittee on American Competitiveness and Innovation to the ASCR office, “Can the United States Maintain Its Leadership in High-Performance Computing?” says of the Program, “The CSGF program provides a barometer for disciplines that will be of interest to future DOE computing. Computational biology, machine learning, and quantum computing are among the subjects that began to swell in the ranks of CSGF applicants before the labs were hiring as high a percentage of employees in these categories.” The explosion of scientific and technological data has heightened the demand for advanced high-performance computing (HPC) to transform these data into meaningful scientific insights. As access to vast amounts of data increases, the fields of Machine Learning and Artificial Intelligence are experiencing a resurgence, enhancing the established practices of computational modeling and simulation. In its September 2020 subcommittee report on "AI/ML, Data Intensive Science, and High-Performance Computing," the DOE Advanced Scientific Computing Advisory Committee (ASCAC) specifically called for a fellowship program to train computational and data scientists to address exascale and data-intensive computing challenges. This integration of empirical and theoretical modeling will increasingly guide federal policymakers in making decisions that impact American society and future generations. It demands a workforce of highly skilled and intellectually agile computational scientists capable of navigating the rapid advancements in scientific computing within the DOE National Laboratory research environment. The DOE CSGF program has consistently addressed this critical need.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

The Vera C. Rubin Observatory Data Preview 1

We present Rubin Data Preview 1 (DP1), the first data from the National Science Foundation–Department of Energy Vera C. Rubin Observatory, comprising raw and calibrated single-epoch images, coadds, difference images, detection catalogs, and ancillary data products. DP1 is based on 1792 optical–near-infrared exposures acquired over 48 distinct nights by the Rubin Commissioning Camera (LSSTComCam) on the Simonyi Survey Telescope at the Summit Facility on Cerro Pachón, Chile in late 2024. DP1 covers ∼15 deg 2 distributed across seven roughly equal-sized noncontiguous fields, each independently observed in six broad photometric bands, ugrizy. The median FWHM of the point-spread function across all bands is approximately 1"14, with the sharpest images reaching about 0." 58. The 5σ point-source depths for coadded images in the deepest field, the Extended Chandra Deep Field South, are u = 24.55, g = 26.18, r = 25.96, i = 25.71, z = 25.07, and y = 23.1. Other fields are no more than 2.2 mag shallower in any band, where they have nonzero coverage. DP1 contains approximately 2.3 million distinct astrophysical objects, of which 1.6 million are extended in at least one band in coadds, and 431 solar system objects, of which 93 are new discoveries. DP1 is approximately 3.5 TB in size and is available to Vera C. Rubin Observatory data rights holders via the Rubin Science Platform, a cloud-based environment for the analysis of petascale astronomical data. While small compared to future LSST releases, its high quality and diversity of data support a broad range of early science investigations ahead of full operations in 2026.

Ground-based astronomy↗

DOE OSTI2025

RTN-095: The Vera C. Rubin Observatory Data Preview 1

We present Rubin Data Preview 1 (DP1), the first release of data from the NSF-DOE Vera C. Rubin Observatory, consisting of raw and calibrated single-epoch images, coadds, difference images, detection catalogs, and other derived data products. DP1 is based on 1792 science-grade optical/near-infrared exposures acquired over 48 distinct nights by the Rubin Commissioning Camera, LSSTComCam, on the Simonyi Survey Telescope at the Summit Facility on Cerro Pachón, Chile during the first on-sky commissioning campaign in late 2024. DP1 covers a total of ~15 sq. deg. over seven roughly equally-sized non-contiguous fields, each independently observed in six broad photometric bands, ugrizy, spanning a range of stellar densities and latitudes and overlapping with external reference datasets. The median image quality across all bands, measured by the FWHM of the point-spread function, is approximately 1.13 arcseconds, with the sharpest images reaching about 0.65 arcseconds. DP1 contains approximately 2.3 million distinct astrophysical objects, of which 1.6 million are extended in at least one band, and 431 solar system objects, of which 93 are new discoveries. DP1 is approximately 3.5 TB in size and available to Rubin data rights holders via the Rubin Science Platform, a cloud-based environment for the analysis of petascale astronomical data. While small compared to future LSST releases, its high quality and diversity of data support a broad range of early science investigations across all four LSST themes, providing a valuable opportunity to engage with Rubin data ahead of the start of full operations in late 2025.

79 ASTRONOMY AND ASTROPHYSICS↗

NASA NTRS1984

Nonparametric analysis of Minnesota spruce and aspen tree data and LANDSAT data

The application of nonparametric methods in data-intensive problems faced by NASA is described. The theoretical development of efficient multivariate density estimators and the novel use of color graphics workstations are reviewed. The use of nonparametric density estimates for data representation and for Bayesian classification are described and illustrated. Progress in building a data analysis system in a workstation environment is reviewed and preliminary runs presented.

Scott, D. W.↗

NASA NTRS2002

Enabling Earth Science: The Facilities and People of the NCCS

The NCCS's mass data storage system allows scientists to store and manage the vast amounts of data generated by these computations, and its high-speed network connections allow the data to be accessed quickly from the NCCS archives. Some NCCS users perform studies that are directly related to their ability to run computationally expensive and data-intensive simulations. Because the number and type of questions scientists research often are limited by computing power, the NCCS continually pursues the latest technologies in computing, mass storage, and networking technologies. Just as important as the processors, tapes, and routers of the NCCS are the personnel who administer this hardware, create and manage accounts, maintain security, and assist the scientists, often working one on one with them.

Source record↗

NASA NTRS2001

Phased-Array Satcom Antennas Developed for Aeronautical Applications

The Advanced Communications (AC) for Aeronautics research at the NASA Glenn Research Center integrates both aeronautics and space communications technologies to achieve the national objective of upgrading the present National Airspace System infrastructure by responding to the agency's aviation capacity and safety goals. One concept for future air traffic management, free flight, presents a significantly increased demand for communications systems capacity and performance in comparison to current air traffic management practices. Current aeronautical communications systems are incapable of supporting the anticipated demands, and the new digital data communications links that are being developed, or are in the early stages of implementation, are not primarily designed to carry the data-intensive free flight air traffic management (ATM) communications loads. Emerging satellite communications technologies are the best potential long-term solution to provide the capacity and performance necessary to enable a mature free flight concept to be deployed. NASA AC/ATM funded the development of a Boeing-designed Ku-band transmit phased-array antenna, a combined in-house and contract effort. Glenn designed and integrated an Aeronautical Mobile Satellite Communications terminal based on the transmit phased-array antenna and a companion receive phased-array antenna previously developed by Boeing.

Kerczewski, Robert J.↗

NASA NTRS2007

High-Capacity Communications from Martian Distances

High capacity communications from Martian distances, required for the envisioned human exploration and desirable for data-intensive science missions, is challenging. NASA s Deep Space Network currently requires large antennas to close RF telemetry links operating at kilobit-per-second data rates. To accommodate higher rate communications, NASA is considering means to achieve greater effective aperture at its ground stations. This report, focusing on the return link from Mars to Earth, demonstrates that without excessive research and development expenditure, operational Mars-to-Earth RF communications systems can achieve data rates up to 1 Gbps by 2020 using technology that today is at technology readiness level (TRL) 4-5. Advanced technology to achieve the needed increase in spacecraft power and transmit aperture is feasible at an only moderate increase in spacecraft mass and technology risk. In addition, both power-efficient, near-capacity coding and modulation and greater aperture from the DSN array will be required. In accord with these results and conclusions, investment in the following technologies is recommended:(1) lightweight (1 kg/sq m density) spacecraft antenna systems; (2) a Ka-band receive ground array consisting of relatively small (10-15 m) antennas; (3) coding and modulation technology that reduces spacecraft power by at least 3 dB; and (4) efficient generation of kilowatt-level spacecraft RF power.

Williams, W. Dan↗

NASA NTRS2016

Development and Characterization of a Small Spacecraft Electro-Optic Scanner for Free-Space Laser Communications

Emergent data-intensive missions coupled with dramatic reductions in spacecraft size plus an increasing number of space-based missions necessitates new high performance, compact and low cost communications technology. Free space optical communications offer advantages including orders of magnitude increase for data rate performance, increased security, immunity to jamming and lack of frequency allocation requirements when compared with conventional radio frequency (RF) means. The spatial coherence and low divergence associated with the optical frequencies of laser communications lends themselves to superior performance, but this increased directionality also creates one of the primary technical challenges in establishing a laser communications link by repeatedly and reliably pointing the beam onto the receive aperture. Several solutions have emerged from wide angle (slow) mechanical articulation systems, fine (fast) steering mirrors and rotating prisms, inertial compensation gyros and vibration isolation cancellation systems, but each requires moving components and imparts a measured amount of burden on the host platform. The complexity, cost and size of current mechanically scanned solutions limits their platform applicability, and restricts the feasibility of deploying optical communications payloads on very compact spacecraft employing critical systems. A high speed, wide angle, non-mechanical solution is therefore desirable. The purpose of this work is to share the development, testing, and demonstration of a breadboard prototype electro-optic (EO) scanned laser-communication link (see Figure 1). This demonstration is a step toward realizing ultra-low Size, Weight and Power (SWaP) SmallSat/MicroSat EO non-mechanical laser beam steering modules for high bandwidth ( greater than Gbps) free-space data links operating in the 1550 nm wavelength bands. The elimination of all moving parts will dramatically reduce SWaP and cost, increase component lifetime and reliability, and simplify the system design of laser communication modules. This paper describes the target mission architectures and requirements (few cubic centimeters of volume, 10's of grams of weight with milliwatts of power) and design of the beam steering module. Laboratory metrology is used to determine the component performance including horizontal and vertical resolution (20urad) as a function of control voltage (see Figure 2), transition time (0.1-1ms), pointing repeatability and optic insertion loss. A test bed system demonstration, including a full laser communications link, is conducted. The capabilities of this new EO beam steerer provide an opportunity to dramatically improve space communications through increased utilization of laser technology on smaller platforms than were previously attainable.

Laser Optical Communications↗

NASA NTRS2015

Climate Model Diagnostic Analyzer

The comprehensive and innovative evaluation of climate models with newly available global observations is critically needed for the improvement of climate model current-state representation and future-state predictability. A climate model diagnostic evaluation process requires physics-based multi-variable analyses that typically involve large-volume and heterogeneous datasets, making them both computation- and data-intensive. With an exploratory nature of climate data analyses and an explosive growth of datasets and service tools, scientists are struggling to keep track of their datasets, tools, and execution/study history, let alone sharing them with others. In response, we have developed a cloud-enabled, provenance-supported, web-service system called Climate Model Diagnostic Analyzer (CMDA). CMDA enables the physics-based, multivariable model performance evaluations and diagnoses through the comprehensive and synergistic use of multiple observational data, reanalysis data, and model outputs. At the same time, CMDA provides a crowd-sourcing space where scientists can organize their work efficiently and share their work with others. CMDA is empowered by many current state-of-the-art software packages in web service, provenance, and semantic search.

cloud computing↗

NASA NTRSDate not supplied

Enabling Space Biological Knowledge Discovery Through Image and Video Data Sharing

Increased biomedical risks and challenges associated with deep space missions and experiments (cis-Lunar, Mars transit/surface) require new knowledge discovery and development of novel ecosystems. Supporting distant and long-duration missions and experiments requires biological data (from yeast, microbes, fruit flies, C. elegans, plants, crops, rodents, humans) be findable, accessible, interoperable, reusable (FAIR), and maximally open-access. As data-intensive, bioinformatic, meta-analytical, and computer-assisted approaches continue to be a centerpiece of modern research, the NASA Biological and Physical Sciences division is expanding its Open Science capabilities beyond NASA GeneLab. The NASA Ames Life Sciences Data Archive (ALSDA) is a repository which is responsible for collecting and access to space biological imagery and video, alongside tabular and environmental data. In this presentation, we will discuss strategies dealing with archiving, curating, and accessibility of images from very distinct imaging modalities (e.g., micro-computed tomography, magnetic resonance imaging, photographic images of plants, fluorescence microscopy, behavioral videos, etc.). There are two main challenges: 1. Open-source data storage and 2. Metadata related to the imagery-video. Both have been solved by leveraging two existing open-source systems. For data storage, ALSDA is utilizing components through the Open Microscopy Environment (OME), which can read most imaging proprietary formats and display on a web interface complex multidimensional images (Z stack, multi-channel, temporal, spectral). Most technical metadata from imaging modalities are captured seamlessly. For metadata capturing experimental details, ALSDA (like GeneLab) uses the ISA-Tab specification which relies on the ISA data model to order and classify metadata. The ISA data model uses a tree structure with three files to capture the metadata: The top layer is the Investigations file, the second layer is the Study file(s), and the last layer is the Assay file(s). We believe such an approach may be useful for other types of image research data from other investigators in the AGU community.

imaging↗