SEARCH · Search NASA

Results for “data processing and archiving”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

All sources NASA NTRS DOE OSTI

DOE OSTI2026

Field and Model Data Associated with the Manuscript “Drivers of Streamflow Intermittency in Humid Regions: 1. Evaluating Above- and Below-ground Controls of Flow Persistence in a Forested Catchment”

This package contains field data, modeling files, and scripts supporting the investigation of the drivers of streamflow intermittency in a forested catchment. It includes the field data collected from electrical resistivity tomography (ERT) surveys, ground penetrating radar (GPR), continuous self-potential (SP) monitoring, electromagnetic (EM) imaging, groundwater and stilling well. In addition, it contains the data and results of the coupled water- and electrical-flow model developed using the COMSOL Multiphysics and Advanced Terrestrial Simulator (ATS), as well as software files and Jupyter notebooks used to process the data and generate figures in the manuscript submitted for peer review. The data archive is organized in the following directories: 1) Climate Includes hourly precipitation and daily evapotranspiration time series (2024 – 2025) provided as CSV files, alongside a text file detailing dataset units. 2) Coupled_model Contains two subfolders: Synthetic and Field_Application subfolder. Synthetic subfolder contains the ATS XML input script (can be opened using any code editor) for the four synthetic hydrological cases tested (Connected and gaining, Connected and losing, Disconnected and losing, and dry stream). It also includes other experimental cases to test the influence of precipitation and concentration gradient. For each synthetic case, the flow model simulation is executed using the ATS XML scripts and the included Python script (generate_data_set.py) to convert ATS output to COMSOL-ready input. COMSOL Multiphysics template (.mph can be opened with the commercial software COMSOL and requires a license) is executed using the ATS output data to simulate the potential field. It also includes the Synthetic_model_plot.ipynb (can be opened using any code editor) to visualize the SP result and generate manuscript figures. The data subfolder contains mesh files to run both the ATS (.exo and .stl files can be viewed using Paraview; .h5 files can be opened using HDFView software and h5py Python package) and COMSOL models. Field_Application subfolder contains two subfolders: ES_MDA_inversion and Final_Model. ES_MDA_inversion contains the Python script (.py can be opened using any code editor) and SP observation data used to run the Ensemble Smoother with Multiple Data Assimilation (ES-MDA) inversion sequence to get the optimal model parameters. The Final_model subfolder contains the ATS XML input scripts, data files, output data for the two SP sites. The same workflow steps outlined for the Synthetic subfolder apply here. It also contains the Jupyter notebook (Plot_final_calib.ipynb) to visualize the results of the modeled SP, stream-groundwater exchange and moisture content. 3) Discharge Includes the electrical conductivity (EC) time series (provided as CSV files) from salt slug injections. It also includes the Jupyter notebook (Discharge_process.ipynyb) used to estimate discharge. All discharge measurements collated into rating_curve_processed.csv 4) EM Contains the CSV file of the EM data from the DUALEM-42, including spatial coordinates (x, y, z), apparent conductivity, and in-phase measurements at 2 m coil separations for horizontal coplanar (HCP) and perpendicular (PRP) geometries. 5) ERT Contains raw resistivity data (provided as CSV files), spatial location of each of the electrodes (provided as CSV files), and files used for the resistivity inversion (.resipy can be opened with the open-source ResIPy software). 6) GPR Includes GPR field datasets collected at 100 MHz and 250 MHz antenna frequencies, along with the processing/interpretation project file (GPR_process.gpz can be viewed using EKKO_Project 6, a commercial software by Sensors & Software that requires a license). 7) Slug_test Includes the slug test data at all the groundwater wells provided as CSV files, as well as the Jupyter notebook (Slug_test.ipynb) for calculating hydraulic conductivity. 8) SP Contains the SP data collected in field at the two SP sites (one in the perennial reach and the other in the intermittent reach), provided as DAT files. 9) Well_data Contains two subfolders: 1) Raw, which provides unprocessed pressure, electrical conductivity and temperature timeseries downloaded from the loggers in all the groundwater and stilling wells, and 2) Processed, which contains sorted, QA/QC timeseries data for each well. The data archive also contains data_process.ipynb, a Jupyter notebook used for field data analysis and generating figures (plotting well, SP, climate, and discharge data, as well as calculating head gradient at sites with nested groundwater wells). It also includes DTW.ipynb, a Jupyter notebook containing the code for the dynamic time warping (DTW) with sliding window to evaluate SP signal synchronicity.

ATS↗

NASA NTRS2017

AIRS-Only Product in Giovanni for Exploring Up-to-Date AIRS Observation and Comparing with AIRS+AMSU Product

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) has been the home of processing, archiving, and distribution services for the Atmospheric Infrared Sounder (AIRS) mission since its launch in 2002 for global observations of the atmospheric state. Giovanni, a Web-based application developed by the GES DISC, provides a simple and intuitive way to visualize, analyze, and access vast amounts of Earth science remote sensing data without having to download the data. Most important AIRS variables, including temperature and humidity profiles, outgoing longwave radiation, cloud properties, and trace gases, are available in Giovanni. AIRS is an instrument suite comprised of a hyperspectral infrared instrument (AIRS) and two multichannel microwave instruments, the Advanced Microwave Sounding Unit (AMSU) and the Humidity Sounder for Brazil (HSB). As HSB ceased operation in the very early stages of the AIRS mission, the AIRS project operates two parallel retrieval algorithms: one using both IR and MW measurements (AIRS+AMSU) and the other using only IR measurements (AIRS-only), which covers most of the mission duration. The AIRS+AMSU product is better quality, and the variables in Giovanni are from this product. However, generation of the AIRS+AMSU product has been suspended since the AMSU instrument anomaly occurred in late September 2016. To continue exploring up-to-date AIRS observations, the same set of variables from the AIRS-only product have been added to Giovanni by the GES DSIC. This will also support comparison of AIRS-only with AIRS+AMSU retrievals. In this presentation, we demonstrate the visualization of the AIRS-only product and plots/statistics of comparison with AIRS+AMSU product using Giovanni.

AIRS↗

NASA NTRS2016

Constructing an AIRS Climatology for Data Visualization and Analysis to Serve the Climate Science and Application Communities

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is the home of processing, archiving, and distribution services for NASA sounders: the present Aqua AIRS mission and the succeeding SNPP CrIS mission. The AIRS mission is entering its 15th year of global observations of the atmospheric state, including temperature and humidity profiles, outgoing longwave radiation, cloud properties, and trace gases. The GES DISC, in collaboration with the AIRS Project, released product from the version 6 algorithm in early 2013. Giovanni, a Web-based application developed by the GES DISC, provides a simple and intuitive way to visualize, analyze, and access vast amounts of Earth science remote sensing data without having to download the data. Most important variables from version 6 AIRS product are available in Giovanni. We are developing a climatology product using 14-year AIRS retrievals. The study can be a good start for the long term climatology from NASA sounders: the AIRS and the succeeding CrIS. This presentation will show the impacts to the climatology product from different aggregation methods. The climatology can serve climate science and application communities in data visualization and analysis, which will be demonstrated using a variety of functions in version 4 Giovanni. The highlights of these functions include user-defined monthly and seasonal climatology, inter annual seasonal time series, anomaly analysis.

AIRS↗

NASA NTRS2002

TRMM Data Mining Service at the Goddard Earth Sciences (GES) DISC DAAC Tropical Rainfall Measuring Mission (TRMM)

TRMM has acquired more than four years of data since its launch in November 1997. All TRMM standard products are processed by the TRMM Science Data and Information System (TSDIS) and archived and distributed to general users by the GES DAAC. Table 1 shows the total archive and distribution as of February 28, 2002. The Utilization Ratio (UR), defined as the ratio of the number of distributed files to the number of archived files, of the TRMM standard products has been steadily increasing since 1998 and is currently at 6.98.

Source record↗

NASA NTRS2012

Landsat Science: 40 Years of Innovation and Opportunity

Landsat satellites have provided unparalleled Earth-observing data for nearly 40 years, allowing scientists to describe, monitor and model the global environment during a period of time that has seen dramatic changes in population growth, land use, and climate. The success of the Landsat program can be attributed to well-designed instrument specifications, astute engineering, comprehensive global acquisition and calibration strategies, and innovative scientists who have developed analytical techniques and applications to address a wide range of needs at local to global scales (e.g., crop production, water resource management, human health and environmental quality, urbanization, deforestation and biodiversity). Early Landsat contributions included inventories of natural resources and land cover classification maps, which were initially prepared by a visual interpretation of Landsat imagery. Over time, advances in computer technology facilitated the development of sophisticated image processing algorithms and complex ecosystem modeling, enabling scientists to create accurate, reproducible, and more realistic simulations of biogeochemical processes (e.g., plant production and ecosystem dynamics). Today, the Landsat data archive is freely available for download through the USGS, creating new opportunities for scientists to generate global image datasets, develop new change detection algorithms, and provide products in support of operational programs such as Reducing Emissions from Deforestation and Forest Degradation in Developing Countries (REDD). In particular, the use of dense (approximately annual) time series to characterize both rapid and progressive landscape change has yielded new insights into how the land environment is responding to anthropogenic and natural pressures. The launch of the Landsat Data Continuity Mission (LDCM) satellite in 2012 will continue to propel innovative Landsat science.

Cook, Bruce D.↗

NASA NTRSDate not supplied

Deep Learning System for Efficient Processing of Geostationary Satellite Imagery

Improved capabilities of Earth monitoring satellites are enabling a wide range of studies on the environmental effects of climate change, often leveraging the recent advancements in machine learning. At the same time, the new capabilities, including higher spatial resolution and temporal frequency, are expanding the amount of data generated at exponential rates. Further, a large majority of archived datasets generated by scientific processing is never used. This motivates the development of an efficient machine learning system for end-to-end processing of multi-level satellite datasets, from level 1 top of atmosphere observations to user friendly environmental variables of interest. Using current generation geostationary satellites GOES-16/17 (NOAA/NASA), and Himawari-8/9 (JAXA), we present an interchangeable set of machine models to perform spectral adjustment among sensors, physical model emulation, LEO-GEO emulation, and optical flow in a high performance computing environment. We use these tools on the NASA Earth eXchange (NEX) to generate consistent virtual observations across sensors, perform atmospheric correction and cloud detection, and estimate surface reflectance, surface temperature and atmospheric winds. This approach aims to improve the robustness of remotely sensed data processing by learning from diverse sets of observations while enabling near real-time and on-demand capabilities.

Thomas Vandal↗

NASA NTRS2013

Data Center at NICT

The Data Center at the National Institute of Information and Communications Technology (NICT) archives and releases the databases and analysis results processed at the Correlator and the Analysis Center at NICT. Regular VLBI sessions of the Key Stone Project VLBI Network were the primary objective of the Data Center. These regular sessions continued until the end of November 2001. In addition to the Key Stone Project VLBI sessions, NICT has been conducting geodetic VLBI sessions for various purposes, and these data are also archived and released by the Data Center.

Ichikawa, Ryuichi↗

NASA NTRS2015

A Summary of Proposed Changes to the Current ICARTT Format Standards and their Implications to Future Airborne Studies

The Atmospheric Science Data Center (ASDC) at NASA Langley Research Center is responsible for the ingest, archive, and distribution of NASA Earth Science data in the areas of radiation budget, clouds, aerosols, and tropospheric chemistry. The ASDC specializes in atmospheric data that is important to understanding the causes and processes of global climate change and the consequences of human activities on the climate. The ASDC currently supports more than 44 projects and has over 1,700 archived data sets, which increase daily. ASDC customers include scientists, researchers, federal, state, and local governments, academia, industry, and application users, the remote sensing community, and the general public.

Northup, Emily↗

NASA NTRS2015

Toolsets for Airborne Data (TAD): Improving Machine Readability for ICARTT Data Files

Early, Amanda Benson↗

NASA NTRS1999

Optical Properties of Aerosols from Long Term Ground-Based Aeronet Measurements

AERONET is an optical ground-based aerosol monitoring network and data archive supported by NASA's Earth Observing System and expanded by federation with many non-NASA institutions including AEROCAN (AERONET CANada) and PHOTON (PHOtometrie pour le Traiteinent Operatonnel de Normalisation Satellitaire). The network hardware consists of identical automatic sun-sky scanning spectral radiometers owned by national agencies and universities purchased for their own monitoring and research objectives. Data are transmitted hourly through the data collection system (DCS) on board the geostationary meteorological satellites GMS, GOES and METEOSAT and received in a common archive for daily processing utilizing a peer reviewed series of algorithms thus imposing a standardization and quality control of the product data base. Data from this collaboration provides globally distributed near real time observations of aerosol spectral optical depths, aerosol size distributions, and precipitable water in diverse aerosol regimes. Access to the AERONET data base has shifted from the interactive program 'demonstrat' (reserved for PI's) to the AERONET homepage allowing faster access and greater development for GIS object oriented retrievals and analysis with companion geocoded data sets from satellites, LIDAR and solar flux measurements for example. We feel that a significant yet under utilized component of the AERONET data base are inversion products made from hourly principal plane and almucanter measurements. The current inversions have been shown to retrieve aerosol volume size distributions. A significant enhancement to the inversion code has been developed and is presented in these proceedings.

Holben, B. N.↗

NASA NTRSDate not supplied

Genesis Solar Wind Sample Curation Documentation

Introduction: A scientist with experience as a sample science analyst, provider of flight hardware for multiple missions, and senior engineer in an ISO 2000-rated manufacturing plant has described the timeline of key participants in any PI-led sample return mission, the breadth of the organizations involved [1,2], and, of interest to this meeting, choosing the types of data to preserve and issues of future data accessibility. This work broadens that perspective by giving similar lessons from Genesis sample curation point-of-view. Curation participation regarding data gathering was part of the mission review process from the beginning. Genesis’ story illustrates outcome of several choices about types of data to record and preserve. Precision analysis of solar wind atoms captured in pure, ultraclean substrates is the driving science goal; therefore, detailed documentation was captured from all mission and curation phases and from investigator laboratories because these processes affect the final analytical results [3]. Pre-flight: Design and fabrication of the spacecraft. Like many modern small sample return missions, Genesis was a tightly managed team integrated across science, engineering and curation. Communication across the team was excellent, and, for the most part, the hands-on engineering technicians understood the impacts of “small choices” they routinely make, and the eyes-on oversight of manufacturing processes by scientists was mindful of details. The payload was designed by the Jet Propulsion Laboratory and the spacecraft by Lockheed Martin. Solar wind collectors and instruments were fabricated by multiple vendors and laboratories. The main portion of the payload was assembled at JSC. Fabrication procedures and contamination-control data (with witness coupons) were stored primarily at JSC. The original composition, dimensions and configuration of components, results of thermal testing, etc. are still needed for interpretation of analytical data. At times, these must be estimated from secondary information acquired pre-flight. Moreover, some files (e.g., original 3-D models and early Powerpoint) cannot be opened using software. Archived curation data includes 2-D drawings, material usage lists, QA documentation and analyses of consumables used during fabrication. Important chemical information still resides in archived hardware, paints and lubricants, material coupons, cleaning coupons, environmental witness plates and reference materials from manufacturing facilities. Purity and cleanliness of collector substrates. Semi-conductor vendors provided surface cleanliness data and some purity data. Purity for specific elements of interest was verified by science team members in their laboratories [4]. Curation archived procurement and shipping records, analysis reports, and non-proprietary fabrication data. A physical archive of flight collector reference materials is maintained for future use so additional data can be collected as analytical techniques improve. These are of increased value due to the hard landing upon re-entry. Cleaning and cleanliness assessments of flight hardware. Cleaning of the science canister payload was performed at JSC in a dedicated ISO 4 cleanroom using ultrapure water (UPW). The cleanliness of this UPW was monitored throughout processing. The archive for the clean lab also includes airborne particle counts, airborne molecular and inorganic contamination measurements as well as cleanroom construction material coupons and witness coupons. Hardware cleanliness was assessed by particle counts in rinse water batches. This information is recorded in batch cleaning forms and logbooks, and are, perhaps, of decreased value due to the hard landing. Post-flight: Curation-generated data. The curation handling history of each Genesis sample is documented in a typical astromaterials sample database which captures sample location, physical description and characterization data. Samples have a “shelf life”. Crucial to the preservation of samples is ongoing documentation of the sample environment, initially under curatorial control but is now a separate facility function with requires coordination. PI-generated data. Data on sample characterization and cleaning techniques continues to be generated by sample users [5]. These are often captured in LPSC abstracts, but these “engineering” results often are not publishable as stand-alone papers. We are actively looking for ways to make this information more accessible to users. Ion implants into samples have aided science return and can be shared among investigators. These (and similar) materials should be added to the curatorial collection with appropriate process and characterization data generated externally. Summary: Complete data archives for returned astromaterial samples must be broad in types and formats, and inclusive of environmental monitoring.

Genesis↗

NASA NTRS1998

SeaWiFS Science Algorithm Flow Chart

This flow chart describes the baseline science algorithms for the Sea-viewing Wide Field-of-view Sensor (SeaWiFS) Data Processing System (SDPS). As such, it includes only processing steps used in the generation of the operational products that are archived by NASA's Goddard Space Flight Center (GSFC) Distributed Active Archive Center (DAAC). It is meant to provide the reader with a basic understanding of the scientific algorithm steps applied to SeaWiFS data. It does not include non-science steps, such as format conversions, and places the greatest emphasis on the geophysical calculations of the level-2 processing. Finally, the flow chart reflects the logic sequences and the conditional tests of the software so that it may be used to evaluate the fidelity of the implementation of the scientific algorithm. In many cases however, the chart may deviate from the details of the software implementation so as to simplify the presentation.

Darzi, Michael↗

NASA NTRS2016

Description of the TCERT Vetting Reports for Data Release 25

This document, the Kepler Instrument Handbook (KIH), is for Kepler and K2 observers, which includes the Kepler Science Team, Guest Observers (GOs), and astronomers doing archival research on Kepler and K2 data in NASAs Astrophysics Data Analysis Program (ADAP). The KIH provides information about the design, performance, and operational constraints of the Kepler flight hardware and software, and an overview of the pixel data sets available. The KIH is meant to be read with these companion documents:1. Kepler Data Processing Handbook (KSCI-19081) or KDPH (Jenkins et al., 2016). The KDPH describes how pixels downlinked from the spacecraft are converted by the Kepler Data Processing Pipeline (henceforth just the pipeline) into the data products delivered to the MAST archive. 2. Kepler Archive Manual (KDMC-10008) or KAM (Thompson et al., 2016). The KAM describes the format and content of the data products, and how to search for them.3. Kepler Data Characteristics Handbook (KSCI-19040) or KDCH (Christiansen et al., 2016). The KDCH describes recurring non-astrophysical features of the Kepler data due to instrument signatures, spacecraft events, or solar activity, and explains how these characteristics are handled by the pipeline.4. Kepler Data Release Notes 25 (KSCI-19065) or DRN 25 (Thompson et al., 2015). DRN 25 describes signatures and events peculiar to individual quarters, and the pipeline software changes between a data release and the one preceding it.Together, these documents supply the information necessary for obtaining and understanding Kepler results, given the real properties of the hardware and the data analysis methods used, and for an independent evaluation of the methods used if so desired.

Instrument↗

NASA NTRS2019

EARTHDATA PUB: A Data Publication Workflow Solution for NASA’s EOSDIS

Each NASA Distributed Active Archive Center (DAAC) faces the challenge of dealing with an increasingly diverse number of publishable data products from diverse data producers. Data producers, on the other hand, may experience pain points when interacting with the EOSDIS for the first time or when publishing different data at different DAACs. As a result, there has been a growing need to develop a common software framework that serves as a common interface for data producers, rigorously defines the data publication procedure for DAAC staff, facilitates the management of various data publication processes, and tracks the progress of data publication. This software should also account for the different configurations at different DAACs. Currently, two primary data publication workflow and tracking tools exist in operation at the EOSDIS: Semi-Automated ingest System (SAuS) and Data Publication workflow Portal (DAPPeR). However, neither tool is cloud-ready. Automated data processing could be managed by Cumulus, an EOSDIS cloud-based data ingest, archive and management system. However, Cumulus does not support manual tasks or on-premise implementations. We propose to develop the Earthdata Publication Minimum Viable Product (Earthdata Pub MVP) -- a cloud-hosted solution that works with both cloud and on-premise systems and implements the communications and exchange requirements generated by the Earthdata Pub information architecture team.

Rice, Justin L.↗

DOE OSTI2025

Automating Anomaly Detection for Target systems at Spallation Neutron Source

The Spallation Neutron Source (SNS) at Oak Ridge National Laboratory, produces the world’s most intense pulse neutrons beams. An accelerated proton beam is directed into a mercury target to generate neutrons via spallation. The target system accounted for over 40% of the overall downtime of the facility in 2022. Thus, early detection in anomalies in the target systems can enable taking corrective actions to avoid failures and reduce downtime. Fault prognostics and anomaly detection in accelerators, both at SNS and outside, has largely focused on the beam side. This paper presents one the first studies exploring leveraging machine learning to automate the detection of anomalies in the target system. The target system consists of over 30 different interconnected subsystems, and the present work focuses on the mercury process system as a use case. Analyzing data from 28 process variables from 2022 and 2023, tree-based and reconstruction-based algorithms are employed to detect anomalies in archived data. The algorithms detected previously unreported anomalies, several of which were deemed alert worthy by human experts, particularly those found by reconstruction-based algorithms. Using data from each production run in the accelerator increased the generalizability of the models in time. Efforts are now underway to implement a workflow for incorporating human feedback to update the models and evaluating performance on unseen data. The models will eventually be integrated into the existing System Tracking and Reliability system with a web interface for automated anomaly detection and reporting along with a pathway for incorporating human feedback for model updates.

Raj, Anant [ORNL] (ORCID:0000000306711244)↗

DOE OSTI2025

A Data Science and Machine Learning Platform Supporting Large Particle Accelerator Control and Diagnostics Applications Final Report: SBIR Initial Phase II DE-SC0022583

The Machine Learning Data Platform (MLDP) is a product providing full-stack support for data science, Machine Learning, and Artificial Intelligence (ML/AI) applications at particle accelerator and large experimental physics facilities. It supports ML/AI applications from front-end, high-speed acquisition of heterogeneous, time-series data, through data archiving and management, to back-end analysis. The MLDP embodies a “data-science ready” platform for data analysis and ML/AI applications in diagnosis, modelling, control, and optimization of these facilities. It provides data scientists and applications a consistent, datacentric interface to archive data standardizing implementation and deployment of ML/AI algorithms to different operations configurations within the same facility, or between facilities. Being an open-source, public-domain project, the MLDP is intended for broadest possible impact by increasing accessibility and minimizing the required expertise for installation and operation. The MLDP can also be deployed at user facilities for experimental data collection, archiving, and analysis. It is capable of acquisition and archiving of heterogeneous data from experimental equipment (e.g., images, arrays, structures, etc.) along with system hardware configurations (e.g., scalars, tables), control system process variables, and any metadata required for provenance. Thus, the MLDP can manage experimental data through its entire lifecycle, from acquisition and archiving, through analysis and investigation, to release and final publication.

43 PARTICLE ACCELERATORS↗

NASA NTRS1992

Update - The Earth Observing System (EOS) forward and return link data processing and communications services

An overview is presented of the EOS ground support services in order to identify interfaces to and drivers of the data processing and communication systems. Generic system requirements are compared with those specifically needed for EOS, including processing requirements for forward link and return link data. Communications requirements for transporting the forward link data from the EOS Operations Center and for the transfer of level zero data to the EOS Data and Information System (EOSDIS) Distributed Active Archive Centers are specified. The forward and return link processing requirements of the EOS instruments from the international partners are also addressed. The overall context of EOSDIS in the Mission to Planet Earth Program is addressed.

Ramapriyan, H. K.↗

NASA NTRS1990

High Resolution Imaging Spectrometer (HIRIS) - A major advance in imaging spectrometry

HIRIS, a facility instrument on the Earth Observing System Platform, will enable scientists to both identify and interpret changes, in the global environment, at a level of detail unavailable with previous instrumentation. This instrument will make localized measurements of biological, ecological, climatological, hydrological, and geological properties, and, with its finer wavelength sampling and full spectral coverage over the wavelength region of interest, will allow these properties to be identified and studied quantitatively from space. The data obtained will be downlinked from space, processed, catalogued, stored in active archives for easy access, and distributed to users.

Rockey, Donald E.↗