ASSIST 10 (NOAA postcampaign) / Processed Data
This dataset contains AWAKEN - NREL thermodynamic profiler (Assist II-10) processed data collected at NOAA in December 2023 - January 2024.
17 WIND ENERGY↗
SEARCH · Search NASA
Results for “Data Processing”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Tower - TTU (SWiFT) Tower, All levels - Processed Data
This dataset includes processed sonic anemometer measurements at the 200-m SWIFT tower near Lubbock, Texas on July 4, 2012. Observations are of a canonical convective atmospheric boundary layer. Subgrid stresses and fluxes are computed by subtracting 15-min running average from the signals measured at 1 Hz.
17 WIND ENERGY↗
Gas-mapping 3D imager measurement techniques and method of data processing
Measurement approaches and data analysis methods are disclosed for combining 3D topographic data with spatially-registered gas concentration data to increase the efficiency of gas monitoring and leak detection tasks. Here, the metric for efficiency is defined as reducing the measurement time required to achieve the detection, or non-detection, of a gas leak with a desired confidence level. Methods are presented for localizing and quantifying detected gas leaks. Particular attention is paid to the combination of 3D spatial data with path-integrated gas concentration measurements acquired using remote gas sensing technologies, as this data can be used to determine the path-averaged gas concentration between the sensor and points in the measurement scene. Path-averaged gas concentration data is useful for finding and quantifying localized regions of elevated (or anomalous) gas concentration making it ideal for a variety of applications including: oil and gas pipeline monitoring, facility leak and emissions monitoring, and environmental monitoring.
Thorpe, Michael↗
A Data Processing Workflow for Fixed-wing Drone Based Radiation Mapping in the Chornobyl Exclusion Zone (CEZ) - 20119
April 2020 marks the 34. anniversary of the high-profile radiological release from the Chornobyl Nuclear Power Plant (ChNPP). The release of radioactive material from reactor number four began on the 26 April 1986 and continued over a period of about 10 days, releasing approximately 1700 PBq of radioactive material (including 85 PBq of 137-Cs) into the environment. To this day, the accident remains the most significant release of radioactive material since civil nuclear power generation began. In the years since the accident, automated and remote radiation monitoring technologies have advanced significantly in their capabilities. One such example of this is the use of unmanned aerial vehicles (UAVs) in radiation mapping investigations. In April 2019, a team of scientists from the University of Bristol showcased a novel radiation mapping system within the CEZ, specifically aiming to map radiation over a large portion of the area immediately surrounding the ChNPP. Over six days of data collection, the system flew a total distance of 583.8 km, covering an area of 14.6 sq.km with an exceptional spatial resolution (sub 20 m/pixel). The work presented herein outlines and explains the data processing procedure to convert the raw data into 137-Cs activity (kBq/sq.m) and cesium-equivalent dose-rate (CED) at 1 m above ground level (μSv/hr). A demonstration of the validity of the method is demonstrated through the successful reduction of the raw data into a single linear relationship between the measured {sup 137}Cs net peak intensity and the {sup 137}Cs activity. (authors)
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
Summary of the 5th IAEA technical meeting on fusion data processing, validation and analysis (FDPVA)
The purpose of the 5th International Atomic Energy Agency technical meeting on fusion data processing, validation and analysis (FDPVA) (Ghent University, Ghent, Belgium, 12–15 June 2023) was to provide a platform during which a set of topics relevant to FDPVA were discussed with the view of meeting the needs of next step fusion devices such as ITER. The validation and analysis of experimental data obtained from diagnostics used to characterize fusion plasmas are crucial for a knowledge-based understanding of the physical processes governing the dynamics of these plasmas. This paper presents the recent progress and achievements in the domain of plasma diagnostics data analysis and synthetic diagnostics reported at the meeting, including concept description of new devices; fusion databases; integrated data analysis; inverse problems; uncertainty propagation, verification and validation; probabilistic methods and machine learning. The relevant results underline trends observed in the current major fusion confinement devices.
fusion databases↗
X-band ARM Scanning Precipitation 2 RHI radar data processed with Corrected Moments in Antenna Coordinates 2.0
This is the X-SAPR RHI scan data processed using Corrected Moments in Antenna Coordinates 2.0.
54 ENVIRONMENTAL SCIENCES↗
Understanding Technical and Psychosocial Barriers to Realizing FAIR Data Process
The present study investigates barriers and facilitators to the implementation of Findable, Accessible, Interoperable, and Reusable (FAIR) data processes within the Physical Sciences Division of the Computational Sciences Directorate (PCSD). Employing a dual-method approach consisting of surveys and focus group discussions, the study aims to illuminate the complex interplay between technical and psychosocial factors that influence FAIR data adoption.
42 ENGINEERING↗
Leveraging Cluster Computing for Large-Scale Data Processing and Automated Analysis for Radiation Portal Monitor Data
Explore the source record for details and available documents.
Smith, Jacob↗
Validation of Modern Nuclear Data Processing in SCALE
The nuclear data (ND) community is continuously developing more accurate, diversified, and comprehensive data for radiation transport modeling to support the nuclear science community. As these community efforts progress, it is crucial that ND processing tools like AMPX (used for SCALE [1] ND) also be developed in parallel to incorporate these new data into transport codes and actually deliver those data to end users. AMPX is a mature, well-tested code that was developed by many people at Oak Ridge National Laboratory (ORNL) over the course of the past few decades. A large portion of the AMPX codebase, however, was outdated, difficult to maintain, and incompatible with modern code development tools. Some of the most important parts of the AMPX code have now been replaced with modern C++ code that can be maintained more cost-effectively and can be tested more rigorously.
AMPX↗
MANIPULATE-2020: An Interface Tool for Nuclear Data Processing
This report provides basic background data on the Manipulate-2020 code. This code is used for processing and "manipulation" of nuclear data in support of radiation metrology applications. The code is made available on the open GitHub repository and is available to the general nuclear data community.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
Fast Data Processing for Hyperspectral Sensors on Small Platforms
Hyperspectral imaging is a very promising technology for nuclear proliferation detection. However, due to size and weight restrictions, small hyperspectral platforms such as satellites and small drones lack the on-board computing resources for accurate, real-time analysis of the enormous flow of data that a continuously operating hyperspectral sensor generates. This severely limits satellite systems, which can collect far more data than what they can telemeter, and hinders the ability of all platforms to adapt their missions on the fly in response to observations. This program addresses the hyperspectral data processing challenge through development of new, fast and accurate algorithms that produce data products in real time. The algorithms circumvent the major computational bottlenecks in existing processing streams, and would be incorporated in lightweight, power-efficient single-board computer systems. The toolkit of fast algorithms will be immediately useful in current and future hyperspectral systems being built by the Government and by private industry, including drone-based systems and satellite constellations that acquire timely global imagery.
45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
A Data Processing Pipeline To Extract A Knowledge Graph From Heterogeneous Data For Socio-technical Analysis Of Critical Infrastructure Influence
The code is written in Python and consists of the following pipeline that is implemented in Apache Airflow. This pipeline intends to understand the companies that are directly or indirectly involved with a type of critical infrastructure system at some point in that system's lifecycle. The pipeline takes a configuration file that specifies a list of initial companies to consider, a geographic region of interest, and a set of SEC form types as well as other data sources (e.g. CrunchBase) from which to extract entities and relations. There are four main components to this pipeline as currently implemented: Entity Extraction, Network Construction, Analysis, and Visualization. First, Entity Extraction, is implemented as the `topear-extract_organizations` Apache Airflow workflow. Given an initial query that specifies a geographic region of interest and a time interval, the software will extract CI facilities of interest and organizations that have a direct influence relationship to those facilities (e.g. ownership). During the course of the LDRD, we focused on Electric Vehicle charging stations and this information is available via the Department of Energy (DOE) database on fueling stations maintained by NREL. Within the context of the DOE CESER project, we have focused on Battery Energy Storage Systems (BESS). Second, the Network Extraction component will iteratively construct a social network graph given the set of organizations and people extracted in the previous step. Organizations (and eventually People if desired) are then fed as a query to the `topgear-construct_social_network` Apache Airflow workflow which given a set of initial companies and data sets (e.g. SEC EDGAR form types, OpenCorporates, Crunchbase). This Airflow workflow will iteratively query such data sources to discover relationships with new organizations and people. For example, this module can iteratively query SEC EDGAR for metadata that documents the number of each type of form for the given set of companies and their location. This forms metadata represents a catalog of data sources from SEC EDGAR for the extracted social network knowledge graph. The pipeline then downloads these forms from the website and saves them in a build directory for further processing. These documents are then parsed for entities and relations. Again, we note that in additional to SEC data sources, this step can also pull in information on organizations via API services such as CrunchBase and OpenCorporates or bulk data sources. At the end of this step, the resultant social network, the Critical Infrastructure network, and the edges that encode relationships between organizations and CI facilities, form the Adversarial Socio-Technical Network (ASTN) that informs the analysis. Third, the Analysis component processes these generated ASTN. Previously, that has included the ability to compare prevalence of different vendors for a given infrastructure component type across different regions as well as identify common public and private investors across those vendors. This was demonstrated for EV Charging Stations across several different metropolitan areas within an IEEE PES GridEdge publication. More recently, we have looked at ways to identify infrastructure owners and operators of BESS with the most nameplate capacity across different states as well as other indictors of risk resulting from changes in ownership over time. Finally, the Visualization component consists of an HTML/CSS/JS framework by which users can interact geospatial, operational, and organizational relationships across a given portfolio of Critical Infrastructure facilities. The objective is to provide a library of UI/UX modules that can be repurposed for stakeholder-specific dashboards. All of the modules are related via a common event model that enables UI actions in one view to percolate across the other views.
Weaver, Gabriel [Idaho National Laboratory (INL), ↗
Integrated package of electron cyclotron emission imaging data processing and forward modeling in OMFIT
An Electron Cyclotron Emission Imaging (ECEI) data analysis module has been developed for the OMFIT platform to accommodate the needs of users at the DIII-D tokamak for physics applications. The user can easily access the ECEI spatial observation windows in the plasma that are calculated based on the automatically retrieved hardware setup and available DIII-D equilibria, perform spectral analysis, and make 2D electron temperature fluctuation images. The module provides a powerful data post processing package for extracting important physics parameters from the 2D measurements, including the radial structure and poloidal mode number of Alfven Eigenmodes, as well as the frequency-vs-wavenumber dispersion relationship of broadband MHD. Lastly, the module propagates characterized synthetic fluctuations for the user, so one can perform forward modeling tasks with simple analytical fluctuations.
70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
Intelligent Experiments Through Real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and Future EIC Detectors (Final Report)
The overall vision of this project was to integrate real-time artificial intelligence (AI) directly into the data acquisition and detector-control systems of nuclear physics experiments, including both fast online event selection and an autonomous detector-control feedback loop. The work carried out under the award focused on the fast online event-selection half of that vision: the efficient recording of low-momentum heavy-flavor (HF) hadron decays in proton-proton collisions at the sPHENIX experiment at the Relativistic Heavy Ion Collider (RHIC)—an observable that requires fast tracking and topological trigger selection not previously demonstrated at RHIC, and that is essential for QCD studies at future facilities such as the Electron-Ion Collider (EIC). The autonomous detector-control (GPU-based feedback) component named in the project title remained a design concept and was not implemented under this award. The Massachusetts Institute of Technology (MIT) group led the offline simulation and data processing needed to train the machine-learning (ML) models, the translation of trained models to Field-Programmable Gate Array (FPGA) firmware using the hls4ml framework, and the physics validation of heavy-flavor reconstruction. Over the award period, the team developed and hardware-tested the principal components of an AI-based heavy-flavor trigger on simulated and recorded sPHENIX tracker data: a software Bipartite Graph Attention Network (BiGAT) trigger model reaching > 95% signal efficiency at 99% background rejection; an FPGA-native hit clusterizer matching the offline clustering; smaller networks synthesized to FPGA within the required sub-10 µs latency; and an assembled decoder–clusterizer–inference firmware chain exercised on the FELIX readout board. A complete, fully integrated hardware demonstrator was not finished within the award period. This report documents the project goals, the MIT group’s contributions, the technical accomplishments, and the outlook toward applications at the future EIC ePIC detector.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DEEP CELLULAR RECURRENT NEURAL ARCHITECTURE FOR EFFICIENT MULTIDIMENSIONAL TIME-SERIES DATA PROCESSING
Efficient processing of time series data is a fundamental yet challenging problem in pattern recognition. Though recent developments in machine learning and deep learning have enabled remarkable improvements in processing large scale datasets in many application domains, most are designed and regulated to handle inputs that are static in time. Many real-world data, such as in biomedical, surveillance and security, financial, manufacturing and engineering applications, are rarely static in time, and demand models able to recognize patterns in both space and time. Current machine learning (ML) and deep learning (DL) models adapted for time series processing tend to grow in complexity and size to accommodate the additional dimensionality of time. Specifically, the biologically inspired learning based models known as artificial neural networks that have shown extraordinary success in pattern recognition, tend to grow prohibitively large and cumbersome in the presence of large scale multi-dimensional time series biomedical data such as EEG. Consequently, this work aims to develop representative ML and DL models for robust and efficient large scale time series processing. First, we design a novel ML pipeline with efficient feature engineering to process a large scale multi-channel scalp EEG dataset for automated detection of epileptic seizures. With the use of a sophisticated yet computationally efficient time-frequency analysis technique known as harmonic wavelet packet transform and an efficient self-similarity computation based on fractal dimension, we achieve state-of-the-art performance for automated seizure detection in EEG data. Subsequently, we investigate the development of a novel efficient deep recurrent learning model for large scale time series processing. For this, we first study the functionality and training of a biologically inspired neural network architecture known as cellular simultaneous recurrent neural network (CSRN). We obtain a generalization of this network for multiple topological image processing tasks and investigate the learning efficacy of the complex cellular architecture using several state-of-the?art training methods. Finally, we develop a novel deep cellular recurrent neural network (CDRNN) architecture based on the biologically inspired distributed processing used in CSRN for processing time series data. The proposed DCRNN leverages the cellular recurrent architecture to promote extensive weight sharing and efficient, individualized, synchronous processing of multi-source time series data. Experiments on a large scale multi-channel scalp EEG, and a machine fault detection dataset show that the proposed DCRNN offers state-of-the-art recognition performance while using substantially fewer trainable recurrent units.
Vidyaratne, Lasitha S.↗
Trigger-based Incremental Data Processing with Unified Sync and Async Model
In recent years, more and more applications in the cloud have needs to process large-scale on-line datasets, which evolve over time as new entries are added and existing entries are modified. Several programming frameworks, such as Percolator and Oolong, are proposed for such incremental data processing and can achieve efficient processing with an event-driven abstraction. However, these frameworks are inherently asynchronous, leaving the heavy burden of managing synchronization to applications' developers, which further significantly restricts their usabilities. In this study, we propose a trigger-based incremental computing framework in the cloud, called Domino, with both synchronous and asynchronous mechanisms to coordinate parallel triggers. With this new framework, both synchronous and asynchronous applications can be seamlessly developed. Use cases and extensive evaluation results confirm that it can deliver sufficient performance, and also is easy to use for incremental applications in large-scale distributed computing.
97 MATHEMATICS AND COMPUTING↗
Preparation of the Multi-Site Data Processing at the Vera C. Rubin Observatory
The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) Camera is scheduled to start taking data in the summer of 2025. The Data Release Production will run the LSST Science Pipe software at data facilities in the US, France and the UK. The LSST Science Pipeline consists of complex directed acyclic graphs (DAGs) of tasks. Rubin will use the Production and Distributed Analysis (PanDA) workflow and workload management system to orchestrate this complex workflow and the distribution of workloads to the data facilities. When run end-to-end by a team of data production staff, this processing (the Science Pipelines, distributed by the workflow and workload management system) is referred to as a 'campaign'. This paper describes the central services and data facility specific services that support this multi-site data process model, including the service deployment infrastructure, the workload and workflow system, the Campaign Management tools, and connection to Rubin Data Management. This paper will also mention the experience of processing the Rubin Commissioning Camera data. All these are part of the effort to scale up the processing capabilities for the expected very large data volume from the LSST Camera.
Yang, Wei [SLAC]↗
First M87 Event Horizon Telescope Results. III. Data Processing and Calibration
We present the calibration and reduction of Event Horizon Telescope (EHT) 1.3 mm radio wavelength observations of the supermassive black hole candidate at the center of the radio galaxy M87 and the quasar 3C 279, taken during the 2017 April 5–11 observing campaign. These global very long baseline interferometric observations include for the first time the highly sensitive Atacama Large Millimeter/submillimeter Array (ALMA); reaching an angular resolution of 25 μas, with characteristic sensitivity limits of ~1 mJy on baselines to ALMA and ~10 mJy on other baselines. The observations present challenges for existing data processing tools, arising from the rapid atmospheric phase fluctuations, wide recording bandwidth, and highly heterogeneous array. In response, we developed three independent pipelines for phase calibration and fringe detection, each tailored to the specific needs of the EHT. The final data products include calibrated total intensity amplitude and phase information. They are validated through a series of quality assurance tests that show consistency across pipelines and set limits on baseline systematic errors of 2% in amplitude and 1° in phase. The M87 data reveal the presence of two nulls in correlated flux density at ~3.4 and ~8.3 Gλ and temporal evolution in closure quantities, indicating intrinsic variability of compact structure on a timescale of days, or several light-crossing times for a few billion solar-mass black hole. These measurements provide the first opportunity to image horizon-scale structure in M87.
79 ASTRONOMY AND ASTROPHYSICS↗