Radar / Processed Data
This dataset contains processed raw data from the UND 94 GHz W-band Cloud Doppler Radar.
17 WIND ENERGY↗
SEARCH · Search NASA
Results for “process data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Big Data Processing for Power Grid Event Detection
In this paper we present the application of big data processing for the development of machine learning(ML) models to detect relevant events in power gridoperations. This is based on almost 20TB of phasormeasurement unit data corresponding to up to two years of operation of three grid interconnections which provide power to most of the United States. A significant aspect of the work consists in having all data processing performed on a single standard GPU server, from pre-processing to ML model training and testing. We describe the data and computational infrastructure, challenges faced and methods used in dataprocessing, main findings and results. The ML approach employed for best utilization of the big data is also discussed, including sample results.
Paes Leao, Bruno↗
DEIMoS GUI: An Open-Source User Interface for a High-Dimensional Mass Spectrometry Data Processing Tool
In this paper, we report the creation of a graphical user interface (GUI) for the Data Extraction for Integrated Multidimensional Spectrometry (DEIMoS) tool. DEIMoS is a Python package to process data from high-dimensional mass spectrometry measurements. It is divided into several modules, each representing a data processing step, such as peak detection, alignment, and tandem mass spectra extraction and deconvolution. The inputs for and outputs from DEIMoS can include millions of N-dimensional data points, which can be challenging to visualize in a way that is interactive, informative, and responsive. Here, we used the HoloViz Python data stack, including DataShader and Param, to create an interactive visualization of mass spectrometry data. We believe the GUI will increase the accessibility of DEIMoS, and the visualization methods could be useful for other open-source mass spectrometry tools.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Intelligent experiments through real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and future EIC detectors
This R&D project, initiated by the DOE Nuclear Physics AI-Machine Learning initiative in 2022, leverages AI to address data processing challenges in high-energy nuclear experiments (RHIC, LHC, and future EIC). Our focus is on developing a demonstrator for real-time processing of high-rate data streams from sPHENIX experiment tracking detectors. The limitations of a 15 kHz maximum trigger rate imposed by the calorimeters can be negated by intelligent use of streaming technology in the tracking system. The approach efficiently identifies low momentum rare heavy flavor events in high-rate p+p collisions (3MHz), using Graph Neural Network (GNN) and High Level Synthesis for Machine Learning (hls4ml). Success at sPHENIX promises immediate benefits, minimizing resources and accelerating the heavy-flavor measurements. The approach is transferable to other fields. For the EIC, we develop a DIS-electron tagger using Artificial Intelligence - Machine Learning (AI-ML) algorithms for real-time identification, showcasing the transformative potential of AI and FPGA technologies in high-energy nuclear and particle experiments real-time data processing pipelines.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NLR Data Processing Pipeline for MADIS [SWR-26-050]
The NLR Data Processing Pipeline for MADIS software package is for downloading, processing, and performing QA/QC on MADIS data. Designed to handle the following steps: 1) Download all MADIS data as compressed netcdf files for a given time period. 2) Unpack netcdf files into timeseries csvs for each coordinate within the given bounding box. 3) Process the csvs to filter according to quality control checks and convert variables to correct units. 4) Write processed csvs to a single nc file.
Benton, Brandon [National Laboratory of the Rockie↗
Site C1 UTD LIDAR Windcube 200S / Processed Data
Site C1a UTD LIDAR Windcube 200S / Processed Data
17 WIND ENERGY↗
Tidal Energy Resource Characterization, Velocity and Turbulence Measurements, Processed Data, Cook Inlet, AK, 2021
This submission contains processed datasets from a long-term deployment of 3 moorings and a transect survey of the proposed tidal energy site off the East Forelands in Cook Inlet, AK. The long-term mooring datasets were created from 8 instruments mounted on a Terrasond High Energy Oceanographic Mooring (THEOM) bottom lander and two Mid-Water Mooring (MWM) Stablemoor buoys from 1 July 2021 to 31 August 2021 (60 days). The west-most mooring (MWM1) was deployed at 60.720225 N, 151.436196 W in ~50 m of water. The middle mooring (THEOM) was deployed at 60.720703 N, 151.429500 W in ~52 m of water. The east-most buoy (MWM2) was deployed at 60.720081 N, 151.420896 W in ~50 m of water. Each Stablemoor carried three instruments: 1. A Nortek Vector acoustic Doppler velocimeter (ADV) mounted at the Stablemoor's nose. Data were recorded at 8 Hz on a 5 minute duty cycle every 20 minutes. Data was motion-corrected using the internal IMU and external ADCP bottom-track data and then bin-averaged into 4 minute bins and converted to the Principal (streamwise, cross-stream, vertical) coordinate system. (Note: 30 seconds were trimmed from the beginning and end of each 5 minute duty cycle to account for the filter end-effects from turning on and turning off the IMU.) 2. A down-looking Nortek Signature 1000 kHz acoustic Doppler current profiler (ADCP) mounted in the first Stablemoor instrument well. Data were recorded in 2 Hz with 5-beam burst and bottom-track enabled. Processed data has been averaged into 10 minute bins and converted into the Principal coordinate system. 3. An up-looking Nortek Signature 1000 kHz acoustic Doppler current profiler (ADCP) mounted in the second Stablemoor instrument well. Data were recorded at 4 Hz with 5 beam burst enabled. Processed data has been averaged into 10 minute bins and converted into the Principal coordinate system. Note: the down-facing ADCP on MWM1 failed on July 10th, 2021, only recording 9 days of data. Because ADV motion-correction required bottom track, the ADV from MWM1 also only has 9 days processed. Additionally, only 25 days of data were processed from the MWM2 ADV because it appeared to have been impacted by debris on 7/25. Two instruments were mounted on the THEOM (see MHKDR link further below for THEOM raw data): 4. A Nortek Vector acoustic Doppler velocimeter (ADV). Data were recorded at 8 Hz on a 5 minute duty cycle every 20 minutes. Data was bin-averaged into 5 minute bins, and converted to the Principal coordinate system. 5. A Nortek Signature 500 kHz acoustic Doppler current profiler (ADCP). Data were recorded in 4 Hz in the beam coordinate system from all 5 beams. Processed data has been averaged into 10 minutes bins and converted to the Principal coordinate system.
16 TIDAL AND WAVE POWER↗
Lidar Wind Cube / Processed Data
This dataset contains processed and standardized data from DOE Buoy 140 deployed during WFIP3.
17 WIND ENERGY↗
Planck 2018 Results - II. Low Frequency Instrument data processing
We present a final description of the data-processing pipeline for the Planck Low Frequency Instrument (LFI), implemented for the 2018 data release. Several improvements have been made with respect to the previous release, especially in the calibration process and in the correction of instrumental features such as the effects of nonlinearity in the response of the analogue-to-digital converters. We provide a brief pedagogical introduction to the complete pipeline, as well as a detailed description of the important changes implemented. Self-consistency of the pipeline is demonstrated using dedicated simulations and null tests. We present the final version of the LFI full sky maps at 30, 44, and 70 GHz, both in temperature and polarization, together with a refined estimate of the solar dipole and a final assessment of the main LFI instrumental parameters.
79 ASTRONOMY AND ASTROPHYSICS↗
PvaPy streaming framework for real-time data processing
User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.
EPICS pvAccess↗
Demonstrating Cross-Facility Data Processing at Scale with Laue Microdiffraction
In February and April 2023 live, at-scale data processing demonstrations were conducted between the Advanced Photon Source (APS), a synchrotron light source, and the Argonne Leadership Computing Facility (ALCF). These tests were run as part of a novel beamline technique: coded aperture laue micro-diffraction. This technique requires a significant amount of compute to decode appeture patterns embedded in the detector stream. An autonomous system was able to send data to ALCF during an experiment, utilize 50 nodes of the Polaris supercomputer to process 6-12 hour scans, and return the data back to the APS within 12-15 minutes behind the detector. With scan points arriving every 72 seconds, the system kept up with the beamline, potentially enabling in-experiment analysis. The data processing system utilizes Globus infrastructure and an on-demand queue to dynamically acquire nodes on Polaris. The underlying reconstruction algorithms were parallelized via MPI and accelerated with custom CUDA kernels.
Prince, Michael↗
Phlex: Parallel, Hierarchical, and Layered EXecution of data-processing algorithms
Phlex is a computing framework supporting the parallel, hierarchical, and layered execution of data-processing algorithms. It is based on the functional-programming paradigm, thus guaranteeing thread-safety when invoking user-defined pure functions. Phlex allows users to specify arbitrary graph-based hierarchies of data organization, enabling more flexible processing of data as required by the constraints of the program.
Knoepfel, KyleJ. [Fermi National Accelerator Labor↗
Lidar / Processed Data
Wind energy research Note that the processed lidar data has the same metadata as the processed buoy data. These were once considered one dataset, but now the lidar data has been separated to form its own dataset.
17 WIND ENERGY↗
Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers
The prompt reconstruction of the data recorded from the Large Hadron Collider (LHC) detectors has always been addressed by dedicated resources at the CERN Tier-0. Such workloads come in spikes due to the nature of the operation of the accelerator and in special high load occasions experiments have commissioned methods to distribute (spill-over) a fraction of the load to sites outside CERN. The present work demonstrates a new way of supporting the Tier-0 environment by provisioning resources elastically for such spilled-over workflows onto the Piz Daint Supercomputer at CSCS. Furthermore, this is implemented using containers, tuning the existing batch scheduler and reinforcing the scratch file system, while still using standard Grid middleware. ATLAS, CMS and CSCS have jointly run selected prompt data reconstruction on up to several thousand cores on Piz Daint into a shared environment, thereby probing the viability of the CSCS high performance computer site as on demand extension of the CERN Tier-0, which could play a role in addressing the future LHC computing challenges for the high luminosity LHC.
97 MATHEMATICS AND COMPUTING↗
Processed Data Reformatted
These are the processed reformatted data from HRRR converted into power
17 WIND ENERGY↗
Data Processing Package for Cyclic Integrated Reversible Bending Fatigue Testing
Data Processing Package for Cyclic Integrated Reversible Bending Fatigue Testing
Wang, Hong↗
Heterogeneous data-processing optimization with CLARA’s adaptive workflow orchestrator
The hardware landscape used in HEP and NP is changing from homogeneous multi-core systems towards heterogeneous systems with many different computing units, each with their own characteristics. To achieve maximum performance with data processing, the main challenge is to place the right computing on the right hardware. In this paper, we discuss CLAS12 charge particle tracking workflow orchestration that allows us to utilize both CPU and GPU to improve the performance. The tracking application algorithm was decomposed into micro-services that are deployed on CPU and GPU processing units, where the best features of both are intelligently combined to achieve maximum performance. In this heterogeneous environment, CLARA aims to match the requirements of each micro-service to the strength of a CPU or a GPU architecture. A predefined execution of a micro-service on a CPU or a GPU may not be the most optimal solution due to the streaming data-quantum size and the data-quantum transfer latency between CPU and GPU. So, the CLARA workflow orchestrator is designed to dynamically assign micro-service execution to a CPU or a GPU, based on the online benchmark results analyzed for a period of real-time data-processing.
Gyurjyan, Vardan↗
Site B - NREL Thermodynamic profiler (Assist II-11) / Processed Data
AWAKEN site B - NREL thermodynamic profiler (Assist II-11) Processed Data
17 WIND ENERGY↗