Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

Scalable, Multithreaded, Partially-in-Place Sorting

A recent trend in hardware development is producing computing systems that are stretching the number of cores and size of shared-memory beyond where most fundamental serial algorithms perform well. The expectation is that this trend will continue. So it makes sense to rethink our fundamental algorithms such as sorting. There are many situations where data that needs to be sorted will actually fit into the shared memory so applications could benefit from an efficient parallel sorting algorithm. When sorting large data (at least hundreds of Gigabytes) in a single shared memory, there are two factors that affect the algorithm choice. First, does the algorithm sort in-place? And second, does the algorithm scale well beyond tens of threads? Surprisingly, existing algorithms posses either one of these factors, but not both. We present an approach that gracefully degrades in performance as the amount of available working memory decreases relative to the size of the input.

HPC↗

Identifying Suspect Instrument Intervals Using Midnight Noise Time Histories

With an intense work ethic, and high levels of devotion to their craft, seismologists continue to battle the elements and wrestle with complex logistics in their efforts to field instrumentation over the entire globe. We are fortunate to see data volumes accelerating in size, and quality continuing to improve; for example, universal timing issues can now be considered rare. However, instrument response issues are still common, and can hinder research that seeks to understand amplitudes of seismic wavefields. We are interested in using the accumulations of seismic data to develop models that will predict high frequency (0.2-20 Hz, and higher) signal amplitudes for explosion monitoring purposes over broad areas, which requires large data sets and extensive quality control. To identify response and station health issues, we have collected noise time histories for global seismic data, focusing on measurements near (but not restricted to) midnight to eliminate diurnal variations, and have manually determined time intervals that appear inconsistent with background behavior. We assign descriptive labels, but do not attempt to diagnose causes. We use these intervals to discard data. To date, we have examined 39,260 channels from 11,105 stations, heavily weighted toward IRIS holdings, dates through 2017 (depending on station, roughly the date we started our manual review), bands between 1 and 8 Hz, finding 24,733 anomalous time intervals. The great majority (90%) of these intervals appear to be shifts of constant offset, often bounded by times of known instrument changes, likely the result of poor documentation of response parameters at one of many stages between the field and the plotter. We hope these results can be of use to our colleagues, and would encourage community efforts to diagnose anomalous behavior, and fix poor responses. We also hope these results will support automation efforts, including application of supervised learning techniques.

47 OTHER INSTRUMENTATION↗

Free-space optical communication system and methods for efficient data delivery

Communication systems and methods for high-data-rate, high-efficiency, free-space communications are described. High-speed optical modems and automatic repeat request can be employed to transmit large data files without data errors between remote devices, such as an earth-orbiting satellite and ground station. Data rates over 100 Gb/s can be achieved.

Boroson, Don M.↗

The Spectrum of the Isotropic Diffuse Gamma-Ray Emission Derived From First-Year Fermi Large Area Telescope Data

We report on the first Fermi Large Area Telescope (LAT) measurements of the so-called 'extra-galactic' diffuse {gamma}-ray emission (EGB). This component of the diffuse {gamma}-ray emission is generally considered to have an isotropic or nearly isotropic distribution on the sky with diverse contributions discussed in the literature. The derivation of the EGB is based on detailed modelling of the bright foreground diffuse Galactic {gamma}-ray emission (DGE), the detected LAT sources and the solar {gamma}-ray emission. We find the spectrum of the EGB is consistent with a power law with differential spectral index {gamma} = 2.41 {+-} 0.05 and intensity, I(> 100 MeV) = (1.03 {+-} 0.17) x 10{sup -5} cm{sup -2} s{sup -1} sr{sup -1}, where the error is systematics dominated. Our EGB spectrum is featureless, less intense, and softer than that derived from EGRET data.

79 ASTRONOMY AND ASTROPHYSICS↗

Data Processing for the SNS EQ-SANS Diffractometer

The scattering data on the Extended Q-Range Small Angle Scattering Diffractometer at the Spallation Neutron Source are stored as neutron events. Each event has two parts: the time of fight of the detected neutron, and an identifier for the neutron's location on the detector. In addition, each event is associated with the source pulse from which the neutron is originated. Due to the high neutron beam intensity at the SNS, the size of the stored data files can easily exceed several Gigabytes. Processing and reducing these data can be very time consuming. The standard way of handling the data at the SNS has been to first transform the data into histograms. On the EQ-SANS, these histograms are further processed into intensity I vs. neutron momentum transfer Q files, I(Q). For a large data, the whole process can take many minutes to hours to complete, which severely limits the turnaround time of an experiment. Experimenters often want to reduce the data quickly and depending on the result, make decisions on the setup of next measurements. They may also need to reduce the data using different inputs and conditions. In this work, we present the implementation of a fast data processing scheme for the EQ-SANS diffractometer. The scattering data are processed directly from neutron event files into I(Q) or other desired formats, greatly reducing the needed computation time. Further speed gains are obtained by the implementation of parallel computing under the Message Passing Interface framework. Most data on the EQ-SANS can now be handled in mere seconds. The process is highly automated. All relevant experimental information that is stored in instrument configuration files are read and processed. In addition, the new implementation is highly flexible. All the data corrections and handlings can be controlled transparently through user inputs. Finally, software implementation enables the slicing of neutron event, which can be used for the study of time dependent phenomena.

Data Processing↗

$W / Z +$ jets production at the Tevatron

Both the DO/ and CDF experiments at Fermilab Tevatron collider at √s=1.8 TeV have accumulated over 50 pb−1 of data. Each experiment collected more than 50,000 W→l+ν and 5000 Z→l+l̄ candidates for each lepton species (e and μ). Using this large data sample of W and Z candidates, the two experiments are actively testing perturbative QCD predictions. Among the studies that are in progress, preliminary results of W/Z+jets characteristics, multiplicity distributions of associated jets, and a study on the determination of the strong coupling constant using the ratio of the W+1 jet to W+0 jet cross sections are presented.

ANGULAR DISTRIBUTION↗

BULKI-Store v0.3.2

BULKI-Store is a distributed object storage system optimized for high-performance computing environments. Built with a Rust core and Python bindings, it efficiently manages scientific and machine learning datasets across HPC clusters. The system employs a client-server architecture with MPI integration, enabling seamless scaling on supercomputers like Perlmutter. BULKI-Store's object-oriented approach provides intuitive data organization with rich metadata support, contrasting with traditional file-based solutions. Key optimizations include selective checkpoint loading, unified checkpoint files, and object chunking for large data transfers. For machine learning workloads, BULKI-Store offers advantages through fine-grained access patterns, dynamic data sharing between training instances, and reduced memory pressure. Memory management features include strategic Python GC calls, minimized data copies, and batch processing capabilities. The system leverages Rayon's thread pool for asynchronous data prefetching and supports multiple CPU architectures (ARM64, x86, AMD, RISC-V). By combining performance optimizations with developer-friendly APIs, BULKI-Store addresses the complex data management challenges of modern HPC applications while maintaining compatibility across heterogeneous computing environments.

Zhang, Wei [Lawrence Berkeley National Laboratory ↗

BLDAP Intro to Python/Data Science Curriculum v1

The Github repository contains the Jupyter notebooks for the intro to Python / Data Science course for Berkeley Lab Director's Apprenticeship Program (BLDAP). This course is designed for students with little to no experience in coding to learn skills in Python necessary for data science. Students utilize Jupyter notebooks throughout the course. The overall goal is for students to learn how to use Python to clean, analyze, and visualize large data sets in order to communicate effectively their conclusions about the data set. Students apply the skills they learned on actual data sets provided by researchers in Berkeley Lab.

Hales, Laurel [Lawrence Berkeley National Laborato↗

In-flight positional and energy use data set of a DJI Matrice 100 quadcopter for small package delivery

Abstract We autonomously directed a small quadcopter package delivery Uncrewed Aerial Vehicle (UAV) or “drone” to take off, fly a specified route, and land for a total of 209 flights while varying a set of operational parameters. The vehicle was equipped with onboard sensors, including GPS, IMU, voltage and current sensors, and an ultrasonic anemometer, to collect high-resolution data on the inertial states, wind speed, and power consumption. Operational parameters, such as commanded ground speed, payload, and cruise altitude, were varied for each flight. This large data set has a total flight time of 10 hours and 45 minutes and was collected from April to October of 2019 covering a total distance of approximately 65 kilometers. The data collected were validated by comparing flights with similar operational parameters. We believe these data will be of great interest to the research and industrial communities, who can use the data to improve UAV designs, safety, and energy efficiency, as well as advance the physical understanding of in-flight operations for package delivery drones.

42 ENGINEERING↗

A GROUP FINDING ALGORITHM FOR MULTIDIMENSIONAL DATA SETS

We describe a density-based hierarchical group finding algorithm capable of identifying structures and substructures of any shape and density in multidimensional data sets where each dimension can be a numeric attribute with arbitrary measurement scale. This has applications in a wide variety of fields from finding structures in galaxy redshift surveys, to identifying halos and subhalos in N-body simulations and group finding in Local Group chemodynamical data sets. In general, clustering schemes require an a priori definition of a metric (a non-negative function that gives the distance between two points in a space) and the quality of clustering depends upon this choice. The general practice is to use a constant global metric which is optimal only if the clusters in the data are self-similar. For complex data configurations even the most finely tuned constant global metric turns out to be suboptimal. Moreover, the correct choice of metric also becomes increasingly important as the number of dimensions increase. To address these problems, we present an entropy-based binary space partitioning algorithm which uses a locally adaptive metric for each data point. The metric is employed to calculate the density at each point and a list of its nearest neighbors, and this information is then used to form a hierarchy of groups. Finally, the ratio of maximum to minimum density of points in a group is used to estimate the significance of the groups. Setting a threshold on this significance can effectively screen out groups arising due to Poisson noise and helps organize the groups into meaningful clusters. For a data set of N points, the algorithm requires only O(N) space and O(N(log N){sup 3}) time which makes it ideally suitable for analyzing large data sets. As an example, we apply the algorithm to identify structures in a simulated stellar halo using the full six-dimensional phase space coordinates.

79 ASTRONOMY AND ASTROPHYSICS↗

A convolutional neural network-based screening tool for X-ray serial crystallography

A new tool is introduced for screening macromolecular X-ray crystallography diffraction images produced at an X-ray free-electron laser light source. Based on a data-driven deep learning approach, the proposed tool executes a convolutional neural network to detect Bragg spots. Automatic image processing algorithms described can enable the classification of large data sets, acquired under realistic conditions consisting of noisy data with experimental artifacts. Outcomes are compared for different data regimes, including samples from multiple instruments and differing amounts of training data for neural network optimization.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Present and Future of Central Production With STAR Detector at RHIC

The present status and future of the physics program of Central Production using the STAR detector at RHIC are described. The program focuses on particle production in the Double Pomeron Exchange (DPE) process. Forward protons from the DPE interaction are detected in the Roman Pot system installed at 55.5 m and 58.5 m on both sides of the STAR interaction point. The recoil system of charged particles from the DPE process is measured in the STAR Time Projection Chamber (TPC). The first data were taken during the 2009 RHIC Run 9 using polarized proton-proton collisions at {radical}(s) 200 GeV. The preliminary spectra of two pion and four pion invariant mass reconstructed by STAR TPC in central region of pseudo-rapidity |{eta}|<1, are presented. Plans to take data with the current system at {radical}(s) = 500 GeV and plans to upgrade the forward proton tagging system, so that it can reach higher masses and obtain large data samples to allow searching for glueballs that could be produced in the DPE process, are also discussed.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

EQ_phase_detection

The EQ_phase_detection software is designed to scan continuous daily waveforms to detect earthquake phase arrivals from local to regional (150 km) events. The detections are made with a deep learning encoder-decoder model. When the model detects an earthquake in the waveforms, a second model is implemented to classify the first arriving motions. Both deep learning models are trained with the Tensorflow package using publicly available benchmark data sets. The software input is a path to a directory that contains waveforms in mseed format and the associated response files in xml format. The output is a data table of time stamped detections, signal amplitude, signal-to-noise ratio, and softmax probability of the detection in a generic format applicable to post-processing association algorithms for event locations. Additionally, the p-wave and s-wave waveforms are saved in a data table for rapid access when producing improved locations using correlation-based techniques. The software is designed for multiprocessing with multiple GPU’s for rapid processing of large data sets. The configuration file provides flexibility in the trained models implemented and allows access to multiple models trained for different sampling rates or input dimensions. This is particularly useful for regions with multiple networks that do not have the same data parameters.

Johnson, Christopher↗

Information Analysis Methodology for Border Security Deployment Prioritization and Post Deployment Evaluation

Due to international commerce, cross-border conflicts, and corruption, a holistic, information driven, approach to border security is required to best understand how resources should be applied to affect sustainable improvements in border security. The ability to transport goods and people by land, sea, and air across international borders with relative ease for legitimate commercial purposes creates a challenging environment to detect illicit smuggling activities that destabilize national level border security. Smuggling activities operated for profit or smuggling operations driven by cross border conflicts where militant or terrorist organizations facilitate the transport of materials and or extremists to advance a cause add complexity to smuggling interdiction efforts. Border security efforts are further hampered when corruption thwarts interdiction efforts or reduces the effectiveness of technology deployed to enhance border security. These issues necessitate the implementation of a holistic approach to border security that leverages all available data. Large amounts of information found in hundreds of thousands of documents can be compiled to assess national or regional borders to identify variables that influence border security. Location data associated with border topics of interest may be extracted and plotted to better characterize the current border security environment for a given country or region. This baseline assessment enables further analysis, but also documents the initial state of border security that can be used to evaluate progress after border security improvements are made. Then, border security threats are prioritized via a systems analysis approach. Mitigation factors to address risks can be developed and evaluated against inhibiting factor such as corruption. This holistic approach to border security helps address the dynamic smuggling interdiction environment where illicit activities divert to a new location that provides less resistance to smuggling activities after training or technology is deployed at a given location. This paper will present an approach to holistic border security information analysis.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

A Parallel EM Algorithm for Model-Based Clustering Applied to the Exploration of Large Spatio-Temporal Data

We develop a parallel EM algorithm for multivariate Gaussian mixture models and use it to perform model-based clustering of a large climate data set. Three variants of the EM algorithm are reformulated in parallel and a new variant that is faster is presented. All are implemented using the single program, multiple data (SPMD) programming model, which is able to take advantage of the combined collective memory of large distributed computer architectures to process larger data sets. Displays of the estimated mixture model rather than the data allow us to explore multivariate relationships in a way that scales to arbitrary size data. We study the performance of our methodology on simulated data and apply our methodology to a high resolution climate dataset produced by the community atmosphere model (CAM5). This article has supplementary material online.

Chen, Wei-Chen↗

Development and test of a mini-Data Acquisition system for the High-Luminosity LHC upgrade of the ATLAS Monitored Drift Tube detector

New front-end electronics including ASICs and FPGA boards are under development for the ATLAS Monitored Drift Tube (MDT) detector to handle the large data rates and harsh environment expected at high-luminosity LHC runs. A mobile Data Acquisition (miniDAQ) system is designed to perform integration tests of these front-end electronics. In addition, it will be used for surface commissioning of 96 small-radius MDT (sMDT) chambers and for integration and commissioning of new front-end electronics on the present ATLAS MDT chambers. Details of the miniDAQ hardware and firmware are described in this article. Here, the miniDAQ system is also used to read out new front-end electronics on an sMDT prototype chamber using cosmic muons and results obtained are shown.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Occam's Razor and Petascale Visual Data Analysis

One of the central challenges facing visualization research is how to effectively enable knowledge discovery. An effective approach will likely combine application architectures that are capable of running on today?s largest platforms to address the challenges posed by large data with visual data analysis techniques that help find, represent, and effectively convey scientifically interesting features and phenomena.

97 MATHEMATICS AND COMPUTING↗

Occam's razor and petascale visual data analysis

One of the central challenges facing visualization research is how to effectively enable knowledge discovery. An effective approach will likely combine application architectures that are capable of running on today's largest platforms to address the challenges posed by large data with visual data analysis techniques that help find, represent, and effectively convey scientifically interesting features and phenomena.

99 GENERAL AND MISCELLANEOUS↗