Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12

Modeling and Simulation Tool to Improve Urban Search Planning (NA22 Final Report)

The venture project Tool to Improve Urban Search Planning (LL14-V-Urban Search Planning ToolPD3RS) supported two major types of effort: 1) development of a software prototype search planning tool, with the goal of transitioning the tool to operational users for evaluation and possible adoption; 2) enabling research to better understand the radiation characteristics of an urban environment, with implications for deployment of radiation detection systems in that environment. The first effort has resulted in prototype software, Optimization Planning Tool for Urban Search (OPTUS), that has been distributed to a limited set of potential users, including users within the NNSA emergency response community. The second effort supported various research topics including high fidelity radiation transport modeling in urban environments, measurement campaigns for model validation, development of vehicle-based high purity germanium (HPGe) detector systems and supporting data analysis, and computational capabilities for archiving and managing large data sets. While some of these related efforts have informed the models and software design of the OPTUS tool, in large part they pursued more basic research, some of which has evolved into separate DNN R&D projects. Project performers included Oak Ridge National Laboratory (ORNL), Lawrence Berkeley National Laboratory (LBNL) and the Remote Sensing Laboratory (RSL), in addition to Lawrence Livermore Laboratory (LLNL), the project lead. The project also supported university research, including several internships at the laboratories.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

A Parallel Graph Environment for Real-World Data Analytics Workflows

Economic competitiveness and national security depend increasingly on the insightful analysis of large data sets. The diversity of real-world data sources and analytic workflows impose challenging hardware and software requirements for parallel graph platforms. The irregular nature of graph methods is not supported well by the deep memory hierarchies of conventional distributed systems, requiring new processor and runtime system designs to tolerate memory and synchronization latencies. Moreover, the efficiency of relational table operations and matrix computations are not attainable when data is stored in common graph data structures. In this paper, we present HAGGLE, a high-performance, scalable data analytics platform. The platform’s hybrid data model supports a variety of distributed, thread-safe data structures, parallel programming constructs, and persistent and streaming data. An abstract runtime layer enables us to map the stack to conventional, distributed computer systems with accelerators. The runtime uses multithreading, active messages, and data aggregation to hide memory and synchronization latencies on large-scale systems.

Graph Analytics↗

Searches for New Physics at the Tevatron and LHC

This is an auspicious moment in experimental particle physics - there are large data samples at the Tevatron and a new energy regime being explored at the Large Hadron Collider with ever larger data samples. The coincidence of these two events suggests that we will soon be able to address the question, what lies beyond the standard model? Particle physics's current understanding of the universe is embodied in it. The model has been tested to extreme precision - better than a part in ten thousand - but we suspect that it is only an approximation, and that physics beyond this standard model will appear in the data of the Tevatron and LHC in the near future. This brief review touches on the status of searches for new physics at the time of the conference.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Improving the reliability of automated non-destructive inspection

In automated NDE a region of an inspected component is often interrogated several times, be it within a single data channel, across multiple channels or over the course of repeated inspections. The systematic combination of these diverse readings is recognized to provide a means to improve the reliability of the inspection, for example by enabling noise suppression. Specifically, such data fusion makes it possible to declare regions of the component defect-free to a very high probability whilst readily identifying indications. Registration, aligning input datasets to a common coordinate system, is a critical pre-computation before meaningful data fusion takes place. A novel scheme based on a multiobjective optimization is described. The developed data fusion framework, that is able to identify and rate possible indications in the dataset probabilistically, based on local data statistics, is outlined. The process is demonstrated on large data sets from the industrial ultrasonic testing of aerospace turbine disks, with major improvements in the probability of detection and probability of false call being obtained.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Accessing Data Federations with CVMFS

Data federations have become an increasingly common tool for large collaborations such as CMS and Atlas to efficiently distribute large data files. Unfortunately, these typically are implemented with weak namespace semantics and a non-POSIX API. On the other hand, CVMFS has provided a POSIX-compliant read-only interface for use cases with a small working set size (such as software distribution). The metadata required for the CVMFS POSIX interface is distributed through a caching hierarchy, allowing it to scale to the level of about a hundred thousand hosts. In this paper, we will describe our contributions to CVMFS that merges the data scalability of XRootD-based data federations (such as AAA) with metadata scalability and POSIX interface of CVMFS. We modified CVMFS so it can serve unmodified files without copying them to the repository server. CVMFS 2.2.0 is also able to redirect requests for data files to servers outside of the CVMFS content distribution network. Finally, we added the ability to manage authorization and authentication using security credentials such as X509 proxy certificates. We combined these modifications with the OSGs StashCache regional XRootD caching infrastructure to create a cached data distribution network. Here, we will show performance metrics accessing the data federation through CVMFS compared to direct data federation access. Additionally, we will discuss the improved user experience of providing access to a data federation through a POSIX filesystem.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

VisIO: enabling interactive visualization of ultra-scale, time-series data via high-bandwidth distributed I/O systems

Petascale simulations compute at resolutions ranging into billions of cells and write terabytes of data for visualization and analysis. Interactive visuaUzation of this time series is a desired step before starting a new run. The I/O subsystem and associated network often are a significant impediment to interactive visualization of time-varying data; as they are not configured or provisioned to provide necessary I/O read rates. In this paper, we propose a new I/O library for visualization applications: VisIO. Visualization applications commonly use N-to-N reads within their parallel enabled readers which provides an incentive for a shared-nothing approach to I/O, similar to other data-intensive approaches such as Hadoop. However, unlike other data-intensive applications, visualization requires: (1) interactive performance for large data volumes, (2) compatibility with MPI and POSIX file system semantics for compatibility with existing infrastructure, and (3) use of existing file formats and their stipulated data partitioning rules. VisIO, provides a mechanism for using a non-POSIX distributed file system to provide linear scaling of 110 bandwidth. In addition, we introduce a novel scheduling algorithm that helps to co-locate visualization processes on nodes with the requested data. Testing using VisIO integrated into Para View was conducted using the Hadoop Distributed File System (HDFS) on TACC's Longhorn cluster. A representative dataset, VPIC, across 128 nodes showed a 64.4% read performance improvement compared to the provided Lustre installation. Also tested, was a dataset representing a global ocean salinity simulation that showed a 51.4% improvement in read performance over Lustre when using our VisIO system. VisIO, provides powerful high-performance I/O services to visualization applications, allowing for interactive performance with ultra-scale, time-series data.

97 MATHEMATICS AND COMPUTING↗

ASCR Workshop on In Situ Data Management: Enabling Scientific Discovery from Diverse Data Sources

In January 2019, the U.S. Department of Energy, Office of Science program in Advanced Scientific Computing Research, convened a workshop to identify priority research directions for in situ data management (ISDM). The workshop defined ISDM as the practices, capabilities, and procedures to control the organization of data and enable the coordination and communication among heterogeneous tasks, executing simultaneously in a high-performance computing system, cooperating toward a common objective. The workshop revealed two primary, interdependent motivations for processing and managing data in situ. The first motivation is that the in situ methodology enables scientific discovery from a broad range of data sources over a wide scale of computing platforms: leadership-class systems, clusters, clouds, workstations, and embedded devices at the edge. The successful development of ISDM capabilities will benefit real-time decision-making, design optimization, and data-driven scientific discovery. The second motivation is the need to decrease data volumes. ISDM can make critical contributions to managing large data volumes from computations and experiments to minimize data movement, save storage space, and boost resource efficiency, often while simultaneously increasing scientific precision. A fundamental finding of this workshop is that the methodologies used to manage data among a variety of tasks in situ can be used to facilitate scientific discovery from many different data sources—simulation, experiment, and sensors, for example—and that being able to do so at numerous computing scales will benefit real-time decision-making, design optimization, and data-driven scientific discovery across the Office of Science mission space. Applications wanting to use the in situ capabilities include those where data analysis feeds back to the simulation, decisions are made autonomously, big data or machine learning is among the tasks to be coordinated, and computations need to be completed in real time. The workshop identified six priority research directions that highlight the components and capabilities needed for ISDM to be successful for the wide variety of applications discussed: making ISDM capabilities more pervasive, controllable, composable, and transparent, with a focus on greater coordination with the software stack and a diversity of fundamentally new data algorithms.

97 MATHEMATICS AND COMPUTING↗

ASCR Workshop on In Situ Data Management

In January 2019, the U.S. Department of Energy, Office of Science program in Advanced Scientific Computing Research, convened a workshop to identify priority research directions for in situ data management (ISDM). The workshop defined ISDM as the practices, capabilities, and procedures to control the organization of data and enable the coordination and communication among heterogeneous tasks, executing simultaneously in a high-performance computing system, cooperating toward a common objective. The workshop revealed two primary, interdependent motivations for processing and managing data in situ. The first motivation is that the in situ methodology enables scientific discovery from a broad range of data sources over a wide scale of computing platforms: leadership-class systems, clusters, clouds, workstations, and embedded devices at the edge. The successful development of ISDM capabilities will benefit real-time decision-making, design optimization, and data-driven scientific discovery. The second motivation is the need to decrease data volumes. ISDM can make critical contributions to managing large data volumes from computations and experiments to minimize data movement, save storage space, and boost resource efficiency, often while simultaneously increasing scientific precision.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Hybrid Storage Solution

With the rise of artificial intelligence and machine learning, data sets used to train models have become increasingly large. The availability, accessibility and integrity of large data sets has become important to the research conducted at Los Alamos National Laboratory. Ceph is a storage solution suitable for use with critical data because of its distributed nature and ability to keep multiple copies of a file in different locations. The amount of data means that bandwidth, latency, and cost are important factors and the reason most storage solutions are on-premises. However, there are distinct advantages to hosting services in the cloud, namely scalability and ease-of-use. In this paper, we explore the possibility of provisioning a hybrid Ceph cluster that leverages the benefits of both cloud architectures and on-premise performance.

97 MATHEMATICS AND COMPUTING↗

Tau (or no) leptons in top quark decays at hadron colliders

Measurements in the final states with taus or with no-leptons are among the most challenging as they are those with the smallest signal-to-background ratio. However, these final states are of particular interest as they can be important probes of new physics. Tau identification techniques and cross section measurements in top quark decays in these final states are discussed. The results, limited by systematical uncertainties, are consistent with standard model predictions, and are used to setstringent limits on new physics searches. The large data samples available at the Fermilab and at the Large Hadron Collider may help further improving the measurements.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

A Comprehensive Characterization of Galaxy-cool CGM Connections at z < 0.4 with DESI Year 1 Data

We investigate the relationships between the cool circumgalactic medium (CGM), traced by Ca II absorption lines, and galaxy properties at z < 0.4 using ∼900,000 galaxy–quasar pairs within 200 kpc from the Year 1 data of the Dark Energy Spectroscopic Instrument (DESI). This large data set enables us to obtain composite spectra with sensitivity reaching to the mÅ level and to explore the Ca II absorption as a function of stellar mass, star formation rate (SFR), redshift, and galaxy types, including active galactic nuclei (AGNs). Our results show a positive correlation between the absorption strength and stellar mass of star-forming galaxies with $\langle$$W$$^{Ca II}_{0}$$\rangle$ α $M$$^{0.5}_{*}$ over 3 orders of magnitude in stellar mass from ∼10 8 to 10 11 M ⊙ , while such a mass dependence is weaker for quiescent galaxies. At a fixed mass, Ca II absorption is stronger around star-forming galaxies than quiescent ones especially within impact parameters <30 kpc. Among star-forming galaxies, the Ca II absorption further correlates with SFR, following ∝SFR 0.3 . However, in contrast to the results at higher redshifts, stronger absorption is not preferentially observed along the minor axis of star-forming galaxies, indicating a possible redshift evolution of CGM dynamics resulting from galactic feedback. Moreover, no significant difference between the properties of the cool gas around AGNs and galaxies is detected. Finally, we measure the absorption profiles with respect to the virial radius of dark matter halos and show that the total Ca II mass in the CGM is comparable to the Ca mass in the ISM of galaxies.

Circumgalactic medium↗

BioPig: Developing Cloud Computing Applications for Next-Generation Sequence Analysis

Next Generation sequencing is producing ever larger data sizes with a growth rate outpacing Moore's Law. The data deluge has made many of the current sequenceanalysis tools obsolete because they do not scale with data. Here we present BioPig, a collection of cloud computing tools to scale data analysis and management. Pig is aflexible data scripting language that uses Apache's Hadoop data structure and map reduce framework to process very large data files in parallel and combine the results.BioPig extends Pig with capability with sequence analysis. We will show the performance of BioPig on a variety of bioinformatics tasks, including screeningsequence contaminants, Illumina QA/QC, and gene discovery from metagenome data sets using the Rumen metagenome as an example.

59 BASIC BIOLOGICAL SCIENCES↗

Data Storage for HEP Experiments in the Era of High-Performance Computing

As particle physics experiments push their limits on both the energy and the intensity frontiers, the amount and complexity of the produced data are also expected to increase accordingly. With such large data volumes, next-generation efforts like the HL-LHC and DUNE will rely even more on both high-throughput (HTC) and high-performance (HPC) computing clusters. Full utilization of HPC resources requires scalable and efficient data-handling and I/O. For the last few decades, ROOT has been used by most HEP experiments to store data. However, other storage technologies like HDF5 may perform better in HPC environments. Initial explorations with HDF5 have begun using ATLAS, CMS and DUNE data; the DUNE experiment has also adopted HDF5 for its data-acquisition system. This paper presents the future outlook of the HEP computing and the role of HPC, and a summary of ongoing and future works to use HDF5 as a possible data storage technology for the HEP experiments to use in HPC environments.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Mitigating Data Center Impact on Grid Stability: A Coordinated Control Strategy Using Verrus StabiliGrid Architecture

Large data centers, which now represent a significant and growing share of the total U.S. grid load, can inadvertently destabilize the electrical grid when they disconnect simultaneously during brief voltage disturbances. The July 10, 2024, Eastern Interconnection incident, in which a sub-100-millisecond transmission fault triggered the cascading loss of approximately 1,500 MW of data center load, illustrates this vulnerability. While commercial battery energy storage systems (BESS) deployed in data centers provide device-level fault ride-through per IEEE 1547, they lack coordination with facility protection logic and uninterruptible power supplies (UPS), limiting their effectiveness as grid-stabilizing assets. This report presents the Verrus StabiliGrid architecture, a coordinated control framework that integrates BESS, UPS, and point-of-interconnection (POI) protection settings to enable data centers to ride through both undervoltage and overvoltage grid contingencies without disconnecting. The four-step strategy encompasses: (1) high-resolution power quality monitoring to detect the grid state during events such as undervoltage, overvoltage, underfrequency, and overfrequency; (2) POI protection settings that allow for extended ride-through and grid-connected operation during grid contingencies; (3) grid state-driven autonomous dispatch of assets to improve grid resilience by reducing power draw during undervoltage or absorbing more power during overvoltage events; and (4) coordinated post-recovery dispatch of data center assets to restore firm load to pre-contingency levels. Validated through controller-hardware-in-the-loop (C-HIL) simulations at the National Laboratory of the Rockies, results show grid import restoration to pre-fault levels within 100 milliseconds of voltage recovery. This work advances the ability of data centers to transition from passive, disturbance-sensitive loads to active participants in grid stability, a capability increasingly required by emerging NERC and ERCOT regulatory frameworks.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Improving the Effectiveness of Burst Buffers for Big Data Processing in HPC Systems with Eley

Here, burst buffer is an effective solution for reducing the data transfer time and the I/O interference in HPC systems. Extending Burst Buffers (BBs) to handle Big Data applications is challenging because BBs must account for the large data inputs of Big Data applications and the Quality-of-Service (QoS) of HPC applications which are considered as first-class citizens in HPC systems. Existing BBs focus on only intermediate data of Big Data applications and incur a high performance degradation of both Big Data and HPC applications. We present Eley , a burst buffer solution that helps to accelerate the performance of Big Data applications while guaranteeing the QoS of HPC applications. To achieve this goal, Eley embraces interference-aware prefetching technique that makes reading data input faster while introducing low interference for HPC applications. Evaluations using a wide range of Big Data and HPC applications demonstrate that Eley improves the performance of Big Data applications by up to 30% compared to existing BBs while maintaining the QoS of HPC applications.

97 MATHEMATICS AND COMPUTING↗

Opportunities for an Integrated Web-Based Workbench for Data Access and Analysis - 20244

Management of environmental issues can require integration of multiple types of data and information, conducting data analysis and interpretation, and providing data visualization for effective communications. These data elements are important for site management to support regulator interactions and provide defensibility for remedial decisions. Databases and information repositories are core elements of managing data; however, efficient data access and analysis also enable effective site management. The U.S. Department of Energy (DOE) Hanford Site is an example of a complex site with a voluminous quantity of environmental data and a need for efficient site management. Different tiers of data and information tools have been developed and deployed to address site needs. These tools are configured for ready access via the web site interfaces and meet the rigorous quality requirements for environmental site management. Evolving efforts are focused on an integrated platform to meet site environmental management needs. In this platform, users can access site information at multiple levels of detail based on their need and permissions, so that data and associated analyses are presented within the context of the site mission and the user's management or technical needs. This concept is not only applicable at individual sites like Hanford but also applicable at other sites within the DOE complex. An integrated web-based architecture that links data visualization, data analytics, and management tools can provide holistic access to large data sets, minimize complexity, and maximize interactivity and technical communication. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

AtomSets as a hierarchical transfer learning framework for small and large materials datasets

Abstract Predicting properties from a material’s composition or structure is of great interest for materials design. Deep learning has recently garnered considerable interest in materials predictive tasks with low model errors when dealing with large materials data. However, deep learning models suffer in the small data regime that is common in materials science. Here we develop the AtomSets framework, which utilizes universal compositional and structural descriptors extracted from pre-trained graph network deep learning models with standard multi-layer perceptrons to achieve consistently high model accuracy for both small compositional data (<400) and large structural data (>130,000). The AtomSets models show lower errors than the graph network models at small data limits and other non-deep-learning models at large data limits. They also transfer better in a simulated materials discovery process where the targeted materials have property values out of the training data limits. The models require minimal domain knowledge inputs and are free from feature engineering. The presented AtomSets model framework can potentially accelerate machine learning-assisted materials design and discovery with less data restriction.

Chen, Chi (ORCID:0000000180087043)↗

THE EAST-WEST METHOD: AN EXPOSURE-INDEPENDENT METHOD TO SEARCH FOR LARGE-SCALE ANISOTROPIES OF COSMIC RAYS

The measurement of large-scale anisotropies in cosmic ray arrival directions at energies above 10{sup 13} eV is performed through the detection of extensive air showers (EAS) produced by cosmic ray interactions in the atmosphere. The observed anisotropies are small, so accurate measurements require small statistical uncertainties, i.e., large data sets. These can be obtained by employing ground detector arrays with large extensions (from 10{sup 4} to 10{sup 9} m{sup 2}) and long operation time (up to 20 years). The control of such arrays is challenging and spurious variations in the counting rate due to instrumental effects (e.g., data taking interruptions or changes in the acceptance) and atmospheric effects (e.g., air temperature and pressure effects on EAS development) are usually present. These modulations must be corrected very precisely before performing standard anisotropy analyses, i.e., harmonic analysis of the counting rate versus local sidereal time. In this paper we discuss an alternative method to measure large-scale anisotropies, the 'East-West method'. It was originally proposed in the 1940s to study asymmetries in the flux of solar cosmic rays and later applied by Nagashima et al. to EAS at higher energies. It is a differential method, as it is based on the analysis of the difference of the counting rates in the east and west directions. Besides explaining the principle, we present here its mathematical derivation, showing that the method is largely independent of experimental effects, that is, it does not require corrections for acceptance and/or for atmospheric effects. We explain the use of the method to derive the amplitude and phase of the anisotropy and demonstrate its power under different conditions of detector operation.

79 ASTRONOMY AND ASTROPHYSICS↗