Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,567 records · Page 87

Towards Autonomous Experiments by Connecting High Performance Microscopy with High Performance Computing

The digitization of controls, data, and analysis in microscopy is bringing the idea of autonomous microscopes closer to reality than ever before. Automated transmission electron microscopy (TEM) is already fairly routine for some experiments the only require simple repetitive tasks such as imaging biological macromolecules for single particle cryoEM [1], tilt series for electron tomography [2], and movies for crystallography [3]. The vast majority of TEM experiments are conducted completely by human operators who choose the regions of interest, optimize experimental parameters, and make decisions about data quality visually during an experiment. The field is still a long way from having completely autonomous TEMs that can adapt to sample difficulties and tune experimental parameters based on data quality and desired experimental outcomes. Part of the issue is the lack of capability for feeding information learned from on-line, live data analysis back into the on-going experiment [4]. Furthermore, this presentation will discuss current capabilities for large scale data reduction and analysis using high performance computing (i.e. supercomputing) and progress towards developing a true feed-back loop that places data analysis and theory in the experimental loop.

97 MATHEMATICS AND COMPUTING↗

The University of Virginia Experimental and Theoretical High Energy Physics Final Closeout Report

The work covered in this report includes a joint project on using gauge-gravity duality to discover qualitatively new results on jet quenching in strongly-coupled QCD-like plasmas. Other topics addressed by the theoretical work include jet stopping and energy loss in weakly-coupled plasmas, perturbative QCD amplitudes, AdS/CMT, dynamical electroweak symmetry breaking with a heavy fourth generation, electroweak-scale ν R model, vacuum topological structure and chiral dynamics in strongly coupled gauge theory. Effort committed to the CMS experiment is reported, particularly the management, maintenance, operation and upgrade of the CMS electromagnetic detector (ECAL). Activities in various physics analyses including Supersymmetry, Higgs, Top, and QCD analyses are reported. Physics projects covering wide areas of physics at the LHC are reported. CY2010 saw the accumulation of a data sample corresponding to approximately 36 pb -1 ; in CY 2011 the data sample swelled to more than 5 fb -1 . The UVa CMS analysis efforts are focused on this large 2011 data sample in a suite of crucial measurements and searches. KTeV physics activities are reported. Efforts are reported pertaining to several experiments, including: HyperCP, CKM, MIPP, D, NO νA, and Mu2e.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Global Temperature Responses to Large Tropical Volcanic Eruptions in Paleo Data Assimilation Products and Climate Model Simulations Over the Last Millennium

Abstract Large volcanic eruptions are one of the dominant perturbations to global and regional atmospheric temperatures on timescales of years to decades. Discrepancies remain, however, in the estimated magnitude and persistence of the surface temperature cooling caused by volcanic eruptions, as characterized by paleoclimatic proxies and climate models. We investigate these discrepancies in the context of large tropical eruptions over the Last Millennium using two state‐of‐the‐art data assimilation products, the Paleo Hydrodynamics Data Assimilation product (PHYDA) and the Last Millennium Reanalysis (LMR), and simulations from the National Center for Atmospheric Research Community Earth System Model‐Last Millennium Ensemble (NCAR CESM‐LME). We find that PHYDA and LMR estimate mean global and hemispheric cooling that is similar in magnitude and persistence once effects from eruptions occurring in short succession are removed. The estimates also compare well to Northern‐Hemisphere reconstructions based solely or partially on tree‐ring density, which have been proposed as the most accurate proxy estimates of surface cooling due to volcanism. All proxy‐based estimates also agree well with the magnitude of the mean cooling simulated by the CESM‐LME. Differences remain, however, in the spatial patterns of the temperature responses in the PHYDA, LMR, and the CESM‐LME. The duration of cooling anomalies also persists for several years longer in the PHYDA and LMR relative to the CESM‐LME. Our results demonstrate progress in resolving discrepancies between proxy‐ and model‐based estimates of temperature responses to volcanism, but also indicate these estimates must be further reconciled to better characterize the risks of future volcanic eruptions.

Tejedor, E.↗

Solar coronal structure: A comparison of NSO/SP ground-based coronal emission line intensities and temperatures with Yohkoh SXT and WSO magnetic data

The large-scale structure of the solar corona is investigated using synoptic maps produced from Fe XIV (530.3 nm), Fe X (637.4 nm) and Ca XV (569.4 nm) data obtained at National Solar Observatory (NSO/SP), Yohkoh/Soft X-ray Telescope (SXT) X-ray data and Wilcox Solar Observatory (WSO) 'source surface' maps. We find that the Fe XIV data are an excellent proxy for spatially-average Yohkoh/SXT data. Isolated emission features and large-scale structures are nearly identical in SXT and Fe XIV maps. In addition, coronal holes and other low-emission regions are very similar. Synoptic temperature maps, calculated from the Fe X/Fe XIV ratio, show a tendency for the highest temperatures to occur where the large-scale magnetic fields change polarity at high latitudes, while lower-latitude features, including active regions, have lower apparent tempertures. Regions of enhanced temperature generally follow the helisopheric current sheet (HCS) as defined by the WSO maps. Further, emission in Ca XV (formed at T is approximately equal to 3 MK), generally occurs only over low-latitude regions that are bright in both FE X (T approximately equal to 1 MK) and Fe XIV (T approximately equal to 2 MK). Thus, there is evidence for low (approximately equal to 1 MK), moderate (approximately equal to 2 MK) and high (approximately 3 MK) temperatures in close proximity in the low corona.

Altrock, R. C.↗

Geomancy: Automated Performance Enhancement through Data Layout Optimization

Large distributed storage systems such as high- performance computing (HPC) systems used by national or international laboratories require sufficient performance and scale for demanding scientific workloads and must handle shifting workloads with ease. Ideally, data is placed in locations to optimize performance, but the size and complexity of large storage systems inhibit rapid effective restructuring of data layouts to maintain performance as workloads shift. To address these issues, we have developed Geomancy, a tool that models the placement of data within a distributed storage system and reacts to drops in performance. Using a combination of machine learning techniques suitable for temporal modeling, Geomancy determines when and where a bottleneck may happen due to changing workloads and suggests changes in the layout that mitigate or prevent them. Our approach to optimizing throughput offers benefits for storage systems such as avoiding potential bottlenecks and increasing overall I/O throughput from 11% to 30%.

Bel, Oceane M.↗

Variability of Load and Net Load in Case of Large Scale Distributed Wind Power

Large scale wind power production and its variability is one of the major inputs to wind integration studies. This paper analyses measured data from large scale wind power production. Comparisons of variability are made across several variables: time scale (10-60 minute ramp rates), number of wind farms, and simulated vs. modeled data. Ramp rates for Wind power production, Load (total system load) and Net load (load minus wind power production) demonstrate how wind power increases the net load variability. Wind power will also change the timing of daily ramps.

17 WIND ENERGY↗

HydroForecast Long-term: Improving hydropower’s resilience to climate change through accurate climate-scale

With hydrologic patterns and water availability across the globe shifting due to climate change, advancements in hydrologic prediction systems can help significantly reduce the uncertainties that utilities and water supply entities have in their decision making. Understanding and estimating hydrology at the climate scale is critical for managing water resources under changing climate scenarios. This project focuses on integrating state-of-the-art neural network modeling with downscaled climate projections to deliver the reliable water supply projections decades into the future to meet an urgent need from hydropower operators and water utilities. In this Phase 1 DOE SBIR proposal, we developed and validated a theory-guided neural network model, HydroForecast Long-term, for climate-scale hydrology and implemented the model within existing HydroForecast infrastructure. HydroForecast Long-term combines the most accurate streamflow modeling system with a flexible and scalable data architecture to generate water supply projections out to the year 2100. This report illustrates that we have achieved our four objectives: 1) create a prototype of HydroForecast Long-term, building the neural network prediction model, 2) build an automated data input pipeline that processes large amounts of data from the latest global temperature and precipitation climate models; 3) benchmark the accuracy of the hydrologic model over the recent two decades over a large set of diverse basins, and 4) create a set of output visuals and summary metrics informed by customer feedback that connect the data to critical decision points. This work empowers water users to make data-informed decisions supporting a resilient, renewable-powered grid and water system. The results advance the Department of Energy’s mission by addressing critical gaps in water supply planning under climate change.

13 HYDRO ENERGY↗

Noise generated by impingement of a jet upon a large flat board

Data were obtained on the noise generated by an air jet impinging on a large flat board. The board was large enough so that the flow leaving the edges of the board generated no significant noise. The impingement angle, nozzle shape and size, jet velocity, and the distance from the nozzle to the board were varied in the experiment. Far-field noise data are presented. The nozzle-alone noise contribution to the total noise was generally small and was subtracted from the total, leaving the impingement-only noise. The impingement-only noise was adequately correlated by eighth power of the peak impingement velocity and first power of the impingement area. The spectral data were correlated by a Strouhal number based on the peak impingement velocity and a characteristic impingement diameter.

Olsen, W. A.↗

The application of artificial intelligence techniques to large distributed networks

Data accessibility and transfer of information, including the land resources information system pilot, are structured as large computer information networks. These pilot efforts include the reduction of the difficulty to find and use data, reducing processing costs, and minimize incompatibility between data sources. Artificial Intelligence (AI) techniques were suggested to achieve these goals. The applicability of certain AI techniques are explored in the context of distributed problem solving systems and the pilot land data system (PLDS). The topics discussed include: PLDS and its data processing requirements, expert systems and PLDS, distributed problem solving systems, AI problem solving paradigms, query processing, and distributed data bases.

Dubyah, R.↗

A scalable transformer model for real-time decision making in neutron scattering experiments

The U.S. Department of Energy's (DOE's) neutron research facilities at Oak Ridge National Laboratory (ORNL), including the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS), are a state-of-the-art neutron scattering facility that allows researchers to study the structure and dynamics of materials at the atomic scale. At the SNS, neutrons are measured using the time-of-flight (TOF) technique as they move through a neutron beamline to interact with a sample. Large volumes of neutron scattering data are collected and recorded in neutron event mode. Optimal productivity of the TOF instrument is limited due to the lack of real-time data analysis tools. The large amount of data generated by the experiments can be challenging to process and analyze in real time, particularly for experiments that require rapid feedback and adjustment of experimental parameters. The regular computer/workstation cannot keep up with the experiment speed to provide real-time feedback to adjust experimental parameters, so connecting the supercomputers available to the neutron facility is necessary to achieve real-time data analysis and experiment steering. To address this challenge, we exploit the Frontier supercomputer at Oak Ridge Leadership Computing Facility (OLCF) to train a scalable temporal fusion transformer model for real-time decision making of TOF neutron scattering experimentation. Here, in this paper, we present the results using Frontier to provide the processing power needed to rapidly process and analyze large volumes of single-crystal diffraction data collected at TOPAZ, a neutron time-of-flight Laue single-crystal diffractometer at the SNS.

97 MATHEMATICS AND COMPUTING↗

Parallel Hybrid Metaheuristics with Distributed Intensification and Diversification for Large-scale Optimization in Big Data Statistical Analysis

Important insights into many problems that are traditionally analyzed via statistical models can be obtained by re-formulating and evaluating within a large-scale optimization framework. The theoretical underpinnings of the statistical model often shift the goal of the solution space traversal from a traditional search for a single optimal solution to a traversal with the purpose of yielding a set of high quality, independent solutions. We examine statistical frameworks with astronomical solution spaces where the independence requirement constitutes a significant additional challenge for standard optimization methodologies. We design a hybrid metaheuristic with specialized intensification and diversification protocols in the base search algorithm. We extend our algorithm to the high-performance-computing realm using the Stampede2 supercomputer. We experimentally demonstrate the effectiveness of our algorithm to utilize multiple processors to collaboratively hill climb, broadcast messages to one another regarding landscape characteristics, diversify across the solution landscape, and request aid in climbing particularly difficult peaks.

Cho, Wendy K. Tam↗

Scalable Performance Measurement and Analysis

Concurrency levels in large-scale, distributed-memory supercomputers are rising exponentially. Modern machines may contain 100,000 or more microprocessor cores, and the largest of these, IBM's Blue Gene/L, contains over 200,000 cores. Future systems are expected to support millions of concurrent tasks. In this dissertation, we focus on efficient techniques for measuring and analyzing the performance of applications running on very large parallel machines. Tuning the performance of large-scale applications can be a subtle and time-consuming task because application developers must measure and interpret data from many independent processes. While the volume of the raw data scales linearly with the number of tasks in the running system, the number of tasks is growing exponentially, and data for even small systems quickly becomes unmanageable. Transporting performance data from so many processes over a network can perturb application performance and make measurements inaccurate, and storing such data would require a prohibitive amount of space. Moreover, even if it were stored, analyzing the data would be extremely time-consuming. In this dissertation, we present novel methods for reducing performance data volume. The first draws on multi-scale wavelet techniques from signal processing to compress systemwide, time-varying load-balance data. The second uses statistical sampling to select a small subset of running processes to generate low-volume traces. A third approach combines sampling and wavelet compression to stratify performance data adaptively at run-time and to reduce further the cost of sampled tracing. We have integrated these approaches into Libra, a toolset for scalable load-balance analysis. We present Libra and show how it can be used to analyze data from large scientific applications scalably.

99 GENERAL AND MISCELLANEOUS↗

Automatic Search of Cataclysmic Variables Based on LightGBM in LAMOST-DR7

The search for special and rare celestial objects has always played an important role in astronomy. Cataclysmic Variables (CVs) are special and rare binary systems with accretion disks. Most CVs are in the quiescent period, and their spectra have the emission lines of Balmer series, HeI, and HeII. A few CVs in the outburst period have the absorption lines of Balmer series. Owing to the scarcity of numbers, expanding the spectral data of CVs is of positive significance for studying the formation of accretion disks and the evolution of binary star system models. At present, the research for astronomical spectra has entered the era of Big Data. The Large Sky Area Multi-Object Fiber Spectroscopy Telescope (LAMOST) has produced more than tens of millions of spectral data. the latest released LAMOST-DR7 includes 10.6 million low-resolution spectral data in 4926 sky regions, providing ideal data support for searching CV candidates. To process and analyze the massive amounts of spectral data, this study employed the Light Gradient Boosting Machine (LightGBM) algorithm, which is based on the ensemble tree model to automatically conduct the search in LAMOST-DR7. Finally, 225 CV candidates were found and four new CV candidates were verified by SIMBAD and published catalogs. This study also built the Gradient Boosting Decision Tree (GBDT), Adaptive Boosting (AdaBoost), and eXtreme Gradient Boosting (XGBoost) models and used Accuracy, Precision, Recall, the F1-score, and the ROC curve to compare the four models comprehensively. Experimental results showed that LightGBM is more efficient. The search for CVs based on LightGBM not only enriches the existing CV spectral library, but also provides a reference for the data mining of other rare celestial objects in massive spectral data.

79 ASTRONOMY AND ASTROPHYSICS↗

Data Management Symposium, Huntsville, Ala., October 18, 19, 1977, Proceedings

The transfer, processing, and use of satellite data are discussed. Topics relating to simulation and processing techniques include computer design assessments through simulation, data system dynamic simulation, and future timing accuracy requirements and procedures for data processing. Subjects relevant to data base management systems and users and their needs include an image-based information system architecture for correlation satellite and topological data base, efficient searching and sorting applications using an associative array processor, analysis of user's needs for a large technical data base, and variable length data formats. The technology development outlook was considered with attention to data systems technology outlook for supporting NASA programs, a user-oriented interactive information extraction system, and flow control and sorting techniques for telemetry packets.

Castelli, A.↗

The 'Weekend Effect' in Tropospheric NO2 Seen from the Ozone Monitoring Instrument

The Ozone Monitoring Instrument has gathered daily global data on NO2 and other atmospheric trace gases since its launch on the EOS Aura satellite in 2004. The large accumulated data set makes it possible to monitor changes of both meteorological and anthropogenic origin in tropospheric NOz amounts. In particular, averages on time scales on the order of a year show a distinct 'weekend effect' in NO2 variation, with smaller NO2 amounts seen on Saturday and/or Sunday than on the remaining weekdays. Using the OMI NO2 Standard Product (SP), we examine this effect in relation to geopolitical boundaries and investigate implications for identifying sources. We also use the SP data to find evidence for other short-term anthropogenic changes in NO2 emissions over heavily polluted regions including the United States, Europe and China.

Bucsela, E.↗

Electra: A Modular-Based Expansion of NASA's Supercomputing Capability

NASA has increasingly relied on high-performance computing (HPC) re- sources for computational modeling, simulation, and data analysis to meet the science and engineering goals of its missions in space exploration, aeronautics, and Earth and space science. The NASA Advanced Supercomputing (NAS) Division at Ames Research Center in Silicon Valley, Calif., hosts NASA’s premier supercomputing resources, integral to achieving and enhancing the success of the agency’s missions. NAS provides a balanced environment, funded under the High-End Computing Capability (HECC) project, comprised of world-class supercomputers, including its flagship distributed-memory cluster, Pleiades; high-speed networking; and massive data storage facilities, along with multi-disciplinary support teams for user support, code porting and optimization, and large-scale data analysis and scientific visualization. However, as scientists have increased the fidelity of their simulations and engineers are conducting larger parameter-space studies, the requirements for supercomputing resources have been growing by leaps and bounds. With the facility housing the HECC systems reaching its power and cooling capacity, NAS undertook a prototype project to investigate an alternative approach for housing supercomputers. Modular supercomputing, or container-based computing, is an innovative concept for expanding NASA’s HPC capabilities. With modular supercomputing, additional containers—similar to portable storage pods—can be connected together as needed to accommodate the agency’s ever-increasing demand for computing resources. In addition, taking advantage of the local weather permits the use of cooling technologies that would additionally save energy and reduce annual water usage. The first stage of NASA’s Modular Supercomputing Facility (MSF) prototype, which resulted in a 1,000 square-foot module on a concrete pad with room for 16 compute racks, was completed in Fall 2016 and an SGI (now HPE) computer system, named Electra, was deployed there in early 2017. Cooling is performed via an evaporative system built into the module, and preliminary experience shows a Power Usage Effectiveness (PUE) measurement of 1.03. Electra achieved over a petaflop on the LINPACK benchmark, sufficient to rank number 96 on the November 2016 TOP500 list [14]. The system consists of 1,152 InfiniBand-connected Intel Xeon Broadwell-based nodes. Its users access their files on a facility-wide file system shared by all HECC compute assets via Mellanox MetroX InfiniBand extenders, which connect the Electra fabric to Lustre routers in the primary facility over fiber-optic links about 900 feet long. The MSF prototype has exceeded expectations and is serving as a blueprint for future expansions. In the remainder of this chapter, we detail how modular data center technology can be used to expand an existing compute resource. We begin by describing NASA’s requirements for supercomputing and how resources were provided prior to the integration of the Electra module-based system.

Biswas, Rupak↗

Diverse Power Iteration Embeddings and Its Applications

Abstract—Spectral Embedding is one of the most effective dimension reduction algorithms in data mining. However, its computation complexity has to be mitigated in order to apply it for real-world large scale data analysis. Many researches have been focusing on developing approximate spectral embeddings which are more efficient, but meanwhile far less effective. This paper proposes Diverse Power Iteration Embeddings (DPIE), which not only retains the similar efficiency of power iteration methods but also produces a series of diverse and more effective embedding vectors. We test this novel method by applying it to various data mining applications (e.g. clustering, anomaly detection and feature selection) and evaluating their performance improvements. The experimental results show our proposed DPIE is more effective than popular spectral approximation methods, and obtains the similar quality of classic spectral embedding derived from eigen-decompositions. Moreover it is extremely fast on big data applications. For example in terms of clustering result, DPIE achieves as good as 95% of classic spectral clustering on the complex datasets but 4000+ times faster in limited memory environment.

97 MATHEMATICS AND COMPUTING↗

Scaling Laws of Graph Neural Networks for Atomistic Materials Modeling

Atomistic materials modeling is a critical task with wide-ranging applications, from drug discovery to materials science, where accurate predictions of the target material property can lead to significant advancements in scientific discovery. Graph Neural Networks (GNNs) represent the state-of-the-art approach for modeling atomistic material data thanks to their capacity to capture complex relational structures. While machine learning performance has historically improved with larger models and datasets, GNNs for atomistic materials modeling remain relatively small compared to large language models (LLMs), which leverage billions of parameters and terabyte-scale datasets to achieve remarkable performance in their respective domains. To address this gap, we explore the scaling limits of GNNs for atomistic materials modeling by developing a foundational model with billions of parameters, trained on extensive datasets in terabytescale. Our approach incorporates techniques from LLM libraries to efficiently manage large-scale data and models, enabling both effective training and deployment of these large-scale GNN models. This work addresses three fundamental questions in scaling GNNs: the potential for scaling GNN model architectures, the effect of dataset size on model accuracy, and the applicability of LLM-inspired techniques to GNN architectures. Specifically, the outcomes of this study include (1) insights into the scaling laws for GNNs, highlighting the relationship between model size, dataset volume, and accuracy, (2) a foundational GNN model optimized for atomistic materials modeling, and (3) a GNN codebase enhanced with advanced LLM-based training techniques. Our findings lay the groundwork for large-scale GNNs with billions of parameters and terabyte-scale datasets, establishing a scalable pathway for future advancements in atomistic materials modeling.

Li, Chaojian [ORNL] (ORCID:0000000340309777)↗