Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,009 records · Page 56

Method and system for data clustering for very large databases

Multi-dimensional data contained in very large databases is efficiently and accurately clustered to determine patterns therein and extract useful information from such patterns. Conventional computer processors may be used which have limited memory capacity and conventional operating speed, allowing massive data sets to be processed in a reasonable time and with reasonable computer resources. The clustering process is organized using a clustering feature tree structure wherein each clustering feature comprises the number of data points in the cluster, the linear sum of the data points in the cluster, and the square sum of the data points in the cluster. A dense region of data points is treated collectively as a single cluster, and points in sparsely occupied regions can be treated as outliers and removed from the clustering feature tree. The clustering can be carried out continuously with new data points being received and processed, and with the clustering feature tree being restructured as necessary to accommodate the information from the newly received data points.

Zhang, Tian↗

Development of the Model of Galactic Interstellar Emission for Standard Point-Source Analysis of Fermi Large Area Telescope Data

Most of the celestial gamma rays detected by the Large Area Telescope (LAT) on board the Fermi Gamma-ray Space Telescope originate from the interstellar medium when energetic cosmic rays interact with interstellar nucleons and photons. Conventional point-source and extended-source studies rely on the modeling of this diffuse emission for accurate characterization. Here, we describe the development of the Galactic Interstellar Emission Model (GIEM),which is the standard adopted by the LAT Collaboration and is publicly available. This model is based on a linear combination of maps for interstellar gas column density in Galactocentric annuli and for the inverse-Compton emission produced in the Galaxy. In the GIEM, we also include large-scale structures like Loop I and the Fermi bubbles. The measured gas emissivity spectra confirm that the cosmic-ray proton density decreases with Galactocentric distance beyond 5 kpc from the Galactic Center. The measurements also suggest a softening of the proton spectrum with Galactocentric distance. We observe that the Fermi bubbles have boundaries with a shape similar to a catenary at latitudes below 20deg and we observe an enhanced emission toward their base extending in the north and south Galactic directions and located within approximately 4deg of the Galactic Center.

gamma rays: diffuse background – gamma rays: gen↗

Machine learning-based inversion for acoustic impedance with large synthetic training data: Workflow and data characterization

Where wells are sparse or training data are difficult to label with high-quality wireline-derived impedance logs, machine learning (ML)-based inversion of acoustic impedance typically depends on small training data sets, leading to biased prediction. We have advanced a novel workflow that applies large synthetic seismic training data to reduce facies-related bias. Using a geologically realistic model as the truth model, we randomly select sparse seed wells to perform sequential Gaussian simulation (SGS) for impedance models of the same geometry and simulate facies variability. We implement random forest regression on 30 features extracted from the synthetic volume. We observe that more seed wells tend to reduce facies-induced bias by sampling more types of facies, resulting in a better prediction. We then focus on the responses of SGS models to facies changes, the number of seed wells necessary for a useful synthetic model, and how much a synthetic model can help ML-based inversion. Here, we observe that the SGS synthetic training model outperforms well-direct training in general. For modeled clastic shore-zone systems in Miocene Gulf of Mexico, two or more seed wells are necessary for a significant reduction of root-mean-square error and outliners, and improvement of facies imaging. In a field-data test, we apply a similar workflow to quantitatively predict acoustic impedance, which is then converted to a sand-volume map at a high-frequency sequence (10–100 m), revealing detailed facies and sandstone patterns. Such results are valuable in many geologic and engineering applications, such as hydrocarbon and CO 2 reservoir prospecting, reserve estimation, simulation, etc.

3D seismic↗

A Testbed Demonstration of an Intelligent Archive in a Knowledge Building System

The last decade's influx of raw data and derived geophysical parameters from several Earth observing satellites to NASA data centers has created a data-rich environment for Earth science research and applications. While advances in hardware and information management have made it possible to archive petabytes of data and distribute terabytes of data daily to a broad community of users, further progress is necessary in the transformation of data into information, and information into knowledge that can be used in particular applications in order to realize the full potential of these valuable datasets. In examining what is needed to enable this progress in the data provider environment that exists today and is expected to evolve in the next several years, we arrived at the concept of an Intelligent Archive in context of a Knowledge Building System (IA/KBS). Our prior work and associated papers investigated usage scenarios, required capabilities, system architecture, data volume issues, and supporting technologies. We identified six key capabilities of an IA/KBS: Virtual Product Generation, Significant Event Detection, Automated Data Quality Assessment, Large-Scale Data Mining, Dynamic Feedback Loop, and Data Discovery and Efficient Requesting. Among these capabilities, large-scale data mining is perceived by many in the community to be an area of technical risk. One of the main reasons for this is that standard data mining research and algorithms operate on datasets that are several orders of magnitude smaller than the actual sizes of datasets maintained by realistic earth science data archives. Therefore, we defined a test-bed activity to implement a large-scale data mining algorithm in a pseudo-operational scale environment and to examine any issues involved. The application chosen for applying the data mining algorithm is wildfire prediction over the continental U.S. This paper reports a number of observations based on our experience with this test-bed. While proof-of-concept for data mining scalability and utility has been a major goal for the research reported here, it was not the only one. The other five capabilities of an WKBS named above have been considered as well, and an assessment of the implications of our experience for these other areas will also be presented. The lessons learned through the testbed effort and presented in this paper will benefit technologists, scientists, and system operators as they consider introducing IA/KBS capabilities into production systems.

Ramapriyan, Hampapuram↗

Impact of transportation network companies on urban congestion: Evidence from large-scale trajectory data

We collect vehicle trajectory data from major transportation network companies (TNCs) in New York City (NYC) in 2017 and 2019, and we use the trajectory data to understand how the growth of TNCs has impacted traffic congestion and emission in urban areas. By mining the large-scale trajectory data and conduct the case study in NYC, we confirm that the rise of TNC is the major contributing factor that makes urban traffic congestion worse. From 2017 to 2019, the number of for-hire vehicles (FHV) has increased by over 48% and served 90% more daily trips. These resulted in an average citywide speed reduction of 22.5% on weekdays, and the average speed in Manhattan decreased from 11.76 km/h in April 2017 to 9.56 km/h in March 2019. The heavier traffic congestion may have led to 136% more NOx, 152% more CO and 157% more HC emission per kilometer traveled by the FHV sector. Our results show that the traffic condition is consistently worse across different times of the day and at different locations in NYC. And we build the connection between the number of available FHVs and the reduction in travel speed between the two years of data and explain how the rise of TNC may impact traffic congestion in terms of moving speed and congestion time. Our findings provide valuable insights for different stakeholders and decision-makers in framing regulation and operation policies towards more effective and sustainable urban mobility.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Ozone climatology at Natal, Brazil, from in situ ozonesonde data

A large ozone-profile data set has been obtained through balloon ozonesonde soundings at Natal, Brazil, during 1978-1988. Maximum ozone concentrations occur during local spring (September-October), and minimum concentrations during late autumn (April-May); the seasonal variation is much larger in the troposphere than in the stratosphere. If there were no seasonal variation in the stratosphere, the seasonal variation observed in the troposphere alone would be sufficient to drive a total ozone column variation of about 5 percent. This is about half the size of the variation observed in the Natal Dobson spectrophotometer data.

Kirchhoff, V. W. J. H.↗

IMPLICATION FOR THE CORE-COLLAPSE SUPERNOVA RATE FROM 21 YEARS OF DATA OF THE LARGE VOLUME DETECTOR

The Large Volume Detector (LVD) has been continuously taking data since 1992 at the INFN Gran Sasso National Laboratory. The LVD is sensitive to neutrino bursts from gravitational stellar collapses with full detection probability over the Galaxy. We have searched for neutrino bursts in LVD data taken over 7,335 days of operation. No evidence of neutrino signals has been found between 1992 June and 2013 December. The 90% C.L. upper limit on the rate of core collapse and failed supernova explosions out to distances of 25 kpc is found to be 0.114 yr{sup −1}.

79 ASTRONOMY AND ASTROPHYSICS↗

Exploring the Performance of Spark for a Scientific Use Case

We present an evaluation of the performance of a Spark implementation of a classification algorithm in the domain of High Energy Physics (HEP). Spark is a general engine for in-memory, large-scale data processing, and is designed for applications where similar repeated analysis is performed on the same large data sets. Classification problems are one of the most common and critical data processing tasks across many domains. Many of these data processing tasks are both computation- and data-intensive, involving complex numerical computations employing extremely large data sets. We evaluated the performance of the Spark implementation on Cori, a NERSC resource, and compared the results to an untuned MPI implementation of the same algorithm. While the Spark implementation scaled well, it is not competitive in speed to our MPI implementation, even when using significantly greater computational resources.

Sehrish, Saba↗

Searching for Dark Matter Annihilation from Milky Way Dwarf Spheroidal Galaxies with Six Years of Fermi Large Area Telescope Data

The dwarf spheroidal satellite galaxies (dSphs) of the Milky Way are some of the most dark matter (DM) dominated objects known. We report on γ-ray observations of Milky Way dSphs based on six years of Fermi Large Area Telescope data processed with the new Pass8 event-level analysis. None of the dSphs are significantly detected in γ rays, and we present upper limits on the DM annihilation cross section from a combined analysis of 15 dSphs. These constraints are among the strongest and most robust to date and lie below the canonical thermal relic cross section for DM of mass ≲100 GeV annihilating via quark and τ-lepton channels.

79 ASTRONOMY AND ASTROPHYSICS↗

A Bloom Filter Based Scalable Data Integrity Check Tool for Large-Scale Dataset

Large scale HPC applications are becoming increasingly data intensive. At Oak Ridge Leadership Computing Facility (OLCF), we are observing the number of files curated under individual project are reaching as high as 200 millions and project data size is exceeding petabytes. These simulation datasets, once validated, often needs to be transferred to archival system for long term storage or shared with the rest of the research community. Ensuring the data integrity of the full dataset at this scale is paramount important but also a daunting task. This is especially true considering that most conventional tools are serial and file-based, unwieldy to use and/or can't scale to meet user's demand.To tackle this particular challenge, this paper presents the design, implementation and evaluation of a scalable parallel checksumming tool, fsum, which we developed at OLCF. It is built upon the principle of parallel tree walk and work-stealing pattern to maximize parallelism and is capable of generating a single, consistent signature for the entire dataset at extreme scale. We also applied a novel bloom-filter based technique in aggregating signatures to overcome the signature ordering requirement. Given the probabilistic nature of bloom filter, we provided a detailed error and trade-off analysis. Using multiple datasets from production environment, we demonstrated that our tool can efficiently handle both very large files as well as many small-file based datasets. Our preliminary test showed that on the same hardware, it outperforms conventional tool by as much as 4×. It also exhibited near-linear scaling properties when provisioned with more compute resources.

97 MATHEMATICS AND COMPUTING↗

The growth of density perturbations in the last ~10 billion years from tomographic large-scale structure data

In order to investigate the origin of the ongoing tension between the amplitude of matter fluctuations measured by weak lensing experiments at low redshifts and the value inferred from the cosmic microwave background anisotropies, we reconstruct the evolution of this amplitude from z ~ 2 using existing large-scale structure data. To do so, we decouple the linear growth of density inhomogeneities from the background expansion, and constrain its redshift dependence making use of a combination of 6 different data sets, including cosmic shear, galaxy clustering and CMB lensing. We analyze these data under a consistent harmonic-space angular power spectrum-based pipeline. We show that current data constrain the amplitude of fluctuations mostly in the range 0.2 < z < 0.7, where it is lower than predicted by . This difference is mostly driven by current cosmic shear data, although the growth histories reconstructed from different data combinations are consistent with each other, and we find no evidence of systematic deviations in any particular experiment. In spite of the tension with , the data are well-described by the model, albeit with a lower value of S 8 ≡ σ 8 (Ω m /0.3) 0.5 . As part of our analysis, we find constraints on this parameter of S 8 = 0.7781 ± 0.0094 (68% confidence level), reaching almost percent-level errors comparable with CMB measurements, and 3.4σ away from the value found by .

79 ASTRONOMY AND ASTROPHYSICS↗

Large-Scale Inference of Multivariate Regression for Heavy-Tailed and Asymmetric Data

Large-scale multivariate regression is a fundamental statistical tool with a wide range of applications. Here, this study considers the problem of simultaneously testing a large number of general linear hypotheses, encompassing covariate-effect analysis, analysis of variance, and model comparisons. The challenge that accompanies a large number of tests is the ubiquitous presence of heavy-tailed and/or highly skewed measurement noise, which is the main reason for the failure of conventional least squares-based methods. For large-scale multivariate regression, we develop a set of robust inference methods to explore data features such as heavy tailedness and skewness, which are not visible to least squares methods. The new testing procedure is based on the data-adaptive Huber regression and a new covariance estimator of regression estimates. Under mild conditions, we show that our methods produce consistent estimates of the false discovery proportion. Extensive numerical experiments and an empirical study on quantitative linguistics demonstrate the advantage of the proposed method over many state-of-the-art methods when the data are generated from heavy-tailed and/or skewed distributions.

97 MATHEMATICS AND COMPUTING↗

2026 Large Load Literature Review and Data Sources [Slides]

The Large Load Literature Review and Data Sources reports are updated monthly, summarizing reports and data published in 2026 that focus on large loads. The 2025 literature review reviewed and categorized over 90 resources into 12 themes.

97 MATHEMATICS AND COMPUTING↗

Investigating wind turbine impacts on near-wake flow using profiling Lidar data and large-eddy simulations with an actuator disk model

In this study, wind turbine impacts on the atmospheric flow are investigated using data from the Crop Wind Energy Experiment (CWEX-11) and large-eddy simulations (LESs) utilizing a generalized actuator disk (GAD) wind turbine model. CWEX-11 employed velocity-azimuth display (VAD) data from two Doppler lidar systems to sample vertical profiles of flow parameters across the rotor depth both upstream and in the wake of an operating 1.5 MW wind turbine. Lidar and surface observations obtained during four days of July 2011 are analyzed to characterize the turbine impacts on wind speed and flow variability, and to examine the sensitivity of these changes to atmospheric stability. Significant velocity deficits (VD) are observed at the downstream location during both convective and stable portions of four diurnal cycles, with large, sustained deficits occurring during stable conditions. Variances of the streamwise velocity component, σ u , likewise show large increases downstream during both stable and unstable conditions, with stable conditions supporting sustained small increases of σ u , while convective conditions featured both larger magnitudes and increased variability, due to the large coherent structures in the background flow. Two representative case studies, one stable and one convective, are simulated using LES with a GAD model at 6 m resolution to evaluate the compatibility of the simulation framework with validation using vertically profiling lidar data in the near wake region. Virtual lidars were employed to sample the simulated flow field in a manner consistent with the VAD technique. Simulations reasonably reproduced aggregated wake VD characteristics, albeit with smaller magnitudes than observed, while σu values in the wake are more significantly underestimated. The results illuminate the limitations of using a GAD in combination with coarse model resolution in the simulation of near wake physics, and validation thereof using VAD data.

17 WIND ENERGY↗