Search NASA⌕ Search

SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22
All sourcesNASA NTRSDOE OSTI
NASA NTRS2016

Using Cloud-Based Storage Technologies for Earth Science Data

Cloud based infrastructure may offer several key benefits of scalability, built in redundancy and reduced total cost of ownership as compared with a traditional data center approach. However, most of the tools and software systems developed for NASA data repositories were not developed with a cloud based infrastructure in mind and do not fully take advantage of commonly available cloud-based technologies. Object storage services are provided through all the leading public (Amazon Web Service, Microsoft Azure, Google Cloud, etc.) and private (Open Stack) clouds, and may provide a more cost-effective means of storing large data collections online. We describe a system that utilizes object storage rather than traditional file system based storage to vend earth science data. The system described is not only cost effective, but shows superior performance for running many different analytics tasks in the cloud. To enable compatibility with existing tools and applications, we outline client libraries that are API compatible with existing libraries for HDF5 and NetCDF4. Performance of the system is demonstrated using clouds services running on Amazon Web Services.

Data↗
DOE OSTI2020

Getting to the Core of PARAFAC2, A Nonnegative Approach

In this paper, the authors present a novel method of performing PARAFAC2 factorization of three-way data using a compact representation of that data. In the standard PARAFAC2 algorithm, two modes of the data are recovered directly during the decomposition while the third mode is returned as a transformation matrix, which is then used to rotate sets of orthogonal third-mode basis factors into interpretable factors. In our new method, the data are first decomposed into a core matrix and orthogonal factor loading matrices in the first two modes as well as sets of orthogonal factors in the third mode (as in standard PARAFAC2). The core matrix is then decomposed using a the standard PARAFAC2 strategy to produce transformation matrices in all three modes. The algorithm is particularly useful for very large data sets and essentially permits imposition of nonnegativity in all three modes.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2017

Online Visualization and Value Added Services of MERRA-2 Data at GES DISC

NASA climate reanalysis datasets from MERRA-2, distributed at the Goddard Earth Sciences Data and Information Services Center (GES DISC), have been used in broad research areas, such as climate variations, extreme weather, agriculture, renewable energy, and air quality, etc. The datasets contain numerous variables for atmosphere, land, and ocean, grouped into 95 products. The total archived volume is approximately 337 TB ( approximately 562K files) at the end of October 2017. Due to the large number of products and files, and large data volumes, it may be a challenge for a user to find and download the data of interest. The support team at GES DISC, working closely with the MERRA-2 science team, has created and is continuing to work on value added data services to best meet the needs of a broad user community. This presentation, using aerosol over Asia Monsoon as an example, provides an overview of the MERRA-2 data services at GES DISC, including: How to find the data? How many data access methods are provided? What are the best data access methods for me? How do download the subsetted (parameter, spatial, temporal) data and save in preferred spatial resolution and data format? How to visualize and explore the data online? In addition, we introduce a future online analytic tool designed for supporting application research, focusing on long-term hourly time-series data access and analysis.

Shen, Suhung↗
NASA NTRS2015

Data Mining and Analysis

The Data Mining project seeks to bring the capability of data visualization to NASA anomaly and problem reporting systems for the purpose of improving data trending, evaluations, and analyses. Currently NASA systems are tailored to meet the specific needs of its organizations. This tailoring has led to a variety of nomenclatures and levels of annotation for procedures, parts, and anomalies making difficult the realization of the common causes for anomalies. Making significant observations and realizing the connection between these causes without a common way to view large data sets is difficult to impossible. In the first phase of the Data Mining project a portal was created to present a common visualization of normalized sensitive data to customers with the appropriate security access. The tool of the visualization itself was also developed and fine-tuned. In the second phase of the project we took on the difficult task of searching and analyzing the target data set for common causes between anomalies. In the final part of the second phase we have learned more about how much of the analysis work will be the job of the Data Mining team, how to perform that work, and how that work may be used by different customers in different ways. In this paper I detail how our perspective has changed after gaining more insight into how the customers wish to interact with the output and how that has changed the product.

iss↗
DOE OSTI2022

Interactive Exploration of High-Dimensional Phase Diagrams

High-dimensional thermodynamic phase stability databases are becoming increasingly common due to the convergence of three recent trends: (i) the widespread interest in so-called “high-entropy” alloys, (ii) the availability of high-throughput computational assessments of phase stability in broad composition spaces and (iii) the ongoing development of ever-increasingly broad, multicomponent, multiphase CALPHAD databases. Although automated computational tools can readily process such high-dimensional data, scientists are often unable to visualize the relevant phase relations, an ability that is crucial to gaining an intuitive understanding of the stability constraints governing materials design. The present work addresses this need by providing algorithms that enable the interactive exploration of phase equilibria in high-dimensional spaces. These algorithms concentrate the complex nonlinear nonsmooth optimization needed into a preprocessing step that generates a large number of high-dimensional yet elementary graphical primitives. Furthermore, these primitives can then be cross-sectioned to yield 3-dimensional views in a computationally efficient manner that enables an interactive exploration of high-dimensional spaces. All of these operations are highly parallelizable, thus facilitating scaling of this method to large data sets.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Machine Learning Approaches to Predicting Induced Seismicity and Imaging Geothermal Reservoir Properties

This project developed machine learning (ML) methods, lab data sets, and field data to advance geothermal exploration and geothermal energy production. The work had three focus areas. One involved the development of ML methods to use microearthquakes (MEQs) for imaging geothermal reservoir properties and improving subsurface characterization – most importantly the evolution of permeability within the evolving reservoir. This part of the work included development of ML approaches for automated MEQ location, focal mechanism determination and identification of earthquake precursors. The second area focused on using MEQ signals generated by geothermal exploration and production to predict the relationship between fluid injection and seismicity. Here, we extended to reservoir scale our success in using ML to predict laboratory earthquakes and fault zone stress state. The third focus area was on lab experiments. Here, we developed new ML models for lab earthquake prediction and identification of precursors to failure to improve earthquake forecasting and early warning in geothermal settings. Major outcomes of our work include ML models that learn from MEQ signals during geothermal exploration and production to predict induced seismicity. MEQs occur naturally in connection with drilling and energy production. We developed ML methods to use the seismic waves from these events to characterize the elastic, hydraulic and poromechanical properties of reservoirs. Our work illuminated fracture geometry and the evolution of fracture permeability by incorporating seismic coda wave analysis and ML methods to relate fluid injection and seismicity. We significantly expanded laboratory earthquake prediction to include methods that use both passive measurements of microearthquakes within the lab fault zones and also active source acoustic measurements of fault zone elastic properties. These methods can now predict fault zone stress state, time to failure and the magnitude of lab earthquakes. Our work showed that repetitive stick- slip failure events during frictional sliding (the lab equivalent of earthquakes) are preceded by a cascade of micro-failure events that radiate energy in a manner that foretells unstable failure – manifest as laboratory MEQs. We documented a mapping between fracture properties and statistical attributes of elastic radiation. We extended existing works to geothermal reservoir scale and developed ML methods to determine reservoir permeability, fracture properties, and their evolution during geothermal energy production. An attractive feature of ML algorithms is their ability to handle big datasets and reveal patterns and correlations that may remain invisible to conventional analyses. Our work connected data from field, laboratory and intermediate scales to study permeability, stress, strength, fracture stiffness and geometry. At the field scale we used data from the Newberry Volcano field site, UtahFORGE, EGS Collab, and also the Bedretto underground research lab in Switzerland. These data sets are bridging the gap between the lab scale, theory, and reservoir scale. Our work produced plain language summaries to improve public understanding of DOE research. We also developed openly distributed ML and seismicity datasets for use by all researchers and we published connections between induced seismicity in geothermal areas and reservoir properties including permeability, fracture properties, and stress state. Our models are designed for the large data sets of induced seismicity typically associated with geothermal sites. We produced labeled event catalogs and used them on geothermal data to assess how ML can facilitate geothermal production and exploration. All datasets are available on the GDR Productivity: The project produced 32 publications in peer reviewed journals (two are in review). It supported the work of 6 PhD students, 40 conference presentations, 6 keynote talks at national meetings, and mentoring and professional development for 4 postdoctoral fellows.

15 GEOTHERMAL ENERGY↗
NASA NTRS1980

High-latitude ion convection in the nighttime F region

A large data base of Atmosphere Explorer C measurements of the bulk ion velocity vector in the high-latitude F region has been examined. The concept of shear and rotational flow reversals is expanded upon to determine the extent to which the boundary between antisunward and sunward convection (i.e., a polar cap boundary) is an electrostatic equipotential. A most important factor in characterizing the nature of the large-scale geometrical configuration of the nighttime F region convection is the location and extent of quasi-equipotential regions of the convection reversal boundary.

Heelis, R. A.↗
NASA NTRS1996

Hierarchical and Parallelizable Direct Volume Rendering for Irregular and Multiple Grids

A general volume rendering technique is described that efficiently produces images of excellent quality from data defined over irregular grids having a wide variety of formats. Rendering is done in software, eliminating the need for special graphics hardware, as well as any artifacts associated with graphics hardware. Images of volumes with about one million cells can be produced in one to several minutes on a workstation with a 150 MHz processor. A significant advantage of this method for applications such as computational fluid dynamics is that it can process multiple intersecting grids. Such grids present problems for most current volume rendering techniques. Also, the wide range of cell sizes (by a factor of 10,000 or more), which is typical of such applications, does not present difficulties, as it does for many techniques. A spatial hierarchical organization makes it possible to access data from a restricted region efficiently. The tree has greater depth in regions of greater detail, determined by the number of cells in the region. It also makes it possible to render useful 'preview' images very quickly (about one second for one-million-cell grids) by displaying each region associated with a tree node as one cell. Previews show enough detail to navigate effectively in very large data sets. The algorithmic techniques include use of a kappa-d tree, with prefix-order partitioning of triangles, to reduce the number of primitives that must be processed for one rendering, coarse-grain parallelism for a shared-memory MIMD architecture, a new perspective transformation that achieves greater numerical accuracy, and a scanline algorithm with depth sorting and a new clipping technique.

Wilhelms, Jane↗
NASA NTRS1993

Lost in space: Onboard star identification using CCD star tracker data without an a priori attitude

There are many algorithms in use today which determine spacecraft attitude by identifying stars in the field of view of a star tracker. Some methods, which date from the early 1960's, compare the angular separation between observed stars with a small catalog. In the last 10 years, several methods have been developed which speed up the process and reduce the amount of memory needed, a key element to onboard attitude determination. However, each of these methods require some a priori knowledge of the spacecraft attitude. Although the Sun and magnetic field generally provide the necessary coarse attitude information, there are occasions when a spacecraft could get lost when it is not prudent to wait for sunlight. Also, the possibility of efficient attitude determination using only the highly accurate CCD star tracker could lead to fully autonomous spacecraft attitude determination. The need for redundant coarse sensors could thus be eliminated at substantial cost reduction. Some groups have extended their algorithms to implement a computation intense full sky scan. Some require large data bases. Both storage and speed are concerns for autonomous onboard systems. Neural network technology is even being explored by some as a possible solution, but because of the limited number of patterns that can be stored and large overhead, nothing concrete has resulted from these efforts. This paper presents an algorithm which, by descretizing the sky and filtering by visual magnitude of the brightness observed star, speeds up the lost in space star identification process while reducing the amount of necessary onboard computer storage compared to existing techniques.

Ketchum, Eleanor A.↗
DOE OSTI2025

First Observation of Neutrino Candidates in ProtoDUNE

The Neutrino Platform at CERN hosts two massive prototypes for the DUNE Far Detector to test the technology for the coming Horizontal Drift and Vertical Drift designs, almost perfectly on axis with the direction of the SPS beamline pointed towards the North Area. The 400 GeV/c SPS protons impacting on the T2 target area may produce exotic particles that could travel ~700 meters to the detectors, in addition to a substantial flux of neutrinos. Simulations of the neutrino flux demonstrate that thousands of neutrino interactions are expected per week in the active volume of NP04 spanning energies from a few GeV up to 180 GeV, constituting the main background for such search. The first step towards establishing a BSM physics program at the Neutrino Platform is therefore to observe neutrinos originating from the SPS beam. A sample of high-energy neutrino events in a DUNE FD-like LArTPC may also be of broader use to the DUNE collaboration. For example, by testing the performance of reconstruction algorithms on highly-energetic neutrino interactions with large hadronic showers. An initial neutrino search was performed by developing filters to remove cosmic events. After filtering, the remaining events were eye-scanned to identify neutrino candidates amongst the residual cosmic background. Over two of the available NP04 runs, 29 neutrino candidates were identified. A separate run taken with the SPS beam off yielded no neutrino candidates. An analysis is now under development that will use all available data, large Monte Carlo samples and full event reconstruction, with the aim of confirming whether NP04 is capable of observing feebly interacting particles originating from the SPS beam.

Pullia, Dario [CERN]↗
NASA NTRS2005

Multi-reference evaluation of uncertainty in earth orientation parameter measurements

Uncertainties in polar motion and length-of-day measurements are evaluated empirically using several data series from the space-geodetic techniques of the global positioning system (GPS), satellite laser ranging (SLR), and very long baseline interferometry (VLBI) during 1997-2002. In the evaluation procedure employed here, known as the three-corner hat (TCH) technique, the signal common to each series is eliminated by forming pair-wise differences between the series, thus requiring no assumed values for the 'truth' signal. From the variances of the differenced series, the uncertainty of each series can be recovered when reasonable assumptions are made about the correlations between the series. In order to form the pair-wise differences, the series data must be given at the same epoch. All measurement data sets studied here were sampled at noon (UTC); except for the VLBI series, whose data are interpolated to noon and whose UT1 values are also numerically differentiated to obtain LOD. The numerical error introduced to the VLBI values by the interpolation and differentiation is shown to be comparable in magnitude to the values determined by the TCH method for the uncertainties of the VLBI series. The TCH estimates for the VLBI series are corrupted by such numerical errors mostly as a result of the relatively large data intervals. Of the remaining data sets studied here, it is found that the IGS Final combined series has the smallest polar motion and length-of-day uncertainties.

Combination↗
DOE OSTI2025

Brillouin Sensing with PCA, and PCA-Based Neural Networks for Efficient Temperature Monitoring

This work explores peak estimation techniques in Brillouin Optical Time Domain Analysis (BOTDA), emphasizing both accuracy and efficiency. Euclidean distance measurement method is applied to principal components derived from Brillouin Gain Spectrum data. It offers a major speed advantage being 180 170 times faster than traditional curve fitting methods such as Lorentzian curve fitting, while maintaining similar accuracy. Additionally, a PCA- based neural network model shows significant reduction of peak estimation time compared to Lorentzian fitting. Results show Brillouin frequency shift errors lie under 0.75 MHz in both Euclidean distance-based and neural network-based methods, both of which utilize PCA components. For large data sets and long length fibers, PCA- assisted neural network for peak estimation would be an efficient solution.

Distributed optical fiber sensing↗
NASA NTRS2009

Characterizing Space Environments with Long-Term Space Plasma Archive Resources

A significant scientific benefit of establishing and maintaining long-term space plasma data archives is the ready access the archives afford to resources required for characterizing spacecraft design environments. Space systems must be capable of operating in the mean environments driven by climatology as well as the extremes that occur during individual space weather events. Long- term time series are necessary to obtain quantitative information on environment variability and extremes that characterize the mean and worst case environments that may be encountered during a mission. In addition, analysis of large data sets are important to scientific studies of flux limiting processes that provide a basis for establishing upper limits to environment specifications used in radiation or charging analyses. We present applications using data from existing archives and highlight their contributions to space environment models developed at Marshall Space Flight Center including the Chandra Radiation Model, ionospheric plasma variability models, and plasma models of the L2 space environment.

Minow, Joseph I.↗
NASA NTRSDate not supplied

The Ejectable Data Recorder: A Lean, Risk-Informed Approach for Hardware Development

NASA is developing the Orion spacecraft to transport crew from the Earth to the Moon as part of the Artemis series of missions. To provide a crew escape capability from pre-launch through ascent, the Orion vehicle is equipped with a Launch Abort System (LAS), built by Lockheed Martin, which pulls the capsule away from the launch vehicle in the event of an abort scenario. The Ascent Abort 2 (AA-2) test flight occurred on July 2, 2019,and tested a production version of the LAS to ensure that it can operate as intended, and to collect a large data set from hundreds of sensors on the vehicle to support Orion flight certification. In the original AA-2 architecture, a single-string set of communications antennas on the LAS would downlink all of the in-flight test data to ground stations. However, that communications architecture was predicted to have data dropouts during abort and jettison of the LAS, and would not support data transmission at all after LAS jettison. As a result, a comprehensive trade study was completed, yielding the addition of antennas on the crew module (CM), a buffer/rebroadcast capability for key portions of the flight, and an ejectable data recorder (EDR) subsystem. This EDR subsystem would serve as a backup to the radio frequency (RF) communications system, and would be non-flight critical, providing a unique capability that enabled management to take a different approach with the hardware and software development. The Crew Module and Separation Ring were developed as “Class 1”Flight Hardware, albeit with some tailoring approaches to enable efficiencies. The Class 1 designation requires full rigor for flight hardware and software, documenting everything that happens to a piece of hardware from procurement through disposal, requiring a full spectrum of acceptance tests, and the highest rigor of quality assurance processes. At the other end of the spectrum, Class 3hardware is controlled, but not intended for flight, and leaves the level of rigor up to the project manager. This classification is often used for research and development projects. Similarly,Class-1E has been recently defined at NASA for ISS payloads and technology development projects that are not flight critical and do not need the full rigor of Class 1 to be successful. The EDR subsystem was challenged at commencement to adopt a skunkworks and agile-like approach to hardware development, allowing for a different risk posture than the rest of the AA-2 hardware. After initially pursuing Class 1 processes, the EDR subsystem design evolved to incorporating numerous commercial components, leading to re-designation as a Class-1E subsystem. The resulting EDR subsystem was fully successful in meeting all flight system requirements, and achieved 100% retrieval of flight test data. This paper will discuss the risk posture of the EDR subsystem and the subsequent tailoring that was enacted as part of its Class-1E status.

EDR↗
DOE OSTI2021

On the Use of Smart Meter Data to Estimate the Voltage Magnitude on the Primary Side of Distribution Service Transformers

This paper develops a novel method to estimate the voltage magnitude on the primary side of distribution service transformers. The proposed method relies exclusively on smart meters, and therefore it is fully data-driven. This is an important feature because electric utilities have detailed models of only the primary network - that is, the network between the distribution substation and the primary side of service transformers that are installed closer to end-customer sites. The network that connects the secondary side of service transformers to end-customer sites, referred to as the secondary network, is simply represented by a lumped load. For each secondary network, the proposed method uses data acquired from only 2 smart meters: the closest and the farthest-in the sense of electrical distance - from the service transformer. As a reference to this feature, the proposed method is named SM2Vp. To our knowledge, this is the first time a method is shown to provide actionable information for realtime operation and control of power distribution grids using only two smart meters per secondary network. This is important because utilities have experienced barriers in managing and using large data sets for real-time operation and control. SM2Vp is primarily intended to provide pseudo-measurements for distribution system state estimation, but it can also be used directly for voltage control schemes. The performance of SM2Vp is demonstrated by numerical simulations carried out on three secondary network synthetic models and by using field data provided by a utility partner serving customers in southwestern California. A maximum relative error of approximately 3.9% or less is observed for the primary voltage magnitude estimates in all numerical experiments.

distribution service transformer↗
DOE OSTI2021

On the Use of Smart Meter Data to Estimate the Voltage Magnitude on the Primary Side of Distribution Service Transformers: Preprint

This paper develops a novel method to estimate the voltage magnitude on the primary side of distribution service transformers. The proposed method relies exclusively on smart meters, and therefore it is fully data-driven. This is an important feature because electric utilities have detailed models of only the primary network--that is, the network between the distribution substation and the primary side of service transformers that are installed closer to end-customer sites. The network that connects the secondary side of service transformers to end-customer sites, referred to as the secondary network, is simply represented by a lumped load. For each secondary network, the proposed method uses data acquired from only 2 smart meters: the closest and the farthest--in the sense of electrical distance--from the service transformer. As a reference to this feature, the proposed method is named SM2Vp. To our knowledge, this is the first time a method is shown to provide actionable information for real-time operation and control of power distribution grids using only two smart meters per secondary network. This is important because utilities have experienced barriers in managing and using large data sets for real-time operation and control. SM2Vp is primarily intended to provide pseudo-measurements for distribution system state estimation, but it can also be used directly for voltage control schemes. The performance of SM2Vp is demonstrated by numerical simulations carried out on three secondary network synthetic models and by using field data provided by a utility partner serving customers in southwestern California. A maximum relative error of approximately 3.9% or less is observed for the primary voltage magnitude estimates in all numerical experiments.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2023

Piecewise linear approximation with minimum number of linear segments and minimum error: A fast approach to tighten and warm start the hierarchical mixed integer formulation

In several areas of economics and engineering, it is often necessary to fit discrete data points or approximate nonlinear functions with continuous functions. Piecewise linear (PWL) functions are a convenient way to achieve this. PWL functions can be modeled in mathematical problems using only linear and integer variables. Moreover, there is a computational benefit in using PWL functions that have the least possible number of segments. This work proposes a novel hierarchical mixed integer linear programming (MILP) formulation that identifies a continuous PWL approximation with minimum number of linear segments for a given target maximum error. The proposed MILP formulation also identifies the solution with the least maximum error among the solutions with minimum number of segments. Then, this work proposes a fast iterative algorithm that identifies non necessarily continuous PWL approximations by solving O(S log N) linear programming (LP) problems, where N is the number of data points and S is the minimum number of segments in the non necessarily continuous case. This work demonstrates that tight bounds for the MILP problem can be derived from these approximations. Next, a fast algorithm is introduced to transform a non necessarily continuous PWL approximation into a continuous one. Finally, the tight bounds and the continuous PWL approximations are used to tighten and warm start the MILP problem. The tightened formulation is shown in experimental results to be more efficient, especially for large data sets, with a solution time that is up to two orders of magnitude less than the existing literature.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Detailed study of quark-hadron duality in spin structure functions of the proton and neutron

Background: The response of hadrons, the bound states of the strong force (QCD), to external probes can be described in two different, complementary frameworks: as direct interactions with their fundamental constituents, quarks and gluons, or alternatively as elastic or inelastic coherent scattering that leaves the hadrons in their ground state or in one of their excited (resonance) states. The former picture emerges most clearly in hard processes with high momentum transfer, where the hadron response can be described by the perturbative expansion of QCD, while at lower energy and momentum transfers, the resonant excitations of the hadrons dominate the cross section. The overlap region between these two pictures, where both yield similar predictions, is referred to as quark-hadron duality and has been extensively studied in reactions involving unpolarized hadrons. Some limited information on this phenomenon also exists for polarized protons, deuterons, and 3 He nuclei, but not yet for neutrons. Purpose: In this paper, we present comprehensive and detailed results on the correspondence between the extrapolated deep inelastic structure function g 1 of both the proton and the neutron with the same quantity measured in the nucleon resonance region. Thanks to the fine binning and high precision of our data, and using a well-controlled perturbative QCD (pQCD) fit for the partonic prediction, we can make quantitative statements about the kinematic range of applicability of both local duality and global duality. Method: We use the most updated QCD global analysis results at high x from the Jefferson Lab Angular Momentum Collaboration to extrapolate the spin structure function g 1 into the nucleon resonance region and then integrate over various intervals in the scaling variable x. We compare the results with the large data set collected in the quark-hadron transition region by the CLAS Collaboration, including, for the first time, deconvoluted neutron data, integrated over the same intervals. We present this comparison as a function of the momentum transfer Q 2 . Results: We find that, depending on the integration interval and the minimum momentum transfer chosen, a clear transition to quark-hadron duality can be observed in both nucleon species. Furthermore, we show, for the first time, the approach to scaling behavior for g 1 measured in the resonance region at sufficiently high momentum transfer. Conclusions: Here, our results can be used to quantify the deviations from the applicability of pQCD for data taken at moderate energies and can help with extraction of quark distribution functions from such data.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗