Search NASA⌕ Search

SEARCH · Search NASA

Results for “Big data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

End-System Aware Peak Link Utilization Transport for Big Data Transfer over Dedicated Connections

High-performance networks provisioning 10/40-Gbps dedicated channels have been rapidly developed andincreasingly deployed to support big data transfers on the order of terabytes or petabytes in large-scale network-intensiveapplications. However, end users have not seen corresponding increase in application throughput mainly because traditional end-to-end transport methods are not optimized for such connections. New congestion or flow control mechanisms are desirable to meet the challenges brought to transport protocol design by dedicated connections. The advent and proliferation of multi-core processors make it now possible to improve application throughput by providing multiple processing and network resources to a single data transfer. Based on the existing PLUT method, we proposed a new dynamic buffer management scheme and an automatic parallelism tuning mechanism to achieve the maximal bottleneck rate. The performance superiority of PLUT is justified by the experimental results collected over a local 10Gbps dedicated connection.

Rao, Nageswara↗

The big data challenges of connectomics

The structure of the nervous system is extraordinarily complicated because individual neurons are interconnected to hundreds or even thousands of other cells in networks that can extend over large volumes. Mapping such networks at the level of synaptic connections, a field called connectomics, began in the 1970s with a the study of the small nervous system of a worm and has recently garnered general interest thanks to technical and computational advances that automate the collection of electron-microscopy data and offer the possibility of mapping even large mammalian brains. However, modern connectomics produces ‘big data’, unprecedented quantities of digital information at unprecedented rates, and will require, as with genomics at the time, breakthrough algorithmic and computational solutions. Here in this paper we describe some of the key difficulties that may arise and provide suggestions for managing them.

60 APPLIED LIFE SCIENCES↗

Big Data Analytics for Demand Response: Clustering Over Space and Time

The pervasive deployment of advanced sensing infrastructure in Cyber-Physical systems, such as the Smart Grid, has resulted in an unprecedented data explosion. Such data exhibit both large volumes and high velocity characteristics, two of the three pillars of Big Data, and have a time-series notion as datasets in this context typically consist of successive measurements made over a time interval. Time-series data can be valuable for data mining and analytics tasks such as identifying the “right” customers among a diverse population, to target for Demand Response programs. However, time series are challenging to mine due to their high dimensionality. In this paper, we motivate this problem using a real application from the smart grid domain. We explore novel representations of time-series data for BigData analytics, and propose a clustering technique for determining natural segmentation of customers and identification of temporal consumption patterns. Our method is generizable to large-scale, real-world scenarios, without making any assumptions about the data. We evaluate our technique using real datasets from smart meters, totaling ~ 18,200,000 data points, and show the efficacy of our technique in efficiency detecting the number of optimal number of clusters.

97 MATHEMATICS AND COMPUTING↗

Time resolved surface photovoltage measurements using a big data capture approach to KPFM

Optoelectronic behavior in materials such as organic/inorganic hybrid perovskites depend on a complex interplay between fast (electronic) and slower (ionic) processes. These processes are thought to be influenced by structural inhomogeneities (e.g. interfaces and grain boundaries) bringing forward the necessity for development of techniques capable of correlating nanostructure and photo-transport behavior. While Kelvin probe force microscopy (KPFM) is ideally suited to map surface potentials on relevant length scales, it lacks sufficient temporal resolution to extract the meaningful system dynamics. Here, we develop a time resolved surface photovoltage (SPV) measurement based on full information capture of the photodetector stream during open loop KPFM operation. G-Mode, or G-KPFM allows quantification of SPV with microsecond temporal and nanoscale spatial resolution. Using this technique, we observe concurrent spatial and fast temporal variations in the SPV generated across a methylammonium lead bromide (MAPbBr3) thin film, a possible indicator relating microstructure with heterogenous photo-transport behavior. Here, we further demonstrate the advantage of adopting big data analytics including unsupervised clustering methods to quickly discern spatial variability in the information rich SPV dataset. Beyond G-KPFM, such clustering methods will be useful for interpretation of the multidimensional datasets arising from the growing number of time resolved KPFM approaches now available.

36 MATERIALS SCIENCE↗

Big-Data RHEED analysis for understanding epitaxial film growth processes

Reflection high energy electron diffraction (RHEED) has by now become a standard tool for in-situ monitoring of film growth by pulsed laser deposition and molecular beam epitaxy. Yet despite the widespread adoption and wealth of information in RHEED image, most applications are limited to observing intensity oscillations of the specular spot, and much additional information on growth is discarded. With ease of data acquisition and increased computation speeds, statistical methods to rapidly mine the dataset are now feasible. Here, we develop such an approach to the analysis of the fundamental growth processes through multivariate statistical analysis of RHEED image sequence. This approach is illustrated for growth of LaxCa1-xMnO3 films grown on etched (001) SrTiO3 substrates, but is universal. The multivariate methods including principal component analysis and k-means clustering provide insight into the relevant behaviors, the timing and nature of a disordered to ordered growth change, and highlight statistically significant patterns. Fourier analysis yields the harmonic components of the signal and allows separation of the relevant components and baselines, isolating the assymetric nature of the step density function and the transmission spots from the imperfect layer-by-layer (LBL) growth. These studies show the promise of big data approaches to obtaining more insight into film properties during and after epitaxial film growth. Furthermore, these studies open the pathway to use forward prediction methods to potentially allow significantly more control over growth process and hence final film quality.

Big-Data↗

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer low performance when subject to highly concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids, or discrete event generators. In this paper we tackle this issue. We present SLoG, shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG’s design on up to 100,000 cores of the Theta supercomputer: the results show high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.

big data↗

Data Driven Smart Proxy for CFD: Application of Big Data Analytics & Machine Learning in Computational Fluid Dynamics (Part 1)

Simulation technologies can reduce the time and cost of the development and deployment of advanced technologies and allow rapid scale-up of these technologies for fossil fuel based energy systems. However, to ensure their usefulness in practice, the credibility of the simulations needs to be established with Uncertainty Quantification (UQ) methods. National Energy Technology Laboratory (NETL) has been applying non-intrusive UQ methodologies to categorize and quantify uncertainties in CFD simulations of gas-solid multiphase flows. To reduce the computational cost associated with gas-solid flow simulations required for UQ analysis, techniques commonly used in the area of Artificial Intelligence (AI) and Data Mining (DM) are used to construct smart proxy models, which can reduce the computational cost of conducting large number of multiphase CFD simulations.

97 MATHEMATICS AND COMPUTING↗

Data Driven Smart Proxy for CFD Application of Big Data Analytics & Machine Learning in Computational Fluid Dynamics, Report Two: Model Building at the Cell Level

To ensure the usefulness of simulation technologies in practice, their credibility needs to be established with Uncertainty Quantification (UQ) methods. In this project, smart proxy is introduced to significantly reduce the computational cost of conducting large number of multiphase CFD simulations, which is typically required for non-intrusive UQ analysis. Smart proxy for CFD models are developed using pattern recognition capabilities of Artificial Intelligence (AI) and Data Mining (DM) technologies. Several CFD simulation runs with different inlet air velocities for a rectangular fluidized bed are used to create a smart CFD proxy that is capable of replicating the CFD results for the entire geometry and inlet velocity range. The smart CFD proxy is validated with blind CFD runs (CFD runs that have not played any role during the development of the smart CFD proxy). The developed and validated smart CFD proxy generates its results in seconds with reasonable error (less than 10%). Upon completion of this project, UQ studies that rely on hundreds or thousands of smart CFD proxy runs can be accomplished in minutes. Following figure demonstrates a validation example (blind CFD run) showing the results from the MFiX simulation and the smart CFD proxy for pressure distribution across a fluidized bed at a given time-step (the layer number corresponds to the vertical location in the bed).

97 MATHEMATICS AND COMPUTING↗

Data Driven Smart Proxy for CFD Application of Big Data Analytics & Machine Learning in Computational Fluid Dynamics, Part Three: Model Building at the Layer Level

To ensure the usefulness of simulation technologies in practice, their credibility needs to be established with Uncertainty Quantification (UQ) methods. In this project smart proxy is introduced to significantly reduce the computational cost of conducting large number of multiphase CFD simulations. Smart proxy for CFD models are developed using pattern recognition capabilities of Artificial Intelligence (AI), Machine Learning (LM) and Data Mining (DM) technologies. Several CFD simulation runs with different inlet air velocities for a rectangular fluidized bed are used to create a smart CFD proxy that is capable of replicating the CFD results for a wide variety of inlet velocities. The smart CFD proxy is validated with blind CFD runs (CFD runs that have not played any role during the development [training, calibration and validation] of the smart CFD proxy). In our earlier work, CFD data was used to train ANN at the cell level. That is an ANN was trained for each computational cell used in the CFD simulations. In the present work, the ANN is constructed and trained based on cross sectional area average of each variable, such as pressure, velocities and volume fraction (Layer Level). This leads to improvements in the training time at the expense of less spatial resolution. The resulting trained ANN provides spatially average value of parameters of interest, along the length of the fluidized bed. Upon completion of this project, UQ studies that rely on hundreds or thousands of smart CFD proxy runs can be accomplished in minutes. Following figure demonstrates a validation example (blind CFD run) showing the results from the MFiX and the smart CFD proxy for pressure drop across a fluidized bed at time-step of 1400 (the layer number corresponds to the vertical location in the bed).

97 MATHEMATICS AND COMPUTING↗

Distribution system model calibration with big data from AMI and PV inverters

Efficient management and coordination of distributed energy resources with advanced automation schemes requires accurate distribution system modeling and monitoring. Big data from smart meters and photovoltaic (PV) micro-inverters can be leveraged to calibrate existing utility models. This paper presents computationally efficient distribution system parameter estimation algorithms to improve the accuracy of existing utility feeder radial secondary circuit model parameters. The method is demonstrated using a real utility feeder model with advanced metering infrastructure (AMI) and PV micro-inverters, along with alternative parameter estimation approaches that can be used to improve secondary circuit models when limited measurement data is available. Lastly, the parameter estimation accuracy is demonstrated for both a three-phase test circuit with typical secondary circuit topologies and single-phase secondary circuits in a real mixed-phase test system.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Big Data Challenges at CCMC

Like other research centers, the Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) at NASA Goddard Space Flight Center (GSFC) is also experiencing the big data challenges. CCMC hosts over 80 space weather models for Runs On Request (ROR), Continuous Runs and Instant Runs simulation services for the research community. In addition, CCMC has started to support simulation output onboarding in response to the Open Science initiative. Overall, we have accumulated over petabytes of simulation output data and are rapidly growing. In this presentation, we will discuss our data and storage challenges. We will present our attempts to address our challenges and any associated lessons learned. CCMC uses Apache Airflow to ensure data transfer is consistent. We will give a brief overview on how we leverage Apache Airflow to enhance our environment.

space weather↗

Using NVMe Gen3 PCIe SSD Cards in High-density Servers for High-performance Big Data Transfer Over Multiple Network Channels

This Technical Note describes how the Zettar team came up with a data transfer cluster design that convincingly proved the feasibility of using high-density servers for high-performance Big Data transfers. It then outlines the tests, operations, and observations that address a potential over-heating concern regarding the use of Non-Volatile Memory Host Controller Interface Specification (NVMHCI aka NVM Express or NVMe) Gen 3 PCIe SSD cards in high-density servers. Finally, it points out the possibility of developing a new generation of high-performance Science DMZ data transfer system for the data-intensive research community and commercial enterprises.

97 MATHEMATICS AND COMPUTING↗

Big Data Challenges for Large Radio Arrays

Future large radio astronomy arrays, particularly the Square Kilometre Array (SKA), will be able to generate data at rates far higher than can be analyzed or stored affordably with current practices. This is, by definition, a "big data" problem, and requires an end-to-end solution if future radio arrays are to reach their full scientific potential. Similar data processing, transport, storage, and management challenges face next-generation facilities in many other fields.

Combining↗

Promising Innovations for the Lighting Profession: Big Data, New Tools

The use of parametric modeling is increasing as computing power increases, supporting the use of generative design software to design shoes and plan architectural spaces. This big data inclusive design process is used for daylighting studies, and is ripe for development by the larger lighting community.

Lighting, generative design, virtual reality↗

A Computing Environment to Support Repeatable Scientific Big Data Experimentation of World-Wide Scientific Literature

A principal tenant of the scientific method is that experiments must be repeatable and relies on ceteris paribus (i.e., all other things being equal). As a scientific community, involved in data sciences, we must investigate ways to establish an environment where experiments can be repeated. We can no longer allude to where the data comes from, we must add rigor to the data collection and management process from which our analysis is conducted. This paper describes a computing environment to support repeatable scientific big data experimentation of world-wide scientific literature, and recommends a system that is housed at the Oak Ridge National Laboratory in order to provide value to investigators from government agencies, academic institutions, and industry entities. The described computing environment also adheres to the recently instituted digital data management plan mandated by multiple US government agencies, which involves all stages of the digital data life cycle including capture, analysis, sharing, and preservation. It particularly focuses on the sharing and preservation of digital research data. The details of this computing environment are explained within the context of cloud services by the three layer classification of Software as a Service , Platform as a Service , and Infrastructure as a Service .

"Software as a Service↗

Big Data Challenges at the Community Coordinated Modeling Center (CCMC)

Like other research centers, the Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) at NASA Goddard Space Flight Center (GSFC) is also experiencing the big data challenges. CCMC hosts over 80 space weather models for Runs On Request (ROR), Continuous Runs and Instant Runs simulation services for the research community. In addition, CCMC has started to support simulation output onboarding in response to the Open Science initiative. Overall, we have accumulated over petabytes of simulation output data and are rapidly growing. In this presentation, we will discuss our data and storage challenges. We will present our attempts to address our challenges and any associated lessons learned. CCMC uses Apache Airflow to ensure data transfer is consistent. We will give a brief overview on how we leverage Apache Airflow to enhance our environment.

space weather↗