Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data visualization and analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26

High Performance Descriptive Semantic Analysis of Semantic Graph Databases

As semantic graph database technology grows to address components ranging from extant large triple stores to SPARQL endpoints over SQL-structured relational databases, it will become increasingly important to be able to understand their inherent semantic structure, whether codified in explicit ontologies or not. Our group is researching novel methods for what we call descriptive semantic analysis of RDF triplestores, to serve purposes of analysis, interpretation, visualization, and optimization. But data size and computational complexity makes it increasingly necessary to bring high performance computational resources to bear on this task. Our research group built a novel high performance hybrid system comprising computational capability for semantic graph database processing utilizing the large multi-threaded architecture of the Cray XMT platform, conventional servers, and large data stores. In this paper we describe that architecture and our methods, and present the results of our analyses of basic properties, connected components, namespace interaction, and typed paths such for the Billion Triple Challenge 2010 dataset.

99 GENERAL AND MISCELLANEOUS↗

'Big Data' Collaboration: Exploring, Recording and Sharing Enterprise Knowledge

As data sources and data size proliferate, knowledge discovery from "Big Data" is starting to pose several challenges. In this paper, we address a specific challenge in the practice of enterprise knowledge management while extracting actionable nuggets from diverse data sources of seemingly-related information. In particular, we address the challenge of archiving knowledge gained through collaboration, dissemination and visualization as part of the data analysis, inference and decision-making lifecycle. We motivate the implementation of an enterprise data-discovery and knowledge recorder tool, called SEEKER based on real world case-study. We demonstrate SEEKER capturing schema and data-element relationships, tracking the data elements of value based on the queries and the analytical artifacts that are being created by analysts as they use the data. We show how the tool serves as digital record of institutional domain knowledge and a documentation for the evolution of data elements, queries and schemas over time. As a knowledge management service, a tool like SEEKER saves enterprise resources and time by avoiding analytic silos, expediting the process of multi-source data integration and intelligently documenting discoveries from fellow analysts.

Big Data↗

Big Data as a Service from an Urban Information System

Big Data has already proven itself as a valuable tool that lets geographers and urban researchers utilize large data re- sources to generate new insights. However, wider adoption of Big Data techniques in these areas is impeded by a number of difficulties in both knowledge discovery and data and sci- ence production. Typically users face such problems as dis- parate and scattered data, data management, spatial search- ing, insufficient computational capacity for data-driven anal- ysis and modelling, and the lack of tools to quickly visual- ize and summarize large data and analysis results. Here we propose an architecture for an Urban Information Sys- tem (UrbIS) that mitigates these problems by utilizing the Big Data as a Service (BDaaS) concept. With technolog- ical roots in High-performance Computing (HPC), BDaaS is based on the idea of outsourcing computations to dif- ferent computing paradigms, scalable to super-computers. UrbIS aims to incorporate federated metadata search, in- tegrated modeling and analysis, and geovisualization into a single seamless workflow. The system is under active devel- opment and is built around various emerging technologies that include hybrid and NoSQL databases, massively par- allel systems, GPGPU computing, and WebGL-based geo- graphic visualization. UrbIS is designed to facilitate using Big Data across multiple cities to better understand how ur- ban areas impact the environment and how climate change and other environmental change impact urban areas.

Sorokine, Alexandre↗

Method and apparatus for offloading compute resources to a flash co-processing appliance

Solid-State Drive (SSD) burst buffer nodes are interposed into a parallel supercomputing cluster to enable fast burst checkpoint of cluster memory to or from nearby interconnected solid-state storage with asynchronous migration between the burst buffer nodes and slower more distant disk storage. The SSD nodes also perform tasks offloaded from the compute nodes or associated with the checkpoint data. For example, the data for the next job is preloaded in the SSD node and very fast uploaded to the respective compute node just before the next job starts. During a job, the SSD nodes perform fast visualization and statistical analysis upon the checkpoint data. The SSD nodes can also perform data reduction and encryption of the checkpoint data.

97 MATHEMATICS AND COMPUTING↗

scraps: An open-source Python-based analysis package for analyzing and plotting superconducting resonator data

We present "scraps" (SuperConducting Analysis and Plotting Software), a Python package designed to aid in the analysis and visualization of large amounts of superconducting resonator data, specifically complex transmission as a function of frequency, acquired at many different temperatures and driving powers. The package includes a least-squares fitting engine as well as a Monte-Carlo Markov Chain sampler for sampling the posterior distribution given priors, marginalizing over nuisance parameters, and estimating covariances. A set of plotting tools for generating publication-quality figures is also provided in the package. Lastly, we discuss the functionality of the software and provide some examples of its utility on data collected from a niobium-nitride coplanar waveguide resonator fabricated at Argonne National Laboratory.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

Energy Finance Data Warehouse Manual

The Office of Energy Policy and Systems Analysis s finance team (EPSA-50) requires a suite of automated applications that can extract specific data from a flexible data warehouse (where datasets characterizing energy-related finance, economics and markets are maintained and integrated), perform relevant operations and creatively visualize them to provide a better understanding of what policy options affect various operators/sectors of the electricity system. In addition, the underlying data warehouse should be structured in the most effective and efficient way so that it can become increasingly valuable over time. This report describes the Energy Finance Data Warehouse (EFDW) framework that has been developed to accomplish the defined requirement above. We also specifically dive into the Sankey generator use-case scenario to explain the components of the EFDW framework and their roles. An excel-based data warehouse was used in the creation of the energy finance Sankey diagram and other detailed data finance visualizations to support energy policy analysis. The framework also captures the methodology, calculations and estimations analysts used for the calculation as well as relevant sources so newer analysts can build on work done previously.

97 MATHEMATICS AND COMPUTING↗

Large Scale Comparative Visualisation of Regulatory Networks with TRNDiff

The advent of Next Generation Sequencing (NGS) technologies has seen explosive growth in genomic datasets, and dense coverage of related organisms, supporting study of subtle, strain-specific variations as a determinant of function. Such data collections present fresh and complex challenges for bioinformatics, those of comparing models of complex relationships across hundreds and even thousands of sequences. Transcriptional Regulatory Network (TRN) structures document the influence of regulatory proteins called Transcription Factors (TFs) on associated Target Genes (TGs). TRNs are routinely inferred from model systems or iterative search, and analysis at these scales requires simultaneous displays of multiple networks well beyond those of existing network visualisation tools [1]. In this paper we describe TRNDiff, an open source system supporting the comparative analysis and visualization of TRNs (and similarly structured data) from many genomes, allowing rapid identification of functional variations within species. The approach is demonstrated through a small scale multiple TRN analysis of the Fur iron-uptake system of Yersinia, suggesting a number of candidate virulence factors; and through a larger study exploiting integration with the RegPrecise database (http://regprecise.lbl.gov; [2]) - a collection of hundreds of manually curated and predicted transcription factor regulons drawn from across the entire spectrum of prokaryotic organisms.

Bioinformatics, Visualisation, Transcription, Regu↗

Why wait? Let's start computing while data is still on the wire

In this era of Big Data, computing useful information from data is becoming increasingly complicated, particularly due to the ever increasing volumes of data that need to travel over the network to data centers to be stored and processed, all highly expensive operations in the long haul. In this paper we suggest that we can do computing and analysis of data "on the wire," i.e., while data is still in transit. The nature of these computations include analysis, visualization, pattern recognition, and prediction, or forecasting, on the streaming data. We follow a service function chaining architecture to implement this, assuming that the data packets arrive within a single network administrative domain. As a demonstration of this new computing paradigm, we present three examples. Firstly, we demonstrate pattern recognition and data visualization on streaming forex data, which can be used for lucrative trading in the forex market. In our second example, we analyze and learn user buying patterns from clickstream data streaming from multiple websites. Finally, we monitor solar sensors for a zero reading while the packets are still on their way to the data center, to schedule any maintenance and requisite repairs with no time delay.

97 MATHEMATICS AND COMPUTING↗

Towards elastic in situ analysis for high-performance computing simulations

In situ analysis and visualization have grown increasingly popular for enabling direct access to data from high-performance computing (HPC) simulations. As a simulation progresses and interesting physical phenomena emerge, however, the data produced may become increasingly complex, and users may need to dynamically change the type and scale of in situ analysis tasks being carried out and consequently adapt the amount of resources allocated to such tasks. To date, none of the production in situ analysis frameworks offer such an elasticity feature, and for good reason: the assumption that the number of processes could vary during run time would force developers to rethink software and algorithms at every level of the in situ analysis stack. In this paper we present Colza, a data staging service with elastic in situ visualization capabilities. We demonstrate the use of Colza with the Deep Water Impact and the AMR-Wind simulations, coupling them with the ParaView Catalyst and Ascent in situ libraries, and show that Colza enables dynamic rescaling of these widely-used frameworks with no interruption to the simulation or staging service. Here, we highlight the challenges of enabling such elasticity, which requires overcoming these frameworks' reliance on MPI, using distinct engineering approaches, namely dependency injection and dependency overload. To the best of our knowledge, this work is the first to enable elastic in situ visualization capabilities for HPC applications on top of existing production analysis tools.

97 MATHEMATICS AND COMPUTING↗

Supporting Co-Design of Extreme-Scale Systems with In Situ Visual Analysis of Event-Driven Simulations (Final Report)

Extreme-scale systems for compute- and data-centric science are pushing the boundaries of system architectures in order to achieve ambitious goals in terms of computing capability, resilience, and power efficiency. Future systems necessitate emerging designs that incorporate new technologies, system organizations, and computer science techniques, and these changes are not yet well understood. Discrete-event-driven simulation of computing system architectures and subsystems has emerged as a productive and cost-effective means to evaluating potential designs, along with capabilities for executing these simulations of extreme-scale systems. However, techniques for analyzing the behavior of these simulations have not kept pace with our ability to perform the simulations, preventing us from extracting the most value from this promising approach. The objective of this project is to support the codesign of extreme-scale system architectures for compute and data-centric science through research and development of novel methods for analysis and visualization of large-scale event-driven simulations.

97 MATHEMATICS AND COMPUTING↗

Earth System Grid Center for Enabling Technologies (ESG-CET): A Data Infrastructure for Data-Intensive Climate Research

For the Earth System Grid Federation (ESGF), the ESG-CET team has led international development and delivered a production environment for managing and accessing ultrascale climate data. This production environment includes multiple national and international climate projects (e.g., Couple Model Intercomparison Project, Community Earth System Model), ocean model data (such as the Parallel Ocean Program), observation data (Carbon Dioxide Information and Analysis Center, Atmospheric Infrared Sounder, and so forth), and analysis and visualization tools, all of which serve a diverse community of users. These data holdings and services are distributed across multiple ESG-CET sites (such as LANL, LBNL, LLNL, NCAR, and ORNL) as well as at unfunded partners sites such as the Australian National University National Computational Infrastructure, the British Atmospheric Data Centre, the National Oceanic and Atmospheric Administration Geophysical Fluid Dynamics Laboratory, the Max Planck Institute for Meteorology, the German Climate Computing Centre, and the National Aeronautics and Space Administration Jet Propulsion Laboratory. More recently, ESG-CET has been extending services beyond data-file access and delivery to develop more detailed information products (scientific graphics, animations, etc.), secure binary data-access services (based upon the OPeNDAP protocol), and server-side analysis capabilities. These will allow users to request data subsets transformed through commonly used analysis and intercomparison procedures. As we transition from development activities to production and operations, the ESG-CET team is tasked with making data available to all users seeking to understand, process, extract value from, visualize, and/or communicate it to others. This ongoing effort, though daunting in scope and complexity, will greatly magnify the value of numerical climate model outputs and climate observations for future national and international climate-assessment reports. Continued ESGF progress will result in a production ultrascale data system for empowering scientists who attempt new and exciting data exchanges that could ultimately lead to breakthrough climate-science discoveries.

54 ENVIRONMENTAL SCIENCES↗

Visual analysis of code security

To help increase the confidence that software is secure, researchers and vendors have developed different kinds of automated software security analysis tools. These tools analyze software for weaknesses and vulnerabilities, but the individual tools catch different vulnerabilities and produce voluminous data with many false positives. This paper describes a system that brings together the results of disparate software analysis tools into a visual environment to support the triage and exploration of code vulnerabilities. Our system allows software developers to explore vulnerability results to uncover hidden trends, triage the most important code weaknesses, and show who is responsible for introducing software vulnerabilities. By correlating and normalizing multiple software analysis tools' data, the overall vulnerability detection coverage of software is increased. A visual overview and powerful interaction allows the user to focus attention on the most pressing vulnerabilities within huge volumes of data, and streamlines the secure software development workflow through integration with development tools.

99 GENERAL AND MISCELLANEOUS↗

Performance Analysis of Large Scale HPC Workflows for Earth System Models

Climate assessment reports, such as the Intergovernmental Panel on Climate Change’s 5th assesment report [IPCC AR5; IPCC et al., 2014] or the United States National Climate Assessment [U.S. Global Change Research Program, 2014], increasingly rely on many-ensemble climate projections from fully-coupled Earth system models (ESM) to asses possible future climate states. ESM participate in a large set of Coupled Model Intercomparson Projects (CMIP) to better understand their models and the climate system, so that they may provide credible climate projections. However, fully participating in these intercomparison exercises represents an extremely large computational personnel burden for modeling centers [Eyring et al., 2016, Williams et al., 2016a]. However, there has been little attention within the climate modeling community paid to the in situ performance of ESMs up until now. Recently a computational performance model intercomparison project (CPMIP) for CMIP6 has begun [Balaji et al., 2017], but there has not yet been an effort to gather, process, and analyze the historical performance data that has been saved within archived log files from previous modeling efforts, nor the development of performance analysis system capable of providing detailed statistical analyses or visualizations of this performance data. Therefore, we’ve begun a joint effort to analyze the historical and current performance data on both GFDL and OLCF modeling systems. Here, we have developed a preliminary performance analysis system to captue, analyize, and visualize end-to-end in situ performance data and applied it to NOAA/GFDL’s post-processing system. Through the course of the project a number of performance improvements have been made, with the steps necessary to handle the expected volume of data (Big Data) likely to come out of the CMIP6 exercises (TB to PB) provided. We have also begun developing a statistical workload model that will allow modeling centers to determined their expected scientific throughput in the context of expected resource (computational and personnel) availability and accounting for typical workflow disruptions. This will allow them to develop a CMIP participation strategy and maximize their level of participation in the exercises. We expect this infrastructure, and the data it produces, to ultimately allow us to identify performance issues and workflow errors that exist across Earth system modeling centers, and possibly HPC modeling centers in general. Center-independent solutions to workflow or performance issues would help alleviate the computational burden for CMIP6+ modeling projects and should realize both labor and cost savings benefits throughout the CMIP community.

54 ENVIRONMENTAL SCIENCES↗

RCSB PDB Mobile : iOS and Android mobile apps to provide data access and visualization to the RCSB Protein Data Bank

The Research Collaboratory for Structural Bioinformatics Protein Data Bank (RCSB PDB) resource provides tools for query, analysis and visualization of the 3D structures in the PDB archive. As the mobile Web is starting to surpass desktop and laptop usage, scientists and educators are beginning to integrate mobile devices into their research and teaching. In response, we have developed the RCSB PDB Mobile app for the iOS and Android mobile platforms to enable fast and convenient access to RCSB PDB data and services. Lastly, using the app, users from the general public to expert researchers can quickly search and visualize biomolecules, and add personal annotations via the RCSB PDB's integrated MyPDB service.

59 BASIC BIOLOGICAL SCIENCES↗

Data planning and analysis for synthesis of multidimensional laws (nonlinear multifactor analysis)

The methodology of data planning and analysis for synthesis of multidimensional laws using visualization is described along with the ensuing method of numerical data approximation by functions with 'separable' variables. The method is developed for the cases of source data presented as (a) a table where all cells are filled, (b) an orthogonal table where quite certain cells are filled, and (c) a table where, generally speaking, arbitrary cells are not filled. The method was successfully applied for different problems of nuclear science and technology.

97 MATHEMATICS AND COMPUTING↗

In Situ Visualization and Production of Extract Databases

Simulations running at high concurrency on HPC systems generate large volumes of data that are impractical to write to disk due to time and storage constraints. Applications often adapt by saving data infrequently, resulting in datasets with poor temporal resolution. This can make datasets difficult to interpret during post hoc visualization and analysis, or worse, it can lead to lost science. In Situ visualization and analysis can enable efficient production of small data products such as rendered images or surface extracts that consist of polygonal geometry plus fields. These data products are far smaller than their source data and can be processed much more economically in a traditional post hoc workflow using far fewer computational resources. We used the SENSEI and Libsim in situ infrastructures to implement rendering workflow and surface data extraction workflows in the AVF-LESLIE combustion code. Finally, these workflows were then demonstrated at high levels of concurrency and showed significant data reductions and limited impact on the simulation runtime.

97 MATHEMATICS AND COMPUTING↗

A Science Gateway for the Repeatable Analysis of Machine Learning Predicted Gravity Anomalies

In recent years, deep learning has become an increasingly popular alternative for modeling in geoscience applications due to its scalability and efficiency. However, the interpretability, compute, data volume, and hyperparameter tuning requirements of deep learning models make development and monitoring difficult. Furthermore, model explainability and communicating results obtained by these models to users or domain experts is a challenge, as domain experts in geoscience also need to have a deep understanding of how those models function in order to support their scientific works. Here, we describe a science gateway and machine learning pipeline for predicting gravity anomalies from geophysical data. The gateway, built on open-source technologies, provides a holistic view of the pipeline through interactive visualizations aimed at enabling efficient exploratory data analysis. The repeatability, reproducibility, and monitoring capabilities of this overall system allow us to iterate and analyze at scale. Using this pipeline and gateway, we can repeatedly produce accurate high-resolution gravity anomaly datasets. By describing the underlying technologies, implementation, and results, here we provide a foundation for the broader adoption of science gateways into cross-cutting geoscience and machine learning research projects as a means to improve the scientific discovery and collaboration in the geophysics and computational sciences community.

58 GEOSCIENCES↗