Search NASA⌕ Search

SEARCH · Search NASA

Results for “big data visualization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

Big Data Framework with Machine Learning for D and D Applications - 19108

The nuclear industry is experiencing a steady increase in maintenance costs even though plants are maintained under high levels of safety, capability and reliability. Nuclear power plants are expected to run every unit at maximum capacity at all times, efficiently utilizing assets with minimal downtime. Surveillance and maintenance of nuclear decommissioning infrastructure provides lot challenges with respect to maintenance or decommissioning of the buildings. There is a need for a framework to analyze the huge amount of data generated by the sensors on the nuclear reactor components as well as structures, to monitor the conditions of these building over a period of time. Emerging technologies such as big data analytics have become a requirement in the nuclear industry to improve structural health monitoring and diagnostics. FIU will make use of existing mature technologies in the areas of imaging, robotics, big data, and machine learning/deep learning to assess the structural integrity of aging facilities at DOE sites. As these facilities await decommissioning, there is a need to understand the structural health of these structures. Many of these facilities were built over 50 years ago and in some cases these facilities have gone beyond the operational life expectancy. In other cases, the facilities have been placed in a state of 'cold and dark' and they are sitting unused, awaiting decommissioning. Finally, some older facilities are one-of-a-kind operational/production facilities supporting critical DOE missions and cannot be shut down. In any of these scenarios, the structural integrity of these facilities may be compromised, so it is imperative that adequate inspections and data collection/analysis be performed on a continuous and ongoing basis. The primary goals of the research include collecting various formats of data such as structured, unstructured and streaming data from the various sensors deployed in buildings, collect videos/pictures from various imaging sources, ingest them using a Hadoop distributed file system and process the data using Spark to perform batch processing and real time analytics. Research and development on various machine learning/deep learning algorithms will also be performed to analyze the heterogeneous data collected from nuclear decommission infrastructure. FIU will design the big data framework to ingest, store and process huge amounts of heterogeneous data collected from many sources and optimize the algorithms to provide insights into the data and predict anomalies observed when compared against baseline conditions. Various modules of this framework will include heterogeneous data sources, message broker, Hadoop distributed file system, Spark for stream and batch processing, machine learning/ deep learning, Cassandra - persistent data store and visualization. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA NTRS2020

Air Traffic Management TestBed Traffic Viewer: Developer's Guide

The Air Traffic Management (ATM) TestBed is a Platform as a Service that is being developed by the National Aeronautics and Space Administration (NASA) to help design, configure, integrate, run, and monitor air traffic simulations. The platform is designed to provide cloud services including back-end, big-data analytics tools, on-demand computing resource management, data storage, and communication middleware. The ATM TestBed reduces the time to test concepts and technologies, supports interactions among various methods such as human-in-the-loop and automation-in-the-loop simulations, and enables collaborative simulations by sharing technologies and tools in the ATM community. The Traffic Viewer application provides a graphical user interface tool for visualizing real and simulated air traffic as well as airspace definition in two-dimensional space. This guide describes a high-level design and implementation of Traffic Viewer and provides information for a new developer or a user to add new capabilities by following the software design and leveraging existing capabilities.

Lai, Chok Fung↗
DOE OSTI2017

Why wait? Let's start computing while data is still on the wire

In this era of Big Data, computing useful information from data is becoming increasingly complicated, particularly due to the ever increasing volumes of data that need to travel over the network to data centers to be stored and processed, all highly expensive operations in the long haul. In this paper we suggest that we can do computing and analysis of data "on the wire," i.e., while data is still in transit. The nature of these computations include analysis, visualization, pattern recognition, and prediction, or forecasting, on the streaming data. We follow a service function chaining architecture to implement this, assuming that the data packets arrive within a single network administrative domain. As a demonstration of this new computing paradigm, we present three examples. Firstly, we demonstrate pattern recognition and data visualization on streaming forex data, which can be used for lucrative trading in the forex market. In our second example, we analyze and learn user buying patterns from clickstream data streaming from multiple websites. Finally, we monitor solar sensors for a zero reading while the packets are still on their way to the data center, to schedule any maintenance and requisite repairs with no time delay.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Extreme-Scale Distribution-Based Data Analysis

Scientists overview and identify regions of interest by transforming data into compact information descriptors that characterize simulation results and allow detailed analysis on demand. Among many existing feature descriptors, statistical information derived from data samples is a promising approach to taming the big data avalanche because data distributionscomputed from a population can compactly describe the presence and characteristics of salient data features with minimal data movement. The ability to computationally summarize and process data using distributions also provides an efficient and representative capture of information that can adjust to size and resource constraints, with the added benefit that uncertainty associated with results can be quantified and communicated. In this project, we have developed a distribution-based data analysis and visualization framework for optimizing in situ processing and analysis of extreme-scale scientific data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Suite of Comprehensive Rapid Analysis Tools for Estimation (SOCRATES) - 19076

Long-term monitoring and remedy implementation require ongoing assessment and analytics, much of which relies on intensive and time-consuming data integration and engineering, data manipulation, and 'big-data' analytics. Many of these analyses, such as groundwater flow direction, contaminant rate attenuation, and non-parametric statistics, are spatial-temporal in nature and involve coalescing disparate data sets from a variety of sources with measurement frequencies from milliseconds to years, as well as total data volumes often exceeding the petabyte scale. Web-based analytical tools are advantageous for ongoing real-time or near-real-time ingestion of data from various sources for general data access and visualization, and also allow for complex, data rich calculations by leveraging expansive networks and virtually unlimited compute power. A web-based framework offers a natural architecture for such endeavors, as access to compute nodes through auto-scaling and dynamic frameworks enables speedy data manipulation and unlimited access to any data set that is made available via an http request. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA NTRS2020

Harnessing Big Data to Support the Conservation and Rehabilitation of Mangrove Forests Globally

Mangrove forests are found on sheltered coastlines in tropical, subtropical, and some warm temperate regions. These forests support unique biodiversity and provide a range of benefits to coastal communities, but as a result of large-scale conversion for aquaculture, agriculture, and urbanization, mangroves are considered increasingly threatened ecosystems. Scientific advances have led to accurate and comprehensive global datasets on mangrove extent, structure, and condition, and these can support evaluation of ecosystem services and stimulate greater conservation and rehabilitation efforts. To increase the utility and uptake of these products, in this Perspective we provide an overview of these recent and forthcoming global datasets and explore the challenges of translating these new analyses into policy action and on-the-ground conservation. We describe a new platform for visualizing and disseminating these datasets to the global science community, non-governmental organizations, government officials, and rehabilitation practitioners and highlight future directions and collaborations to increase the uptake and impact of large-scale mangrove research.

mangrove↗
DOE OSTI2022

Optimizing Error-Bounded Lossy Compression for Scientific Data With Diverse Constraints

Vast volumes of data are produced by today's scientific simulations and advanced instruments. These data cannot be stored and transferred efficiently because of limited I/O bandwidth, network speed, and storage capacity. Error-bounded lossy compression can be an effective method for addressing these issues: not only can it significantly reduce data size, but it can also control the data distortion based on user-defined error bounds. In practice, many scientific applications have specific requirements or constraints for lossy compression, in order to guarantee that the reconstructed data are valid for post hoc analysis. For example, some datasets contain irrelevant data that should be isolated in particular and users often have intuition regarding value ranges, geospatial regions, and other data subsets that are crucial for subsequent analysis. Existing state-of-the-art error-bounded lossy compressors, however, do not consider these constraints during compression, resulting in inferior compression ratios with respect to user's post hoc analysis, due to the fact that the data itself provides little or no value for post hoc analysis. In this work we address this issue by proposing an optimized framework that can preserve diverse constraints during the error-bounded lossy compression, e.g., cleaning the irrelevant data, efficiently preserving different precision for multiple value intervals, and allowing users to set diverse precision over both regular and irregular regions. We perform our evaluation on a supercomputer with up to 2,100 cores. Experiments with six real-world applications show that our proposed diverse constraints based error-bounded lossy compressor can obtain a higher visual quality or data fidelity on reconstructed data with the same or even higher compression ratios compared with the traditional state-of-the-art compressor SZ. Furthermore, our experiments also demonstrate very good scalability in compression performance compared with the I/O throughput of the parallel file system.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

FNETVision: A WAMS Big Data Knowledge Discovery System

With the growing awareness of the power system stability, wide area measurement system (WAMS) has been widely studied and deployed. In WAMS, a phasor data concentrator (PDC) collects and persists the GPS-time-synchronized phasor data from hundreds of phasor measurement units (PMU). Due to the size and complexity of the modern power systems, the number of deployed PMUs has been increasing and their data size becomes increasingly large. The large data volume challenges the analysis, and the knowledge discovery of the measurement data. This paper takes the pioneering distribution level WAMS- the frequency monitoring network FNET/GridEye as an example. It develops an adaptive visualization system- FNETVision to visualize the large-volume WAMS data by interconnection. The case studies suggest, based on the FNETVision system, the system administrator can quickly dig out issues of the FNET system and accordingly pinpoint the cause of these issues.

Wang, Weikang↗
NASA NTRS2018

Introduction to NASA Goddard Workshop on Artificial Intelligence

Artificial Intelligence (AI) is a collection of advanced technologies that allows machines to think and act, both humanly and rationally, through sensing, comprehending, acting and learning. AI's foundations lie at the intersection of several traditional fields Philosophy, Mathematics, Economics, Neuroscience, Psychology and Computer Science. Although the inception of AI started in the 1950's, it has recently made a strong comeback in all aspects of society and all over the world; this is mainly due to the timely combination of increased data volumes, advanced and mature algorithms, and improvements in computing power and storage. Current AI applications include big data analytics, robotics, intelligent sensing, assisted decision making, and speech recognition just to name a few.This workshop will be investigating how AI technologies can be adapted or developed to address the following challenges: Discover events of interest and correlations in large amounts of science data; improve the outcomes of science modeling and data assimilation using improved data processing, integration, and analysis. Design advisors for mission planning and operations, including anomaly detection and spacecraft health monitoring. Develop tools for engineering support, including advanced manufacturing, orbit determination, new component design and system engineering. Customize intelligent user interfaces, including visual analytics and natural language processing.

Le Moigne, Jacqueline↗
DOE OSTI2022

Findings on Subtask 3.1 - Bakken Rich Gas Enhanced Oil Recovery Project

Total in-place oil for the Bakken petroleum system (BPS) (which includes the Bakken and Three Forks Formations) has been estimated to be 600 billion barrels (bbl). However, BPS wells have decline rates as high as 85% over the first 3 years of their lives, and primary recovery factors typically range from 3% to 10% of original oil in place. Given the low initial recovery rates, even small incremental productivity improvements could dramatically increase technically recoverable oil in the BPS. One potential solution is enhanced oil recovery (EOR) using gas injection, such as carbon dioxide (CO2) or hydrocarbon (HC) gases. While commonly used in conventional reservoirs, CO2 EOR in unconventional tight oil reservoirs has been limited to pilot tests. EOR using rich gas (mixture of methane, ethane, and propane) has also been employed in numerous pilots in several unconventional plays and has recently been successfully applied in the Eagle Ford play. If successful, large-scale gas-based EOR in the BPS could dramatically increase oil productivity and recovery factors and extend the life of the play for decades. While CO2 may be a technically suitable working fluid for EOR in the BPS, supplies are limited and costs for using CO2 in EOR pilots are prohibitively high. Meanwhile, produced gas flaring has presented challenges for BPS operators in North Dakota. Analysis conducted by the North Dakota Pipeline Authority indicates that the current gas-gathering infrastructure in North Dakota is insufficient to accommodate all of the associated gas that is produced from the BPS. The geographically isolated location of North Dakota relative to large natural gas markets, combined with suppressed natural gas prices, has made it economically challenging for industry to invest capital in expanding gas-gathering infrastructure in the state. These circumstances led to a research program conducted by the Energy & Environmental Research Center (EERC) in partnership with Liberty Resources Management Company LLC (LR) to examine the potential to use rich gas injection for EOR and mitigate flaring. A rich gas EOR pilot test was designed and executed by LR at its Stomping Horse development area in Williams County, North Dakota. From July 2018 through May 2019, a total of 160 million standard cubic feet (MMscf) of rich produced gas was injected into the BPS using five different wells in a sequential injection strategy. LR’s Leon–Gohrick drill spacing unit (DSU) was used as the test site. Regulatory oversight was provided by the North Dakota Industrial Commission (NDIC). Technical support was provided by the EERC through a series of laboratory, modeling, and field-based activities, and additional post-pilot research activities incorporated learnings from the test, developed new laboratory data, improved fracture modeling methods, and developed machine learning and big data analytics. The results from the Stomping Horse rich gas EOR pilot activities indicate that developing an effective, economical EOR approach for the BPS will require more field tests. Another key lesson learned from the Stomping Horse tests is that detailed pre- and posttest data on reservoir conditions and fluids production are essential. Robust reservoir characterization provides information that is crucial to creating realistic geomodels and conducting valid dynamic simulations of potential EOR scenarios. A detailed understanding of the completions and production history of offset wells is also necessary for valid test result interpretations. This knowledge is essential to designing the operational parameters of injectivity tests and interpreting the results. A conformance control strategy is also essential to success. Laboratory-based examinations of rich gas interactions with reservoir fluids and rocks were conducted, with an emphasis on determining the ability to mobilize oil in the tight reservoir rocks and shales of the BPS. Injection fluid composition was shown to have a positive impact on reducing reservoir oil minimum miscibility pressure (MMP), reducing interfacial tension (IFT), and altering wettability. IFT and contact angle measurements demonstrated that wettability can be altered in the presence of rich gas, suggesting the potential to improve oil recovery. Iterative modeling of surface infrastructure and reservoir performance using data generated by the various project activities was conducted. A geologic model of the Stomping Horse area was built; history-matched oil, gas, and water production was used in simulations of various EOR scenarios. Early programmatic modeling results were used to support LR’s design and operation of the EOR pilot and to provide insight regarding optimization of future commercial-scale BPS EOR design and operations. Post-pilot modeling focused on alternative methods of understanding complex fracture networks and accelerating simulation time. These led to improved simulation run times and provide excellent history-matching results. Several of these iterative models were used as the bases for developing algorithms into machine learning and big data analytics. History matching in reservoir simulation is time-consuming and computer processing-intensive. Machine learning algorithms were created, and an automated history-matching tool was developed. A large set of synthetic reservoir simulations were created to generate well responses (oil, gas, and water production, well bottomhole pressure [BHP], and tracer or propane breakthrough) for a set of EOR operating parameters that included offset well status (open or closed), injectate (rich gas or propane), injection rate, and injection well BHP. A user interface was developed to provide real-time visualization. Machine learning-based models were developed to provide rapid forecasting of well performance given a set of user-defined EOR operating parameters. These predictive models allow the user to modify the offset well status, injection rate, and injection well BHP and rapidly forecast future production performance. The combination of real-time visualization tools with real-time forecasting tools provides a framework for real-time control—operational changes that the EOR site operator can enact (e.g., changing gas injection rates) to affect the observed performance and potentially improve the EOR outcome. There is great reason to be optimistic about the future of EOR in the Bakken. The results of the laboratory studies suggest significant potential for high rates of oil mobilization using produced field gas injection under the right conditions. The results of the lab studies, combined with rigorous statistical analysis of well production data and associated modeling efforts, confirm the notion that fluid mobility within the reservoir is controlled by fractures. As more knowledge is gained about the nature and distribution of fracture networks in the Bakken, the industry will be in a better position to predict and, ultimately, influence fluid mobility. New field tests are necessary to develop a more complete understanding of those conditions. Thoughtful and creatively engineered field tests within a well-characterized geologic setting will yield the fundamental knowledge needed to take Bakken oil production to the next level. This subtask was cofunded through the EERC–U.S. Department of Energy Joint Program on Research and Development for Fossil Energy-Related Resources Cooperative Agreement No. DE-FE0024233. Nonfederal funding was provided by the North Dakota Industrial Commission’s Oil and Gas Research Program and Computer Modelling Group.

04 OIL SHALES AND TAR SANDS↗
DOE OSTI2026

BiG-SCAPE 2.0 and BiG-SLiCE 2.0: scalable, accurate and interactive sequence clustering of metabolic gene clusters

Microbial metabolic gene clusters encode the biosynthesis or catabolism of metabolites that facilitate ecological specialization, mediate microbiome interactions and constitute a major source of medicines and crop protection agents. Here, we present BiG-SCAPE and BiG-SLiCE 2.0, next-generation methods that facilitate scalable, accurate and interactive gene cluster analyses. BiG-SCAPE 2.0 updates its classification, alignment methods, and visualizations, enabling more accurate analysis, up to 8x faster runtimes and halved memory requirements. BiG-SLiCE 2.0 updates its distance metric, pHMM database, and classification logic, resulting in increased sensitivity nearing that of BiG-SCAPE. Analysis of 260,630 biosynthetic gene clusters from publicly available genomes reveals that both tools generate concurring estimates of gene cluster diversity, thus providing significantly extended methodological support for recent evidence indicating that the vast majority of natural product diversity remains unexplored. Together, these updates will facilitate global genome mining efforts for natural product discovery and microbiome analyses scalable with current data sizes.

Draisma, Arjan [Wageningen University & Research (↗
DOE OSTI2020

Examining Perceptions of Spanish Language Health Information Among Hispanics Living in the United States: A Qualitative Study Assessing Videos, Brochures, and Websites

Background Nationally, Hispanics experience health disparities manifested as a higher prevalence of chronic diseases in comparison to non-Hispanic Whites. Factors that influence health disparities in this population include inadequate dissemination of culturally and linguistically appropriate health information. Method Our qualitative analysis is based on data obtained from three focus groups with Spanish-speaking Hispanic adults conducted at a clinic in Nebraska. Participants were asked about their perceptions of health information in the form of videos, brochures, and websites. We followed the consolidated criteria for reporting qualitative research and used the theoretical framework of content analysis. Results For videos, attributes associated with higher levels of trust included a friendly health professional, the logo or name of a health care institute, and a reference specifying “se habla Español.” For brochures, attributes associated with trust included use of visual aids, use of numerous colors and big font sizes, the year of printing, and a health care agency logo. For websites, characteristics associated with trust were inclusion of plain language, use of pictures and videos, and date of last update. In all focus groups, participants perceived the use of mixed English/Spanish language in health information from pamphlets and websites as unprofessional. The use of unknown governmental logos in health promotional videos and websites was perceived as untrustworthy information. Conclusions Spanish-speaking Hispanics prefer health information supported by a health care agency or delivered by a health care professional. Health communication strategies should avoid the use of mixed English/Spanish language in sensitive information especially when combined with governmental logos.

De Alba Rosales, Armando↗
NASA NTRSDate not supplied

Development of an Airspace Simulation and Modeling Tool for Enhanced Spectrum Management

The emergence of new aerial vehicles into the National Airspace System creates an increased demand for aeronautical communications to support aviation operations. However, the issue of spectrum scarcity remains an ever-present concern, and the growing demand cannot be supported using existing spectrum allocation strategies. As a result, a new spectrum management approach is required, and the National Aeronautics and Space Administration (NASA) is investigating advanced concepts to modernize the management and use of aviation spectrum by leveraging the latest advancements in wireless communications, big data and machine learning. This research proposes an autonomous spectrum allocation concept, which allocates communications resources, such as spectrum and power, based on the predicted communications and air traffic demands throughout the airspace, as opposed to the use of fixed allocations as is done today. This approach will result in improved spectrum utilization efficiency and enhanced airspace capacity. The autonomous spectrum allocation concept decomposes into three research areas: demand prediction, resource allocation, and use case evaluation. As part of the use case evaluation effort, a modeling and simulation capability is currently under development. This simulation capability includes the implementation of various features, including visualization of both live or virtually-generated airspace traffic, simulation scenario development, simulation management with data collection, and flight plan creation with corresponding trajectory generation. This modeling and simulation capability will continue to evolve as new and advanced airspace applications are introduced into existing and emerging operational environments.

Eric J. Knoblock↗
DOE OSTI2019

Materials informatics: From the atomic-level to the continuum

We report that in recent years materials informatics, which is the application of data science to problems in materials science and engineering, has emerged as a powerful tool for materials discovery and design. This relatively new field is already having a significant impact on the interpretation of data for a variety of materials systems, including those used in thermoelectrics, ferroelectrics, battery anodes and cathodes, hydrogen storage materials, polymer dielectrics, etc. Its practitioners employ the methods of multivariate statistics and machine learning in conjunction with standard computational tools (e.g., density-functional theory) to, for example, visualize and dimensionally reduce large data sets, identify patterns in hyperspectral data, parse microstructural images of polycrystals, characterize vortex structures in ferroelectrics, design batteries and, in general, establish correlations to extract important physics and infer structure-property-processing relationships. In this Overview, we critically examine the role of informatics in several important materials subfields, highlighting significant contributions to date and identifying known shortcomings. We specifically focus attention on the difference between the correlative approach of classical data science and the causative approach of physical sciences. From this perspective, we also outline some potential opportunities and challenges for informatics in the materials realm in this era of big data.

36 MATERIALS SCIENCE↗
NASA NTRS2014

Analyzing a 35-Year Hourly Data Record: Why So Difficult?

At the Goddard Distributed Active Archive Center, we have recently added a 35-Year record of output data from the North American Land Assimilation System (NLDAS) to the Giovanni web-based analysis and visualization tool. Giovanni (Geospatial Interactive Online Visualization ANd aNalysis Infrastructure) offers a variety of data summarization and visualization to users that operate at the data center, obviating the need for users to download and read the data themselves for exploratory data analysis. However, the NLDAS data has proven surprisingly resistant to application of the summarization algorithms. Algorithms that were perfectly happy analyzing 15 years of daily satellite data encountered limitations both at the algorithm and system level for 35 years of hourly data. Failures arose, sometimes unexpectedly, from command line overflows, memory overflows, internal buffer overflows, and time-outs, among others. These serve as an early warning sign for the problems likely to be encountered by the general user community as they try to scale up to Big Data analytics. Indeed, it is likely that more users will seek to perform remote web-based analysis precisely to avoid the issues, or the need to reprogram around them. We will discuss approaches to mitigating the limitations and the implications for data systems serving the user communities that try to scale up their current techniques to analyze Big Data.

computational performance↗
DOE OSTI2019

Atom-by-atom fabrication with electron beams

Assembling matter atom-by-atom into functional devices is the ultimate goal of nanotechnology. The possibility of achieving this goal is intrinsically dependent on the ability to visualize matter at the atomic level, induce and control atomic-scale motion, facilitate and direct chemical reactions, and coordinate and guide fabrication processes towards desired structures atom-by-atom. In this Perspective, we summarize recent progress in chemical transformations, material alterations and atomic dynamics studies enabled by the converged, atomic-sized electron beam of an aberration-corrected scanning transmission electron microscope. We discuss how such top-down observations have led to the concept of controllable, beam-induced processes and then of bottom-up, atom-by-atom assembly via electron-beam control. Finally, the progress in this field, from electron-beam-induced material transformations to atomically precise doping and multi-atom assembly, is reviewed, as are the associated engineering, theoretical and big-data challenges.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2022

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2017

Final Project Report: Composition and Realization of Source-to-Sink High-Performance Flows: File Systems, Storage, Hosts, LAN and WAN

A number of Department of Energy (DOE) science applications, involving exascale computing systems and large experimental facilities, are expected to generate large volumes of data, in the range of petabytes to exabytes, which will be transported over wide-area networks for the purpose of storage, visualization, and analysis. The objectives of this proposal are to (1) develop and test the component technologies and their synthesis methods to achieve source-to-sink high-performance flows, and (2) develop tools that provide these capabilities through simple interfaces to users and applications. In terms of the former, we propose to develop (1) optimization methods that align and transition multiple storage flows to multiple network flows on multicore, multibus hosts; and (2) edge and long-haul network path realization and maintenance using advanced provisioning methods including OSCARS and OpenFlow. We also propose synthesis methods that combine these individual technologies to compose high-performance flows using a collection of constituent storage-network flows, and realize them across the storage and local network connections as well as long-haul connections. We propose to develop automated user tools that profile the hosts, storage systems, and network connections; compose the source-to-sink complex flows; and set up and maintain the needed network connections.

97 MATHEMATICS AND COMPUTING↗