Search NASA⌕ Search

SEARCH · Search NASA

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Environmental sustainability in basic research: a perspective from HECAP+

The climate crisis and the degradation of the world's ecosystems require humanity to take immediate action. The international scientific community has a responsibility to limit the negative environmental impacts of basic research. The HECAP+ communities (High Energy Physics, Cosmology, Astroparticle Physics, and Hadron and Nuclear Physics) make use of common and similar experimental infrastructure, such as accelerators and observatories, and rely similarly on the processing of big data. Our communities therefore face similar challenges to improving the sustainability of our research. This document aims to reflect on the environmental impacts of our work practices and research infrastructure, to highlight best practice, to make recommendations for positive changes, and to identify the opportunities and challenges that such changes present for wider aspects of social responsibility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Hardening DOE R&D Software Tools for Web-based Visualization SBIR Phase I Final Report

Ubiquitous web-based visualization is essential to delivering large-scale data visualization to various stakeholders, from the scientist to the board member. These stakeholders will not tolerate a stalled application or a pop-up window asking them to wait for the processing to complete. They require a responsive and interactive visualization environment with high-quality imagery suitable for detailed analysis and boardroom presentations. At Kitware, Inc., we have accomplished web visualization to this point, leveraging state-of-the-art tools like HTML5, CSS3, SVG, Canvas, and WebGL. Solutions that leverage a combination of these technologies are necessary to handle workloads that vary significantly in data size efficiently. However, it is not always practical to move large data to the web client for visualization. Kitware's ParaView as a Service combines client-side visualization using both distributed processing and remote rendering on big data impractical to move. Existing distributed processing and remote rendering solution's interactivity is below the expectations of web-based applications. Our project examined proposed solutions to the areas outlined above in ParaView as a Service. We have investigated concurrent pipelines, streaming images, progressive rendering, and optimization of algorithms and data movement to address these concerns. For the Phase I project, we completed the proposed work plan. As a result, the project produced three prototypes of essential importance for web visualization and the ParaView as a Service community. We created a simple desktop application for an interactive streamline placement prototype, a web-based interactive streamline placement prototype, and a web-based progressive rendering utilizing raytracing prototype. These prototypes relied on the hardening of emerging software toolkits funded by the Department of Energy (DOE) Advanced Scientific Computing Research (ASCR) program (such as ParaView, VTK-m, and Mochi). We blended these components into web-based visualization prototypes that meet the industry's expectations for interactivity and responsiveness. The Phase I project had four essential focus areas: 1. Develop prototype ParaView as a Service backend server using asynchronous, non-blocking design principles. 2. Develop a prototype web application that uses the ParaView as a Service backend server for remote data visualization. 3. Implement image streaming with encoding/compression and progressive rendering capabilities in the proposed platform. 4. Evaluate the prototype developed and summarize observations, including the challenges and pitfalls of our approach. After our successful completion of Phase I, we are strongly positioned to propose a successful Phase II project.

Geveci, Berk↗

Integrated Large-Scale Data Management Platform for Photovoltaic Power Conversion Equipment (PCE) Reliability Data

To meet the demand for accuracy and real-time capability of PV system degradation evaluation, massive volume data is needed to run high-fidelity and high-efficiency simulations and perform advanced data analysis. However, PV farm operators have a series of difficulties with PV inverter data, such as data collection from multiple channels, massive data storage, data management and massive data analysis. To address these challenges, we developed an integrated data management platform capable of data acquisition, processing, storage, query, and performing big data analysis utilizing AI algorithms. The platform can also achieve data correctness verification and provide an effective distributed data management solution to retrieve massive data and establish a connection to distributed computational frameworks.

data management platform↗

Integrated Large-Scale Data Management Platform for Photovoltaic Power Conversion Equipment (PCE) Reliability Data: Preprint

To meet the demand for accuracy and real-time capability of PV system degradation evaluation, massive volume data is needed to run high-fidelity and high-efficiency simulations and perform advanced data analysis. However, PV farm operators have a series of difficulties with PV inverter data, such as data collection from multiple channels, massive data storage, data management and massive data analysis. To address these challenges, we developed an integrated data management platform capable of data acquisition, processing, storage, query, and performing big data analysis utilizing AI algorithms. The platform can also achieve data correctness verification and provide an effective distributed data management solution to retrieve massive data and establish a connection to distributed computational frameworks.

data management↗

EDX ClaiMM

EDX ClaiMM is a centralized data & analytical platform designed to revolutionize U.S. critical minerals and materials (CMM) activities. By providing a robust digital infrastructure, ClaiMM will accelerate the combination, leveraging, and rapid utilization of vital data, advanced tools, and cutting-edge research advancements in CMM. This adaptive digital research hub connects the CMM community to essential knowledge products and offers access to interoperable datasets, databases, models, software, and tools from the National Energy Technology’s (NETL’s) Energy Data eXchange (EDX) and other authoritative sources, serving both public and private sectors. EDX ClaiMM delivers AI-informed solutions to address fundamental knowledge gaps and fosters the innovation of new techniques for enhanced characterization and recovery of CMMs within the U.S. By leveraging cloud-hosted, scalable digital infrastructure, ClaiMM meets public–private applied energy needs. It equips the CMM community with priority digital resources that harness on-site and cloud compute capabilities, enabling big data storage, advanced processing, analytics, and visualization.

Critical Materials; Critical Minerals; Rare Earth ↗

A spark‐based big data analysis framework for real‐time sentiment prediction on streaming data

Summary There are many data sources that produce large volumes of data. The Big Data nature requires new distributed processing approaches to extract the valuable information. Real‐time sentiment analysis is one of the most demanding research areas that requires powerful Big Data analytics tools such as Spark. Prior literature survey work has shown that, though there are many conventional sentiment analysis researches, there are only few works realizing sentiment analysis in real time. One major point that affects the quality of real‐time sentiment analysis is the confidence of the generated data. In more clear terms, it is a valuable research question to determine whether the owner that generates sentiment is genuine or not. Since data generated by fake personalities may decrease accuracy of the outcome, a smart/intelligent service that can identify the source of data is one of the key points in the analysis. In this context, we include a fake account detection service to the proposed framework. Both sentiment analysis and fake account detection systems are trained and tested using Naïve Bayes model from Apache Spark's machine learning library. The developed system consists of four integrated software components, ie, (i) machine learning and streaming service for sentiment prediction, (ii) a Twitter streaming service to retrieve tweets, (iii) a Twitter fake account detection service to assess the owner of the retrieved tweet, and (iv) a real‐time reporting and dashboard component to visualize the results of sentiment analysis. The sentiment classification performances of the system for offline and real‐time modes are 86.77% and 80.93%, respectively.

Kılınç, Deniz↗

Lidar Mapping and Surveillance of Nuclear Infrastructure

Wear is unavoidable, whether it occurs in a nuclear facility or abridge on the highway its detection is crucial to preventing the failure of a system. While tools capable of detecting localized damage exist, industry lacks methods to evaluate the health of larger infrastructure. Routine inspection of nuclear facilities must be undertaken to ensure the safety of personnel and proper containment of nuclear materials. Most systems in a nuclear facility are designated as radioactive and confined spaces. To constantly monitor such systems in person would be hazardous but with a remote inspection tool it can be done in a safer and easier way. Lidar: Operates by emitting a laser pulse and looking for the reflected light. Allows user to gain 3D information of the environment. Project Objective: To develop an inspection tool for evaluating the health of large scale infrastructure using Lidar and machine learning. Autonomous Platform + Lidar + Data Fusion + Big Data. Use SLAM to build a 2D map of the environment. Point cloud created by transforming the points between reference frames. Point cloud to Surface: Inherent representation of surface geometry; Difficult to process with conventional techniques. Point cloud to Image: Easily processed by conventional machine learning algorithms. The quantity of data generated will necessitate a big data framework to manage the data for processing. Additional Sensors: Detecting the temperature (a) chemical makeup (b) and radiation emission (c) of a surface could assist in detecting problems. Human inspection can detect irregularities in both physical objects and their point-cloud representations. This is a good indication that mapping nuclear sites using Lidar will provide qualitative insight into the health of the facility. If deep learning can be used to automatically identify problem areas then the completed tool could become an invaluable asset.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Integrated parameter and process learning for hydrologic and biogeochemical modules in Earth System Models

Focus area: Primary focal area #2; secondary focal area #3: Learning about parameters and processes of land surface hydrologic and biogeochemical models in Earth System models by integrating machine learning, physics, and big data. Science challenges: How do we maximally leverage big-data observations to improve hydrobiogeochemical process description and parameterization so that such modules more realistically capture hydrologic and vegetation responses and feedbacks under the future climate? For example, how can we leverage physics, limited observations of vegetation and streamflow to better estimate evapotranspiration, and, relatedly, net primary productivity, especially for drought areas? Vegetation plays a critical role in regional and global water cycles; however, existing vegetation models have failed to predict vegetation response to droughts (McDowell & Xu, 2017) , arctic greening (Keenan & Riley, 2018) , and critical transitions between forest and savanna (Hirota et al., 2011) . These studies suggest that when we build process-based models (PBM) parameterized from regional and global plant traits, we tend to poorly describe plant adaptation and local-scale competition processes. The models and their associated parameters assigned for different regions in the world are not capturing essential heterogeneity in vegetation responses at finer spatial scales. Many parameters of the land surface models control hydrology and vegetation dynamics at the same time. The heterogeneity in vegetation response is a function of (i) plant type, (ii) plant size, (iii) competition and succession, (iv) environmental controls, and (v) local variations due to the unique ecological community that are very difficult to describe (e.g., the size of gaps resulting from fire that facilitated the coexistence of pioneering species). In the demographic models, only factors (i) and (iv) were captured, and plant types were generally described only by leaf phenology and climate zones. With current demographic models, we generally consider more traits to define plant types (i) and calibrate these traits to consider factors (ii), (iii) and (iv); however, it is substantially challenging to scale to regional and global simulations due to trait variations across space (Ali et al., 2016). Moreover, it has been noted that hillslope processes, including ridge-to-valley flow and sunny vs. shady slopes are primary organizers of water, energy, and vegetation (Clark et al., 2015; Fan et al., 2019) . Although gradual improvements in the hydrologic model component in earth system models may reduce this error (at a remarkably slow pace), the long-term, gradual impact of hydrology on plant traits are not well captured. Recent work showed that the hydrologic controls exerted by groundwater and lateral flow are primary regulators of rooting depth (Fan et al., 2017) . Such hydrologic controls have seldom been reflected in vegetation model parameterizations.

54 ENVIRONMENTAL SCIENCES↗

Big Data Analysis and Technical Review of Regeneration for Carbon Capture Processes

Carbon capture remains an integral technology to mitigate pollution from one of the most prevalent greenhouse gases. CO 2 desorption/absorbent regeneration for both solid- and liquid-based systems is widely recognized as an energy-intensive and costly process operation. Consequently, tremendous work was devoted towards developing new absorbents and regeneration processes to promote their economic feasibility for extensive implementation. In this review, we broadly and deeply review more than 10,000 papers and extract the hidden trends of carbon capture and absorbents regeneration in the past few decades, using a novel data-mining analysis technique. We comprehensively analyzed an array of recent absorbent regeneration methods utilized in post-combustion, pre-combustion, carbon capture from industrial point sources, and direct air carbon capture, with an emphasis on sorbent and solvent-based techniques. In conclusion, advanced regeneration methods in these techniques were illustrated and discussed, followed by recommendations for further research efforts.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Enhanced Signal Processing of Distributed Brillouin Fiber Sensors using a Decoupled Radial Basis Function Network

A novel decoupled radial basis function network (D-RBFN) is proposed to accelerate signal processing and address the big data challenges associated with ultra-long distance Brillouin optical time-domain analysis (BOTDA) systems. The proposed frame- work is demonstrated on a dataset measured over a 100 km distance using a bi-directional Raman assisted BOTDA system.

Venketeswaran, Abhishek↗

Nanoscale defect evaluation framework combining real-time transmission electron microscopy and integrated machine learning-particle filter estimation

Observation of dynamic processes by transmission electron microscopy (TEM) is an attractive technique to experimentally analyze materials’ nanoscale phenomena and understand the microstructure-properties relationships in nanoscale. Even if spatial and temporal resolutions of real-time TEM increase significantly, it is still difficult to say that the researchers quantitatively evaluate the dynamic behavior of defects. Images in TEM video are a two-dimensional projection of three-dimensional space phenomena, thus missing information must be existed that makes image’s uniquely accurate interpretation challenging. Therefore, even though they are still a clustering high-dimensional data and can be compressed to two-dimensional, conventional statistical methods for analyzing images may not be powerful enough to track nanoscale behavior by removing various artifacts associated with experiment; and automated and unbiased processing tools for such big-data are becoming mission-critical to discover knowledge about unforeseen behavior. We have developed a method to quantitative image analysis framework to resolve these problems, in which machine learning and particle filter estimation are uniquely combined. The quantitative and automated measurement of the dislocation velocity in an Fe-31Mn-3Al-3Si autunitic steel subjected to the tensile deformation was performed to validate the framework, and an intermittent motion of the dislocations was quantitatively analyzed. The framework is successfully classifying, identifying and tracking nanoscale objects; these are not able to be accurately implemented by the conventional mean-path based analysis.

36 MATERIALS SCIENCE↗

Summary report of the 4th IAEA Technical Meeting on Fusion Data Processing, Validation and Analysis (FDPVA)

The objective of the Fourth Technical Meeting on Fusion Data Processing, Validation and Analysis was to provide a platform during which a set of topics relevant to fusion data processing, validation and analysis are discussed with the view of extrapolating needs to next step fusion devices such as ITER. The validation and analysis of experimental data obtained from diagnostics used to characterize fusion plasmas are crucial for a knowledge-based understanding of the physical processes governing the dynamics of these plasmas. This paper presents the recent progress and achievements in the domain of plasma diagnostics and synthetic diagnostics data analysis (including image processing, regression analysis, inverse problems, deep learning, machine learning, big data and physics-based models for control) reported at the meeting. The progress in these areas highlight trends observed in current major fusion confinement devices. A special focus is dedicated on data analysis requirements for ITER and DEMO with a particular attention paid to artificial intelligence for automatization and improving reliability of control processes.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

High-Throughput Computing: Case Study of Medical Image Processing Applications

HPC is designed for large-scale simulations using monolithic codes of tightly coupled processes highly optimized to deliver decreased time to solution. Medical image processing is not a traditional field of HPC. Similar to AI applications, medical image processing parses large datasets, typically multiple times, to support a variety of studies for classification, diagnosis or monitoring purposes. The convergence of AI, HPC and Big Data encouraged more fields using image processing to transition to HPC. However, not all applications benefit from the same optimizations. In this paper we focus on high throughput medical image processing applications that analyze a huge dataset of small MRI images and that require HPC systems to decrease the time of parsing the entire dataset and not individual MRIs. We show in this research the performance of running SLANT, an image processing application for a whole brain segmentation, on large-scale systems and highlight performance limitations. We present optimizations prioritizing throughput that exhibit a 3.5x speed-up on the Summit Supercomputer that can be used as a baseline for building a high-throughput execution framework for other HPC systems.

Predescu, Maria↗

Big Data Framework with Machine Learning for D and D Applications - 19108

The nuclear industry is experiencing a steady increase in maintenance costs even though plants are maintained under high levels of safety, capability and reliability. Nuclear power plants are expected to run every unit at maximum capacity at all times, efficiently utilizing assets with minimal downtime. Surveillance and maintenance of nuclear decommissioning infrastructure provides lot challenges with respect to maintenance or decommissioning of the buildings. There is a need for a framework to analyze the huge amount of data generated by the sensors on the nuclear reactor components as well as structures, to monitor the conditions of these building over a period of time. Emerging technologies such as big data analytics have become a requirement in the nuclear industry to improve structural health monitoring and diagnostics. FIU will make use of existing mature technologies in the areas of imaging, robotics, big data, and machine learning/deep learning to assess the structural integrity of aging facilities at DOE sites. As these facilities await decommissioning, there is a need to understand the structural health of these structures. Many of these facilities were built over 50 years ago and in some cases these facilities have gone beyond the operational life expectancy. In other cases, the facilities have been placed in a state of 'cold and dark' and they are sitting unused, awaiting decommissioning. Finally, some older facilities are one-of-a-kind operational/production facilities supporting critical DOE missions and cannot be shut down. In any of these scenarios, the structural integrity of these facilities may be compromised, so it is imperative that adequate inspections and data collection/analysis be performed on a continuous and ongoing basis. The primary goals of the research include collecting various formats of data such as structured, unstructured and streaming data from the various sensors deployed in buildings, collect videos/pictures from various imaging sources, ingest them using a Hadoop distributed file system and process the data using Spark to perform batch processing and real time analytics. Research and development on various machine learning/deep learning algorithms will also be performed to analyze the heterogeneous data collected from nuclear decommission infrastructure. FIU will design the big data framework to ingest, store and process huge amounts of heterogeneous data collected from many sources and optimize the algorithms to provide insights into the data and predict anomalies observed when compared against baseline conditions. Various modules of this framework will include heterogeneous data sources, message broker, Hadoop distributed file system, Spark for stream and batch processing, machine learning/ deep learning, Cassandra - persistent data store and visualization. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Managing genomic variant calling workflows with Swift/T

Bioinformatics research is frequently performed using complex workflows with multiple steps, fans, merges, and conditionals. This complexity makes management of the workflow difficult on a computer cluster, especially when running in parallel on large batches of data: hundreds or thousands of samples at a time. Scientific workflow management systems could help with that. Many are now being proposed, but is there yet the “best” workflow management system for bioinformatics? Such a system would need to satisfy numerous, sometimes conflicting requirements: from ease of use, to seamless deployment at peta- and exa-scale, and portability to the cloud. We evaluated Swift/T as a candidate for such role by implementing a primary genomic variant calling workflow in the Swift/T language, focusing on workflow management, performance and scalability issues that arise from production-grade big data genomic analyses. In the process we introduced novel features into the language, which are now part of its open repository. Additionally, we formalized a set of design criteria for quality, robust, maintainable workflows that must function at-scale in a production setting, such as a large genomic sequencing facility or a major hospital system. The use of Swift/T conveys two key advantages. (1) It operates transparently in multiple cluster scheduling environments (PBS Torque, SLURM, Cray aprun environment, etc.), thus a single workflow is trivially portable across numerous clusters. (2) The leaf functions of Swift/T permit developers to easily swap executables in and out of the workflow, which makes it easy to maintain and to request resources optimal for each stage of the pipeline. While Swift/T’s data-level parallelism eliminates the need to code parallel analysis of multiple samples, it does make debugging more difficult, as is common for implicitly parallel code. Nonetheless, the language gives users a powerful and portable way to scale up analyses in many computing architectures.

Science & Technology - Other Topics↗

Macromolecules & Manufacturing Science

Outline • SRNL Overview • Mission overview • Polymers enabling the mission • R&D Highlights • Polymers in radiation environments • Tooling in shielded cells • Packaging for nuclear material shipments • Polymers supporting tank waste remediation • Ref electrode • Epoxy and polymer grout • Polymers for fusion energy • Deuterium labelling • Polymers for additive manufacturing • Coalescence and blends: experimental and predictive • Process modelling and sorting through big data (peregrine and latticeJ)

Chatham, Camden [Savannah River National Laborator↗

DICER: Data Intensive Computing Environment and Runtime for Evaluating Unprecedented Scale of Geospatial-Temporal Human Mobility Data

With the significant increase in sources and volume of human mobility data through commercial data vendors as well as microsimulation of cities, the scale of geospatial-temporal data to analyze and assess for mobility characterization has grown to the level of Big Data. There are mobility related commercial organizations deploying scalable computing, but often the system architecture, workflow, and intermediate processing components are not fully disclosed in relevant scope. Current research literature has a notable lack of studies demonstrating architectures and workflows for human mobility analytics that are implemented on a TeraByte scale of geospatial-temporal data. In this context, this paper presents a hyperscale-level system solution named DICER (Data Intensive Computing Environment and Runtime) for processing and analytics of geospatial-temporal data at big data scale. Although the cluster computing architecture of DICER with Apache Spark job running on Kubernetes cluster is not new, there are innovations in the workflow, hierarchical processing logic, and a wide range of intermediate preprocessing and mobility metrics calculation. We have performed case studies to validate the effectiveness of DICER system solution by performing detailed analytics and assessment of human mobility microsimulation output at three different scopes and scale, including a usecase with 16.97 TeraByte and 259.2 Billion rows of data. In addition, we have presented another case study of utilizing DICER to perform the same mobility processing and comparative analytics on large-scale commercially available geospatial-temporal data. All these case studies validate the efficiency and usefulness of DICER in computing population mobility characteristics from geospatial-temporal trajectory data at an unprecedented scale (not only just data volume, but also combination of: number of user entities, temporal frequency, spatial resolution, data duration).

De, Debraj↗