Algorithms Data Science and Cyber Analytics at Sandia National Laboratories.
Abstract not provided.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Abstract not provided.
The paper presents initial results from the development of a cloud-based, Big Data analysis platform for power systems. The computational pipeline uses the Apache Spark framework running in an OpenStack cloud infrastructure. A real-world phasor measurement unit (PMU) dataset has been used to carry out the analysis. Actual examples of power system events detection using synchrophasor data are presented. It has been shown that applications of the cloud based computing environment and the Apache Spark framework enable a significant increase in the computational efficiency of large-scale PMU data analysis.
Abstract not provided.
As the energy industry continues to invest in data, big data, and advanced analytics to improve production, reduce operation and maintenance costs, improve safety, and reduce risks, uncertainty will serve as a limiting factor in the success of these operations. Intuitive communication of uncertainty can provide more transparency in the decision-making process; offering the ability to more accurately evaluate the risk and impact of decisions, identify data and knowledge gaps, and evaluate trade-offs. The issue of uncertainty will thus play a large role in the success of investments that rely on big data and data analytics. However, no robust method for representation of uncertainty in spatial data visuals currently exists. As such, uncertainty is, at best, communicated using multiple independent visuals and aggregated in final displays, or frequently omitted altogether. Failure to accurately and effectively quantify and communicate uncertainty poses a barrier, limiting critical information that would better allow businesses to make more accurate and intelligent decisions.
Currently, organizations have to maintain separate systems for transactions and analytics inside the company. Notably, different vendors provide the capabilities for either of these tasks that require specialized hardware or software. Data engineers are required to retrieve data from one source and transform it into another format to obtain the maximum benefit in a minimum time. Organizations strive for a competitive advantage that is achieved by fetching data from their customers and getting insights earliest for timely decision making. Present practices do not permit the view of the latest data for analytics since the first data have to be fetched from the source, transformed, and loaded to other systems to be utilized for analysis by relevant teams. This paper introduces a single system for both transactions and analytics. Our proposed solution would permit companies to seamlessly adapt our solution without the need to shift all of their data to newer systems and allow all the teams. It would grant all the teams to have a view of the latest available data without extra expertise and budget.
Global climate researchers rely upon many forms of sensor data and analytical methods to help profile subtle changes in climate conditions. The U.S. Department of Energy Atmospheric Radiation Measurement (ARM) program provides researchers with curated products called Value Added Products (VAPs) resulting from continuous instrumentation streams, data fusion, and analytical profiling. To provide these projects the ARM operations and research teams rely upon a number of techniques to ensure strict quality control and quality assurance codes are maintained. End users in the climate research community are highly interested in obtaining as much causal evidence as possible and currently either not all the evidence are easily attainable or easily identifiable without significant effort. Our research interests are to identify a provenance model that serves both the producers and consumers of the VAP maintaining the quality assurance/quality control standards and tailored to meeting the individual researcher’s needs.
Sampling has been completed for the characterization of the residual material on the floor of Tank 5 in the F-Area Tank Farm at the Savannah River Site (SRS), near Aiken, SC. The sampling was performed by Savannah River Remediation (SRR) LLC using a stratified random sampling plan with volume-proportional compositing. The plan consisted of partitioning the residual material on the floor of Tank 5 into three non-overlapping strata: two strata enclosed accumulations, and a third stratum consisted of a thin layer of material outside the regions of the two accumulations. Each of three composite samples was constructed from five primary sample locations of residual material on the floor of Tank 5. Three of the primary samples were obtained from the stratum containing the thin layer of material, and one primary sample was obtained from each of the two strata containing an accumulation. This report documents the statistical analyses of the analytical results for the composite samples. The objective of the analysis is to determine the mean concentrations and upper 95% confidence (UCL95) bounds for the mean concentrations for a set of analytes in the tank residuals. The statistical procedures employed in the analyses were consistent with the Environmental Protection Agency (EPA) technical guidance by Singh and others [2010]. Savannah River National Laboratory (SRNL) measured the sample bulk density, nonvolatile beta, gross alpha, and the radionuclide1, elemental, and chemical concentrations three times for each of the composite samples. The analyte concentration data were partitioned into three separate groups for further analysis: analytes with every measurement above their minimum detectable concentrations (MDCs), analytes with no measurements above their MDCs, and analytes with a mixture of some measurement results above and below their MDCs. The means, standard deviations, and UCL95s were computed for the analytes in the two groups that had at least some measurements above their MDCs. The identification of distributions and the selection of UCL95 procedures generally followed the protocol in Singh, Armbya, and Singh [2010]. When all of an analyte's measurements lie below their MDCs, only a summary of the MDCs can be provided. The measurement results reported by SRNL are listed, and the results of this analysis are reported. The data were generally found to follow a normal distribution, and to be homogenous across composite samples.
Sampling has been completed for the characterization of the residual material on the floor of Tank 5 in the F-Area Tank Farm at the Savannah River Site (SRS), near Aiken, SC. The sampling was performed by Savannah River Remediation (SRR) LLC using a stratified random sampling plan with volume-proportional compositing. The plan consisted of partitioning the residual material on the floor of Tank 5 into three non-overlapping strata: two strata enclosed accumulations, and a third stratum consisted of a thin layer of material outside the regions of the two accumulations. Each of three composite samples was constructed from five primary sample locations of residual material on the floor of Tank 5. Three of the primary samples were obtained from the stratum containing the thin layer of material, and one primary sample was obtained from each of the two strata containing an accumulation. This report documents the statistical analyses of the analytical results for the composite samples. The objective of the analysis is to determine the mean concentrations and upper 95% confidence (UCL95) bounds for the mean concentrations for a set of analytes in the tank residuals. The statistical procedures employed in the analyses were consistent with the Environmental Protection Agency (EPA) technical guidance by Singh and others [2010]. Savannah River National Laboratory (SRNL) measured the sample bulk density, nonvolatile beta, gross alpha, radionuclide, inorganic, and anion concentrations three times for each of the composite samples. The analyte concentration data were partitioned into three separate groups for further analysis: analytes with every measurement above their minimum detectable concentrations (MDCs), analytes with no measurements above their MDCs, and analytes with a mixture of some measurement results above and below their MDCs. The means, standard deviations, and UCL95s were computed for the analytes in the two groups that had at least some measurements above their MDCs. The identification of distributions and the selection of UCL95 procedures generally followed the protocol in Singh, Armbya, and Singh [2010]. When all of an analyte's measurements lie below their MDCs, only a summary of the MDCs can be provided. The measurement results reported by SRNL are listed in Appendix A, and the results of this analysis are reported in Appendix B. The data were generally found to follow a normal distribution, and to be homogeneous across composite samples.
There has been considerable discussion within the national security community, including a recent workshop sponsored by the U.S. State Department, about the use of social media for extracting patterns of collective behavior and influencing public perception in areas relevant to arms control and nonproliferation. This paper seeks to explore if, and how, social media can be used to supplement nonproliferation and arms control inspection and monitoring activities on states and sites of greatest proliferation relevance. In this paper, we set the stage for how social media can be applied in this problem space and describe some of the foreseen challenges, including data validation, sources and attributes, verification, and security. Using information analytics and data visualization capabilities available at Pacific Northwest National Laboratory (PNNL), we provide graphical examples of some social media "signatures" of potential relevance for nonproliferation and arms control purposes. We conclude by describing a proposed case study and offering recommendations both for further research and next steps by the policy community.
Because of the vast volume of data being produced by today’s scientific simulations and experiments, lossy data compressor allowing user-controlled loss of accuracy during the compression is a relevant solution for significantly reducing the data size. However, lossy compressor developers and users are missing a tool to explore the features of scientific data sets and understand the data alteration after compression in a systematic and reliable way. To address this gap, we have designed and implemented a generic framework called Z-checker. On the one hand, Z-checker combines a battery of data analysis components for data compression. On the other hand, Z-checker is implemented as an open-source community tool to which users and developers can contribute and add new analysis components based on their additional analysis demands. In this article, we present a survey of existing lossy compressors. Then, we describe the design framework of Z-checker, in which we integrated evaluation metrics proposed in prior work as well as other analysis tools. Specifically, for lossy compressor developers, Z-checker can be used to characterize critical properties (such as entropy, distribution, power spectrum, principal component analysis, and autocorrelation) of any data set to improve compression strategies. For lossy compression users, Z-checker can detect the compression quality (compression ratio and bit rate) and provide various global distortion analysis comparing the original data with the decompressed data (peak signal-to-noise ratio, normalized mean squared error, rate–distortion, rate-compression error, spectral, distribution, and derivatives) and statistical analysis of the compression error (maximum, minimum, and average error; autocorrelation; and distribution of errors). Z-checker can perform the analysis with either coarse granularity (throughout the whole data set) or fine granularity (by user-defined blocks), such that the users and developers can select the best fit, adaptive compressors for different parts of the data set. Z-checker features a visualization interface displaying all analysis results in addition to some basic views of the data sets such as time series. To the best of our knowledge, Z-checker is the first tool designed to assess lossy compression comprehensively for scientific data sets.
Data-driven analytical clustering and visualization techniques were applied to the dataset of 9% Cr experimental alloy data generated through the eXtremeMAT project. Techniques and results were compared with the resulting clusters obtained through similar analytical techniques on previous and reduced versions of the dataset. The principal components were generated in order to reduce the dimensionality of the complex dataset and to visualize the underlying trends in the data. Partitioning around medoids was performed on the resulting principal components to determine relevant clusters. Domain knowledge labels were further applied to the principal components to compare the labels with the trends identified through the clustering methods. The resulting clusters show the addition of the new data into the dataset, and show where alignment between the domain knowledge labels and partitioning around medoids (PAM) clusters show where the labels are appropriate. The clusters can be used to compare the tensile properties of the alloys, and to reduce the variation in the dataset.
A nonrelational, distributed computing, data warehouse, and analytics environment (Energy-CRADLE) was developed for the analysis of field and laboratory data from multiple heterogeneous photovoltaic (PV) test sites. This data informatics and analytics infrastructure was designed to process diverse formats of PV performance data and climatic telemetry time-series data collected from a PV outdoor test network, i.e., the Solar Durability and Lifetime Extension global SunFarm network, as well as point-in-time laboratory spectral and image measurements of PV material samples. Using Hadoop/HBase for the distributed data warehouse, Energy-CRADLE does not have a predefined data table schema, which enables ingestion of data in diverse and changing formats. For easy data ingestion and data retrieval, Energy-CRADLE utilizes Hadoop streaming to enable Python MapReduce and provides a graphical user interface, i.e., py-CRADLE. By developing the Hadoop distributed computing platform and the HBase NoSQL database schema for solar energy, Energy-CRADLE exemplifies an integrated, scalable, secure, and user-friendly data informatics and analytics system for PV researchers. Furthermore, an example of Energy-CRADLE enabled scalable, data-driven, analytics is presented, where machine learning is used for anomaly detection across 2.2 million real-world current-voltage (I–V) curves of PV modules in three distinct Koppen–Geiger climatic zones.
A new method for detecting and discriminating pure gases and binary mixtures has been investigated. This approach combines two distinct physical mechanisms within a single piezoresistive microcantilever: heat dissipation and resonant damping in the viscous regime. An experimental study of the heat dissipation mechanism indicates that the sensor response is directly correlated to the thermal conductivity of the gaseous analyte. A theoretical data set of resonant damping was generated corresponding to the gas mixtures examined in the thermal response experiments. The combination of the thermal and resonant response data yields more distinct analyte signatures that cannot otherwise be obtained from the detection modes individually.
The Federal Radiological Monitoring and Assessment Center (FRMAC) relies on accurate and defensible analytical laboratory data to support its mission. Therefore, FRMAC must ensure that the environmental analytical laboratories providing analytical services maintain an ongoing capability to provide accurate analytical results to DOE. It is undeniable that the more Quality Assurance (QA) and Quality Control (QC) measures required of the laboratory, the less resources that are available for analysis of response samples. Being that QA and QC measures in general are understood to comprise a major effort related to a laboratory’s operations, requirements should only be considered if they are deemed “value-added” for the FRMAC mission. This report provides observations of areas for improvement and potential interoperability opportunities in the areas of Batch Quality Control Requirements, Written Communications, Data Review Processes, Data Reporting Processes, along with the lessons learned as they apply to items in the early phase of a response that will be critical for developing a more efficient, integrated response for future interactions between the FRMAC and EPA assets.
Functional data analysis (FDA), which is a branch of statistics on modeling infinite dimensional random vectors resided in functional spaces, has become a major research area for Journal of Multivariate Analysis. We review some fundamental concepts of FDA, their origins and connections from multivariate analysis, and some of its recent developments, including multi-level functional data analysis, high-dimensional functional regression, and dependent functional data analysis. Here, we also discuss the impact of these new methodology developments on genetics, plant science, wearable device data analysis, image data analysis, and business analytics. Two real data examples are provided to motivate our discussions.
We discuss and analyze four magic-angle spinning solid-state NMR methods that can be used to measure internuclear distances and to obtain correlation spectra between a spin I = 1/2 and a half-integer spin S > 1/2 having a small quadrupolar coupling constant. Three of the methods are based on the heteronuclear multiple-quantum and single-quantum correlation experiments, that is, high rank tensors that involve the half spin and the quadrupolar spin are generated. Here, both zero and single-quantum coherence of the half spins are allowed and various coherence orders of the quadrupolar spin are generated, and filtered, via active recoupling of the dipolar interaction. As a result of generating coherence orders larger than one, the spectral resolution for the quadrupolar nucleus increases linearly with the coherence order. Since the formation of high rank tensors is independent of the existence of a finite quadrupolar interaction, these experiments are also suitable to materials in which there is high symmetry around the quadrupolar spin. A fourth experiment is based on the initial quadrupolar-driven excitation of symmetric high order coherences (up to p = 2S, where S is the spin number) and subsequently generating by the heteronuclear dipolar interaction higher rank (l + 1 or higher) tensors that involve also the half spins. Due to the nature of this technique, it also provides information on the relative orientations of the quadrupolar and dipolar interaction tensors. For the ideal case in which the pulses are sufficiently strong with respect to other interactions, we derive analytical expressions for all experiments as well as for the transferred echo double resonance experiment involving a quadrupolar spin. We show by comparison of the fitting of simulations and the analytical expressions to experimental data that the analytical expressions are sufficiently accurate to provide experimental {sup 7}Li–{sup 13}C distances in a complex of lithium, glycine, and water. Discussion of the regime for which such an approach is valid is given.
This project is to develop a data management and analytics platform called Grid Data Crossing (GriD-Xing) that integrates data processing, storage, analysis, and visualization functionalities for users at industry (e.g., EMS and DMS) and universities. It intends to increase the applicability of large set of multi-source data (e.g., SCADA, PMU, μPMU, AMI, inverter, load/DER forecasting, and cyber data) and eventually improve the resilience, reliability, and security of critical energy infrastructure. GriD-Xing also intends to be a research platform to identify the usefulness of large multi-source data via case studies of interest to industry (e.g., 2011 southwest cold weather and 2016 Blue Cut fire events). In the first year, the team focused on the development and demonstration of large multi-source data (e.g., PMU, μPMU, SCADA, AMI, and cyber data) processing, storage, and analysis. Specifically, the team designed GriD-Xing framework in functional and structural aspects, developed GriD-Xing algorithms for large multi-source grid data processing, storage, and analysis, and finally demonstrated GriD-Xing prototype algorithms performance through a series of tests and case studies using real-world and simulation data.
A system and method of integrating diverse sources of data and data streams is presented. The method can include selecting a scenario based on a topic, creating a multi-relational directed graph based on the scenario, identifying and converting resources in accordance with the scenario and updating the multi-directed graph based on the resources, identifying data feeds in accordance with the scenario and updating the multi-directed graph based on the data feeds, identifying analytical routines in accordance with the scenario and updating the multi-directed graph using the analytical routines and identifying data outputs in accordance with the scenario and defining queries to produce the data outputs from the multi-directed graph.