Search NASA⌕ Search

Engineering topics

Saldanha, Emily G.

Publications and source records attributed to Saldanha, Emily G..

Evaluating uncertainty-based active learning for accelerating the generalization of molecular property prediction

Deep learning models have proven to be a powerful tool for the prediction of molecular properties for applications including drug design and the development of energy storage materials. However, in order to learn accurate and robust structure–property mappings, these models require large amounts of data which can be a challenge to collect given the time and resource-intensive nature of experimental material characterization efforts. Additionally, such models fail to generalize to new types of molecular structures that were not included in the model training data. The acceleration of material development through uncertainty-guided experimental design has the promise to significantly reduce the data requirements and enable faster generalization to new types of materials. To evaluate the potential of such approaches for electrolyte design applications, we perform comprehensive evaluation of existing uncertainty quantification methods on the prediction of two relevant molecular properties - aqueous solubility and redox potential. We develop novel evaluation methods to probe the utility of the uncertainty estimates for both in-domain and out-of-domain data sets. Finally, we leverage selected uncertainty estimation methods for active learning to evaluate their capacity to support experimental design.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Machine learning assisted phase and size-controlled synthesis of iron oxide particles

Synthesis of iron oxides with specific phases and particle sizes is a crucial challenge in various fields, including materials science, energy storage, biomedical applications, environmental science, and earth science. However, despite significant advances in this area, much of the current palette of particle outcomes has been based on time-consuming trial-and-error exploration of synthesis conditions. The present study was designed to explore a very different approach to 1) predict the outcome of synthesis from specified reaction parameters based on using machine learning (ML) techniques, and 2) correlate sets of parameters to obtain products with desired outcomes by a newly designed recommendation algorithm. To achieve this, four ML algorithms were tested, namely random forest, logistic regression, support vector machine, and k-nearest neighbor. Among the models, random forest outperformed the others, attaining 96% and 81% accuracy when predicting the phase and size of iron oxide particles in the test dataset. Surprisingly, the permutation feature importance analysis revealed that volume, which may strongly relate to pressure, was one of the important features, along with precursor concentration, pH, temperature, and time, influencing the phase and size of iron oxide particles during synthesis. To verify the robustness of the random forest models, prediction and experimental results were compared based on 24 randomly generated methods in additive and non-additive systems not included in the datasets. The predictions of product phase and particle size from the models agreed well with the experimental results. Furthermore, a searching and ranking algorithm was developed to recommend potential synthesis parameters for obtaining iron oxide products with the desired phase and particle size from previous studies in the dataset. Furthermore, this study lays the foundation for a closed-loop approach in materials synthesis and preparation, beginning with suggesting potential reaction parameters from the dataset and predicting potential outcomes, followed by conducting experiments and analyses, and ultimately enriching the dataset.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

SOMAS: a platform for data-driven material discovery in redox flow battery development

Abstract Aqueous organic redox flow batteries offer an environmentally benign, tunable, and safe route to large-scale energy storage. The energy density is one of the key performance parameters of organic redox flow batteries, which critically depends on the solubility of the redox-active molecule in water. Prediction of aqueous solubility remains a challenge in chemistry. Recently, machine learning models have been developed for molecular properties prediction in chemistry and material science. The fidelity of a machine learning model critically depends on the diversity, accuracy, and abundancy of the training datasets. We build a comprehensive open access organic molecular database “Solubility of Organic Molecules in Aqueous Solution” (SOMAS) containing about 12,000 molecules that covers wider chemical and solubility regimes suitable for aqueous organic redox flow battery development efforts. In addition to experimental solubility, we also provide eight distinctive quantum descriptors including optimized geometry derived from high-throughput density functional theory calculations along with six molecular descriptors for each molecule. SOMAS builds a critical foundation for future efforts in artificial intelligence-based solubility prediction models.

25 ENERGY STORAGE↗

Learning Global Proliferation Expertise Evolution Using AI-Driven Analytics and Public Information

Detecting and anticipating global proliferation expertise and capability evolution from unstructured, noisy, and incomplete public data streams is a highly desired, but extremely challenging task. Here, in this article, we present our pioneering data-driven approach to support the non-proliferation mission to detect and explain the evolution of proliferation expertise and capability development globally from terabytes of publicly available information (PAI), focusing on our knowledge extraction pipeline and descriptive analytics. We first discuss how we fuse nine open-source data streams, including multilingual data, to convert 4 TB of unstructured data to structured knowledge and encode dynamically evolving proliferation expertise representations—content and context graphs. For this, we rely on natural language processing (NLP) and deep learning (DL) models to perform information extraction, topic modeling, and distributed text representation (aka embedding) learning. We then present interactive, usable, and explainable descriptive analytics to refine domain knowledge and present it in a human-understandable form. Finally, we introduce future work avenues that will leverage our dynamic knowledge representations and descriptive analytics to enable predictive and prescriptive inferences to achieve real-time domain understanding and contextual reasoning about global proliferation expertise and capability evolution.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Extracting Material Property Measurement Data from Scientific Articles

Machine learning-based prediction of material properties is often hampered by the lack of sufficiently large training datasets. The majority of such measurement data is embedded in scientific literature and the ability to automatically extract these data is essential to support the development of reliable property prediction methods. In this work, we describe a methodology for an automatic property extraction framework using material solubility as the target property. We create an annotated dataset containing tags for solubility-related entities using a combination of regular expressions and manual tagging. We then compare five entity recognition models leveraging both token-level and span-level architectures on the task of classifying solute names, solubility values, and solubility units. Additionally, we explore a novel pretraining approach that leverages automated chemical name and quantity extraction tools to generate large datasets that do not rely on intensive manual effort. Finally, we perform an analysis to identify the causes of classification errors.

Panapitiya, Gihan U.↗

VAINE: Visualization and AI for Natural Experiments

Natural experiments are observational studies where the assignment of treatment conditions to different populations occur by chance ``in the wild''. Researchers from fields such as economics, healthcare, and the social sciences leverage natural experiments to conduct hypothesis testing and causal effect estimation for treatment and outcome variables that would otherwise be costly, infeasible, or unethical. In this paper, we introduce VAINE (Visualization and AI for Natural Experiments), a visual analytics tool for identifying and understanding natural experiments from observational data. We then demonstrate how VAINE can be used to validate causal relationships, estimate average treatment effects, and identify statistical phenomena such as Simpson’s paradox through two use cases.

Guo, Grace↗

Explaining and predicting human behavior and social dynamics in simulated virtual worlds: reproducibility, generalizability, and robustness of causal discovery methods

Ground Truth program was designed to evaluate social science modeling approaches using simulation test beds with ground truth intentionally and systematically embedded to understand and model complex Human Domain systems and their dynamics Lazer et al. (Science 369:1060–1062, 2020). Our multidisciplinary team of data scientists, statisticians, experts in Artificial Intelligence (AI) and visual analytics had a unique role on the program to investigate accuracy, reproducibility, generalizability, and robustness of the state-of-the-art (SOTA) causal structure learning approaches applied to fully observed and sampled simulated data across virtual worlds. In addition, we analyzed the feasibility of using machine learning models to predict future social behavior with and without causal knowledge explicitly embedded. In this paper, we first present our causal modeling approach to discover the causal structure of four virtual worlds produced by the simulation teams—Urban Life, Financial Governance, Disaster and Geopolitical Conflict. Our approach adapts the state-of-the-art causal discovery (including ensemble models), machine learning, data analytics, and visualization techniques to allow a human-machine team to reverse-engineer the true causal relations from sampled and fully observed data. We next present our reproducibility analysis of two research methods team’s performance using a range of causal discovery models applied to both sampled and fully observed data, and analyze their effectiveness and limitations. We further investigate the generalizability and robustness to sampling of the SOTA causal discovery approaches on additional simulated datasets with known ground truth. Our results reveal the limitations of existing causal modeling approaches when applied to large-scale, noisy, high-dimensional data with unobserved variables and unknown relationships between them. We show that the SOTA causal models explored in our experiments are not designed to take advantage from vasts amounts of data and have difficulty recovering ground truth when latent confounders are present; they do not generalize well across simulation scenarios and are not robust to sampling; they are vulnerable to data and modeling assumptions, and therefore, the results are hard to reproduce. Finally, when we outline lessons learned and provide recommendations to improve models for causal discovery and prediction of human social behavior from observational data, we highlight the importance of learning data to knowledge representations or transformations to improve causal discovery and describe the benefit of causal feature selection for predictive and prescriptive modeling.

97 MATHEMATICS AND COMPUTING↗

Evaluating and Explaining Natural Language Generation with GenX

Current methods for evaluation of natural language generation models focus on measuring text quality but fail to probe the model creativity, i.e., its ability to generate novel but coherent text sequences not seen in the training corpus. We present the GenX tool which is designed to enable interactive exploration and explanation of natural language generation outputs with a focus on the detection of memorization. We demonstrate the utility of the tool on two domain-conditioned generation use cases - phishing emails and ACL abstracts.

Duskin, Kayla R.↗

Studying information recurrence, gatekeeping, and the role of communities during internet outages in Venezuela

Many authoritarian regimes have taken to censoring internet access in order to stop the spread of misinformation, restrict citizens from discussing certain topics, and prevent mobilization, among other reasons. There are several theories about the effectiveness of censorship. Some suggest that censorship will effectively limit the flow of information, whereas others predict that a backlash will form, resulting in ultimately more discussion about the topic. In this work, we analyze the role of communities and gatekeepers during multiple internet outages in Venezuela in January 2019. First, we measure how critical information (e.g., entities and hashtags) spreads during outages focusing on information recurrence and burstiness within and across language and location communities. We discover that information bursts tend to cross both language and location community boundaries rather than being limited to a single community during several outages. Then we identify users who play central roles and propose a novel method to detect gatekeepers—users who prevent critical information from spreading across communities during outages. We show that bilingual and English-speaking users play more central roles compared to Spanish-speaking users, but users inside and outside Venezuela have similar distribution of centrality. Finally, we measure the differences in social network structure before and after each outage event and discuss its effect on how information spreads. We find that with each outage event social connections tend to get less connected with higher mean shortest path, indicating that the effect of censorship makes it harder for information to spread.

97 MATHEMATICS AND COMPUTING↗

Machine Intelligence to Detect, Characterise, and Defend against Influence Operations in the Information Environment

Social media has enabled a new era of manipulation in the information and cognitive domains. Deceptive content—misleading, falsified, and fabricated—is routinely created and spread in the modern social media environment with the intent to create confusion and widen political and social divides, and exploit the societal conflict exacerbated by these divides in the real-world (aka physical domain). Such disinformation campaigns demonstrate a threat to the integrity of economic, political, cultural, public health, and national security institutions around the world. In this work we overview our artificial intelligence (AI) capabilities to detect, describe, and defend against information operations on Twitter as an example social platform to understand the influence of misleading and falsified content diffusion and better enable those charged with defending against such manipulation to enable responsive parties to counter it. We first present novel linguistically-informed deep learning (DL) models for misinformation and disinformation detection, and present an in-depth linguistic analysis of psycho-linguistic markers across broad deception categories. We then demonstrate how our models perform in the multilingual and multimodal setting and categorize falsified and misleading content based on the intent to deceive. We also provide a large-scale analysis to describe user behavior and spread patterns while engaging with deceptive content and report novel findings about the immediate diffusion of deceptive content by characterizing the vulnerable sub-populations and their demographics, and explicitly measuring speed and scale of deception spread to uncover who shares deceptive content, how quickly, how much, and how evenly. In addition, we measure audience reactions to misinformation and disinformation at scale, distinguishing the reactions of users identified as bots versus humans. Finally, we take advantage of deep translation and generation models to create unique solutions for real-time defense against digital deception and discuss how to apply causal inference to prescribe and intervene into strategic communications jointly across information, cognitive, and physical domains.

artificial intelligence, deep learning, neural lan↗

Understanding and Explicitly Measuring Linguistic and Stylistic Properties of Deception via Generation and Translation

Massive digital disinformation is one of the main risks of modern society. Hundreds of models and linguistic analyses have been done to compare and contrast misleading and credible content online. However, most models do not remove the confounding factor of a topic or narrative when training, so the resulting models learn a clear topical separation for misleading versus credible content. We study the feasibility of using two strategies to disentangle the topic bias from the models to understand and explicitly measure linguistic and stylistic properties of content from misleading versus credible content. First, we develop conditional generative models to create news content that is characteristic of different credibility levels. We perform multi-dimensional evaluation of model performance on mimicking both the style and linguistic differences that distinguish news of different credibility using machine translation metrics and classification models. We show that even though generative models are able to imitate both the style and language of the original content, additional conditioning on both the news category and the topic leads to reduced performance. In a second approach, we perform deception style ``transfer" by translating deceptive content into the style of credible content and vice versa. Extending earlier studies, we demonstrate that, when conditioned on a topic, deceptive content is shorter, less readable, more biased, and more subjective than credible content, and transferring the style from deceptive to credible content is more challenging than the opposite direction.

Saldanha, Emily G.↗

Multiple social platforms reveal actionable signals for software vulnerability awareness: A study of GitHub, Twitter and Reddit

Software vulnerabilities are flaws in computer systems that leave users open to attack. In many cases, these vulnerabilities go unnoticed and remain unresolved in codebases. Thus, awareness of software vulnerabilities among the public is crucial to ensure effective cybersecurity practices, the development of high quality software, and ultimately national security. This awareness can be better understood by studying the spread and evolution of software vulnerability discussions in online communities. This work is the first to evaluate and contrast how discussions about software vulnerabilities spread on three social platforms -- Twitter, GitHub, and Reddit. To lay the groundwork, we showcase a novel fundamental framework for measuring information spread that identifies the spread mechanisms and observables across platforms, the units of information, and the groups of measurements that can be applied to focus on a specific phenomena e.g., information cascades. We then analyze and contrast social network topologies for three example social networks and measure the scale and speed of the spread of discussion of specific vulnerabilities to understand how far and how widely they spread, how many users participate in discussions, and the duration of their spread. To demonstrate the awareness of more impactful software vulnerabilities, a subset of our analysis focuses on vulnerabilities targeted during recent major cyber attacks as well as vulnerabilities exploited by advanced persistent threat groups. We discover that usually, vulnerability discussions start on GitHub, before occurring on Twitter and Reddit. While studying how some user-level and content-level characteristics influence vulnerability spread, we observe that Twitter discussions started by users predicted to be humans have larger size, breadth, depth, adoption rate, lifetime, and structural virality compared to those started by users predicted to be bots. On Reddit, we contrast the differences in thread structure that originate from posts with positive, negative and neutral polarity. We find that posts that are positive have larger, deeper and wider discussions compared to negative and neutral posts. We anticipate the results of our analysis to not only increase the understanding of software vulnerability awareness but also inform models for simulating information spread across multiple social environments online.

97 MATHEMATICS AND COMPUTING↗

ReLVis: Visual Analytics for Situational Awareness During Reinforcement Learning Experimentation

Reinforcement learning (RL) is a branch of machine learning where an agent learns to maximize reward through trial and error. RL is challenging and data/compute intensive, and RL practitioners can easily become overwhelmed and make poor modeling decisions. Our contribution is a visual analytics tool designed to help data scientists maintain situation awareness during RL experimentation. Our tool allows the user to under- stand which hyper-parameter values lead to better or worse outcomes, what behaviors are associated with high and low reward, and how behaviors evolve and progress over training time. We evaluated our tool by applying state of the art deep RL models to six different RL tasks. We present three tasks as use cases that show how our tool leads to insightful findings and improves situation awareness.

Saldanha, Emily G.↗

Characterizing Speed and Scale of Cryptocurrency Discussion Spread on Reddit

Cryptocurrencies are a novel and disruptive technology that has prompted a new approach to how currencies work in the modern economy. As such, online discussions related to cryptocurrencies often go beyond posts about the technology and underlying architecture of the various coins, to subjective speculations of price fluctuations and predictions. Furthermore, online discussions, potentially driven by foreign adversaries, criminals or hackers, can have a significant impact on our economy and national security if spread at scale. This paper is the first to qualitatively measure and contrast discussion growth about three popular cryptocurrencies with key distinctions in motivation, usage, and implementation – Bitcoin, Ethereum, and Monero on Reddit. More specifically, we measure how discussions relevant to these coins spread in online social environments – how deep and how wide they go, how long they last, how many people they reach, etc. More importantly, we compare user behavior patterns between the focused community of the official coin subreddits and the general community across Reddit as a whole. Our Reddit sample covers three years of data between 2015 and 2018 and includes a time period of a record high Bitcoin price rise. Our results demonstrate that while the largest discussions on Reddit are focused on Bitcoin, posts about Monero (a cryptocurrency often used by criminals for illegal transactions on the Dark Web) start discussions that are typically longer and wider. Bitcoin posts trigger subsequent discussion more immediately but Monero posts are more likely to trigger a longer lasting discussion. We find that moderately subjective posts across all three coins trigger larger, longer, and more viral discussion cascades within both focused and general communities on Reddit. Our analysis aims to bring the awareness to online discussion spread relevant to cryptocurrencies in addition to informing models for forecasting cryptocurrency price that rely on discussions in social media.

information spread, social network analysis, reddi↗

Evaluation and Validation Approaches for Simulation of Social Behavior: Challenges and Opportunities

While models of social systems have grown in scope to represent increasingly complex phenomena, efforts to validate these models against observations suffer from a number of challenges that include the lack of shared approaches, the use of single-dimensional evaluation criteria, and the difficulty in evaluating causal mechanisms. Moreover, current evaluation approaches suffer from sensitivity of the existing measurements to initial conditions and model assumptions, and failure to account for uncertainty. These effects lead to the lack of generalizability to new contexts, inability to replicate the validation approach for new models, and difficulties in interpretation of validation results. These challenges can be addressed by building a set of diverse and robust evaluation metrics and by developing best practices for comparative, reproducible validation of varied social simulation approaches against real-world data. Herein, we outline several best practices for data-drive model validation, such as defining evaluation strategies consistent with the intended use case and defining broad comparison criteria to highlight both model strengths and weaknesses. Leveraging common validation practices will enable robust analysis of simulation performance across diverse social scenarios and modeling approaches.

Saldanha, Emily G.↗

Towards Rapid Interactive Machine Learning: Evaluating Tradeoffs of Classification without Representation

Our contribution is the design and evaluation of an interac- tive machine learning interface that rapidly provides the user with model feedback after every interaction. To address visual scalability, this interface communicates with the user via a “tip of the iceberg” approach, where the user interacts with a small set of recommended instances for each class. To address computational scalability, we developed an O(n) classification algorithm that incorporates user feedback incrementally, and without consulting the underlying data’s representation matrix. Our computational evaluation showed that this algorithm has similar accuracy to several off-the-shelf classification algo- rithms with small amounts of labeled data. Empirical evalu- ation revealed that users performed better using our design compared to an equivalent active learning setup.

Arendt, Dustin L.↗