SEARCH · Search NASA
Results for “deception detection”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Leveraging Community and Author Context to Explain the Performance and Bias of Text-Based Deception Detection Models
Deceptive news posts shared in online communities can be detected with NLP models, and much recent research has focused on the development of such models. In this work, we use characteristics of online communities and authors --- the context of how and where content is posted --- to explain the performance of a neural network deception detection model and identify sub-populations who are disproportionately affected by model accuracy or failure. We examine who is posting the content, and where the content is posted to. We find that while author characteristics are better predictors of deceptive content than community characteristics, both characteristics are strongly correlated with model performance. Traditional performance metrics such as F1 score may fail to capture poor model performance on isolated sub-populations such as specific authors, and as such, more nuanced evaluation of deception detection models is critical.
Evaluating Deception Detection Model Robustness To Linguistic Variation
With the increasing use of automated, machine learning-driven tools and the downstream impact that algorithmic judgements can have, it is critical to develop models that are robust to evolving or manipulated inputs. Evaluating the reliability of multimodal models across linguistic variations to understand model susceptibility to intentional linguistic adversarial attacks as well as natural linguistic variations is essential in this pursuit. We present extensive analysis of model robustness and susceptibility to linguistic variations in the setting of deceptive news detection, a difficult classification task that is an increasingly important problem to solve with the impact of misinformation spread online. We evaluate the effectiveness of incorporating adversarial defense strategies and measure model susceptibility to state-of-the-art adversarial attacks using two types of linguistic attacks — character and word perturbations. We consider two multiclass prediction tasks — a 3-way classification of tweets as trustworthy, propaganda, or disinformation; and a 4-way classification as clickbait, hoax, satire, or conspiracy — and compare the performance of three embeddings that have been state-of-the-art for several NLP tasks — GloVe, ELMo, and BERT — to highlight consistent trends in susceptibility, high confidence misclassifications, and high impact failures. We find that character or mixed ensemble models are the most effective defense mechanisms and that character perturbations are a more effective attack than word perturbations for deception classification.
Reading Between the Lines: Measuring the Effects of Linguistic-Based Indicators of Deception on Experts’ Identification and Categorization of Disinformation
There is currently very limited research into how experts analyze and assess potentially fraudulent content in their expertise areas, and most research within the disinformation space involves very limited text samples (e.g., news headlines). The overarching goal of the present study was to explore how an individual’s psychological profile and the linguistic features in text might influence an expert’s ability to discern disinformation/fraudulent content in academic journal articles. At a high level, the current design tasked experts with reading journal articles from their area of expertise and indicating if they thought an article was deceptive or not. Half the articles they read were journal papers that had been retracted due to academic fraud. Demographic and psychological inventory data collected on the participants was combined with performance data to generate insights about individual expert susceptibility to deception. Our data show that our population of experts were unable to reliably detect deception in formal technical writing. Several psychological dimensions such as comfort with uncertainty and intellectual humility may provide some protection against deception. This work informs our understanding of expert susceptibility to potentially fraudulent content within official, technical information and can be used to inform future mitigative efforts and provide a building block for future disinformation work.
Misleading or Falsification? Inferring Deceptive Strategies and Types in Online News and Social Media
Deceptive information in online news and social media has had dramatic effect on our society in recent years. This study is the first to gain deeper insights into writers' intent behind digital misinformation by analyzing psycholinguistic signals: moral foundations and connotations extracted from different types of deceptive news ranging from strategic disinformation to propaganda and hoaxes. To ensure consistency of our findings and generalizability across domains, we experiment with data from: (1) confirmed cases of disinformation in news summaries, (2) {propaganda}, hoax, and disinformation news pages, and (3) social media news. We first contrast lexical markers of biased language, syntactic and stylistic signals, and connotations across deceptive news types including disinformation, propaganda, and hoaxes, and {deceptive} strategies including misleading or falsification. We then incorporate these insights to build machine learning and deep learning predictive models to infer deception strategies and deceptive news types. Our experimental results demonstrate that unlike earlier work on deception detection, content combined with biased language markers, moral foundations, and connotations leads to better predictive performance of deception strategies compared to syntactic and stylistic signals (as reported in earlier work on deceptive reviews). Falsification strategy is easier to identify than misleading strategy. Disinformation is more difficult to predict than to propaganda or hoaxes. Deceptive news types (disinformation, propaganda, and hoaxes), unlike deceptive strategies (falsification and misleading), are more salient, and thus easier to identify in tweets than in news reports. Finally, our novel connotation analysis across deception types provides deeper understanding of writers' perspectives and therefore reveals the intentions behind digital misinformation.
Separating Facts from Fiction: Linguistic Models to Classify Suspicious and Trusted News Posts on Twitter
Pew research polls report 62 percent of U.S. adults get news on social media (Gottfried and Shearer, 2016). In a December poll, 64 percent of U.S. adults said that “made-up news” has caused a “great deal of confusion” about the facts of current events (Barthel et al., 2016). Fabricated stories spread in social media, ranging from deliberate propaganda to hoaxes and satire, contributes to this confusion in addition to having serious effects on global stability. In this work we build predictive models to classify 130 thousand news tweets as suspicious or verified, and predict four subtypes of suspicious news – satire, hoaxes, clickbait and propaganda. We demonstrate that neural network models trained on tweet content and social network interactions outperform lexical models. Unlike previous work on deception detection, we find that adding syntax and grammar features to our models decreases performance. Incorporating linguistic features, including bias and subjectivity, improves classification results, however social interaction features are most informative for finer-grained separation between our four types of suspicious news posts.
Discerning Deception: An Empirically-Driven Agent-Based Model of Expert Evaluation of Scientific Content
Both human subject experiments and computational, modeling and simulations have been used to study detection of deception. This work aims to combine these two methods by integrating empirically-derived information (from human subject experiments) into agent-based models to generate novel insights into the complex problems of detection of disinformation content. Computational experiments are used to simulate across multiple scenarios for evaluation and decision-making regarding the validity of potentially deceptive scientific documents. Factors influencing the human agent behaviors in the model were identified through a human subject experiment that was conducted to evaluate and characterize decision making related to disinformation discernment. Correlation and regression analyses were used to translate insights from the human subjects experiment to inform the parameterization of agent features and scenario development. Three scenarios were evaluated with the agent-based models to help evaluate the replicability of the simulations (validation analysis) and assess the influence of human agent and document features (sensitivity analyses). A replication of the human participant experiment demonstrated that the agent-based simulations compare favorably to empirical findings. The agent-based modeling was then used to conduct sensitivity analysis on the accuracy of deception detection as a function of document proportions and human agent features. Results indicate that precision values are adversely impacted when the proportion of deceptive documents is lower in the overall sample, whereas recall values are more sensitive to changes in human agent features. These findings indicate important nuances in accuracy evaluations that should be further considered (including consideration of potential alternate metrics) in future agent-based models of disinformation. Additional areas for future exploration include extension of simulations to consider other ways to align the agent-based model design with psychological theory and inclusion of agent-agent interactions, especially as it pertains to sharing of scientific information within an organizational context.
Use of Deception to Improve Client Honeypot Detection of Drive-by-Download Attacks
This paper presents the application of deception theory to improve the success of client honeypots at detecting malicious web page attacks from infected servers programmed by online criminals to launch drive-by-download attacks. The design of honeypots faces three main challenges: deception, how to design honeypots that seem real systems; counter-deception, techniques used to identify honeypots and hence defeating their deceiving nature; and counter counter-deception, how to design honeypots that deceive attackers. The authors propose the application of a deception model known as the deception planning loop to identify the current status on honeypot research, development and deployment. The analysis leads to a proposal to formulate a landscape of the honeypot research and planning of steps ahead.
Spy the Lie: Detecting Malicious Insiders
Insider threat is a hard problem. There is no ground truth, there are innumerable variables, and the data is sparse. The types of crimes and abuses associated with insider threats are significant; the most serious include espionage, sabotage, terrorism, embezzlement, extortion, bribery, and corruption. Malicious activities include an even broader range of exploits, such as negligent use of classified data, fraud, cybercrime, unauthorized access to sensitive information, and illicit communications with unauthorized recipients. Inadvertent action or inaction without malicious intent (e.g., disposing of sensitive documents incorrectly) can also cause harm to an organization. This review article will explore insider threat, specifically behaviors, beliefs, and current debates within the field. Additionally particular focus is given to deception, a significant behavioral component of the malicious insider. Finally, research and policy implications for law enforcement and the intelligence community are addressed.
Increased Interpretability for Model-Driven Deception: MARS LDRD Project
Machine learning has been proposed as a solution to several cybersecurity solutions and one of the most promising applications is for digital twins for intrusion detection and driving deceptive defense. However, machine learning techniques often result in a black-box function that is difficult for end users to interpret which for deception limits their ability to effectively define decoys. In this report, an approach to validate the equations learned are accurate is provided and demonstrated. Following, begins the process of addressing this issue for a model-driven deception technology that produces equations representing the physical process controlled by operation technology devices. This research was performed by applying subject matter expert context to machine learned models.
Best Paper Award: Gathering Threat Intelligence Through Computer Network Deception
Presented at the 2016 IEEE International Symposium on Technologies for Homeland Security (HST), this LDRD-funded research received a Best Paper award, the highest award presented at the conference. The research focused on developing, implementing, and testing a novel computer network operations architecture that enables proactive defense by managing and monitoring enterprise resource allocations and network flows. The architecture leverages three emerging concepts: software-defined networks, cloud computing, and deception, to enable the detection and identification of anomalous access and intrusions to adjust to the dynamic nature of the adversary and to provide a mechanism to discover and react to the adversary’s attacks in a methodical and proactive manner. Additionally, the work allows network defenders to gather information on an adversary’s tools, tactics and procedures, providing insight into the “what, why, and how” of the adversary’s operations. A summary of the work presented at IEEE can be found at http://tiny.sandia.gov/8d8lu.
Machine Learning for Power System Disturbance and Cyber-attack Discrimination
Power system disturbances are inherently complex and can be attributed to a wide range of sources, including both natural and man-made events. Currently, the power system operators are heavily relied on to make decisions regarding the causes of experienced disturbances and the appropriate course of action as a response. In the case of cyber-attacks against a power system, human judgment is less certain since there is an overt attempt to disguise the attack and deceive the operators as to the true state of the system. To enable the human decision maker, we explore the viability of machine learning as a means for discriminating types of power system disturbances, and focus specifically on detecting cyber-attacks where deception is a core tenet of the event. We evaluate various machine learning methods as disturbance discriminators and discuss the practical implications for deploying machine learning systems as an enhancement to existing power system architectures.
Multifractal Characterization of Distribution Synchrophasors for Cybersecurity Defense of Smart Grids
“Source ID Mix” spoofing emerged as a new type of cyber-attack on Distribution Synchrophasors (DS) where adversaries have the capability to swap the source information of DS without changing the measurement values. Accurate detection of such a highly-deceptive attack is a challenging task especially when the spoofing attack happens on short fragments of DS recorded within a relatively small geographical scale. Herein this letter proposes an effective approach to detect this cyber-attack by realizing the multifractal characteristics of DS measurements. First, the multifractal cross-correlation of DS measured at multiple intra-state locations is revealed. Then the derived correlation is integrated with weighted two-dimensional multifractal surface interpolation to reconstruct quasi high-resolution signals. Finally, informative location-specific signatures are extracted from the high-resolution DS and they are integrated with advanced machine learning techniques for source authentication. Experiments using the real-life DS are performed to verify the proposed method.
Deception-Based Cyber Attacks on Hierarchical Control Systems using Domain-Aware Koopman Learning
Industrial control systems are subject to cyber attacks that produce physical consequences. These attacks can be both hard to detect and protracted. Here, we focus on deception-based sensor bias attacks made against a hierarchical control system where the attacker attempts to be stealthy. We develop a a data-driven, optimization-based attacker model and use the Koopman operator to represent the system dynamics in a domain-aware and computationally efficient manner. Using this model, we compute several different attacks against a high-fidelity commercial building emulator and compare the impacts of those attacks to each other. Finally, we discuss some computational considerations and identify avenues for future research.
Model Driven Deception for Defense of Operational Technology Environments
Due to the strong integration of real-world physics, OT deception platforms must operate differently than traditional IT deceptions. For instance, turning off a valve will be detected downstream by other sensors because the flow will reduce and stop. Additionally, controllers and applications leverage data from sensors to send control commands to each other. A believable deception must be integrated with the system to project the effects of events. An attack will likely attempt to control the physical process in a negative manner. To make the attacker believe they are achieving their objective, it must predict the effects of these actions, to a reasonable degree. Our approach to simulating a model to generate realistic decoy behavior is explored including description of two approaches: a physics model-based approach and a data driven approach. The performance of two machine learning techniques are investigated in their ability to learn a good enough model of the physics of the system.
Metagames and Hypergames for Deception-Robust Control
Cyber-physical systems (CPSs) consist of computing and communication devices integrated with physical components such as sensors and actuators. Increasing connectivity to the Internet for remote monitoring and control has made CPSs more vulnerable to deliberate attacks, which are distinctly different from random perturbations in the system. This provides a way for purely cyber attacks to have physical consequences. Stuxnet is a prominent example of such an attack, one in which the malware acted over an extended period of time while deliberately remaining undetected. Such attacks can be described as Advanced Persistent Threats (APTs) -- long-term, stealthy attacks. Here, we extend our previous work on hypergames to develop defender strategies that are robust to deception and do not rely on attack detection. We prove that the defender can bound the attacker payoff with these strategies even when the attacker can choose between different attack modes, and we numerically demonstrate our approach on a realistic building control system. Finally, we discuss next steps in extending this approach towards an operational capability.
Development of a Simple Field Test for Vehicle Exhaust to Detect Illicit Use of Dyed Diesel Fuel
The use of tax-free dyed fuel on public highways in the United States provides a convenient way of evading taxes. Current enforcement involves visual inspection for the red azo dye added to the fuel to designate its tax-free status. This approach has shortcomings such as the invasive nature of the tests and/or various deceptive tactics applied by tax evaders. A test designed to detect dyed fuel use by analyzing the exhaust would circumvent these shortcomings. This paper describes the development of a simple color spot test designed to detect the use of tax-free (dyed) diesel fuel by analyzing the engine exhaust. Development first investigated the combustion products of C.I. Solvent Red 164 (the azo dye formulation used in the United States to tag tax-free fuel). A variety of aryl amines were identified as characteristic molecular remnants that appear to survive combustion. A number of microanalytical color tests specific for aryl amines were then investigated. One test based on the use of 4-(dimethylamino)benzaldehyde seemed particularly applicable and was used in a proof-of-principle experiment. The 4-(dimethylamino)benzaldehyde color spot test was able to clearly distinguish between engines burning regular and dyed diesel fuel. Further development will refine this color spot test to provide an easy-to-use field test for Internal Revenue Service Field Compliance specialists.
Machine Intelligence to Detect, Characterise, and Defend against Influence Operations in the Information Environment
Social media has enabled a new era of manipulation in the information and cognitive domains. Deceptive content—misleading, falsified, and fabricated—is routinely created and spread in the modern social media environment with the intent to create confusion and widen political and social divides, and exploit the societal conflict exacerbated by these divides in the real-world (aka physical domain). Such disinformation campaigns demonstrate a threat to the integrity of economic, political, cultural, public health, and national security institutions around the world. In this work we overview our artificial intelligence (AI) capabilities to detect, describe, and defend against information operations on Twitter as an example social platform to understand the influence of misleading and falsified content diffusion and better enable those charged with defending against such manipulation to enable responsive parties to counter it. We first present novel linguistically-informed deep learning (DL) models for misinformation and disinformation detection, and present an in-depth linguistic analysis of psycho-linguistic markers across broad deception categories. We then demonstrate how our models perform in the multilingual and multimodal setting and categorize falsified and misleading content based on the intent to deceive. We also provide a large-scale analysis to describe user behavior and spread patterns while engaging with deceptive content and report novel findings about the immediate diffusion of deceptive content by characterizing the vulnerable sub-populations and their demographics, and explicitly measuring speed and scale of deception spread to uncover who shares deceptive content, how quickly, how much, and how evenly. In addition, we measure audience reactions to misinformation and disinformation at scale, distinguishing the reactions of users identified as bots versus humans. Finally, we take advantage of deep translation and generation models to create unique solutions for real-time defense against digital deception and discuss how to apply causal inference to prescribe and intervene into strategic communications jointly across information, cognitive, and physical domains.