Search NASA⌕ Search

SEARCH · Search NASA

Results for “data security”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36

Remote Instrumentation and Data Acquisition: An Internship Research Report

This report outlines the development and implementation of a remote data acquisition system for waveform analysis using a Rohde & Schwarz oscilloscope. The project involved capturing waveform data, and transferring it to a local machine for visualization and analysis. The core logic was developed in C++ with a focus on object oriented programming and the use of polymorphism so the main application can interact with any instrument without knowing its exact type, simplifying the overall logic and making it easier to add or swap out components without changing the rest of the codebase.. The system issues Standard Commands for Programmable Instruments (SCPI) via a socket connection and parses the oscilloscope’s ASCII waveform data. The C++ application was containerized using Docker for ease of portability, and reproducibility. Emphasis was placed on secure networking practices, error handling, and effective data capture. The report describes the technical steps taken, challenges encountered, and lessons learned, providing insight into the practical integration of hardware interfacing with remote computational environments.

Parikh, Jaymil [Fermilab]↗

SoK: What does it Mean to Benchmark Database Forensics?

Relational Database Management Systems are the backbone of modern enterprises and public-sector services, and are thus frequent targets of security incidents, insider threats, and thorough regulatory audits. Consequently, databases have become key sources of digital evidence, requiring investigators to reconstruct past activity from audit logs, transaction logs, and backups. Although benchmarking frameworks such as those developed by the Transaction Processing Performance Council (TPC) are widely used to evaluate database performance, they do not capture forensic requirements such as evidentiary completeness, tamper-evidence, chain of custody, or regulatory compliance under GDPR and CCPA. This survey examines the emerging domain of forensic database benchmarking. We gathered prior research on database forensics, secure logging, and tamper-evident data structures; we analyze modern forensic-ready features in commercial and open-source systems (SQL Server Ledger, Oracle Blockchain Tables, PostgreSQL pgAudit, Db2 Audit, Aurora Database Activity Streams, Oracle Real Application Security and IBM Guardium) and assess why existing benchmarks are insufficient. We propose forensic workloads, metrics, and methodologies that incorporate adversarial stressors, deleted-record recovery, and backup analysis. We also identify open research problems and call for a community-driven forensic benchmark suite. The result is an idea for evaluating not only database performance but also forensic soundness, bridging the gap between system engineering, compliance, and digital investigations.

Lenard, Ben↗

Secure Video Surveillance System (SVSS) for unannounced safeguards inspections.

The Secure Video Surveillance System (SVSS) is a collaborative effort between the U.S. Department of Energy (DOE), Sandia National Laboratories (SNL), and the Brazilian-Argentine Agency for Accounting and Control of Nuclear Materials (ABACC). The joint project addresses specific requirements of redundant surveillance systems installed in two South American nuclear facilities as a tool to support unannounced inspections conducted by ABACC and the International Atomic Energy Agency (IAEA). The surveillance covers the critical time (as much as a few hours) between the notification of an inspection and the access of inspectors to the location in facility where surveillance equipment is installed. ABACC and the IAEA currently use the EURATOM Multiple Optical Surveillance System (EMOSS). This outdated system is no longer available or supported by the manufacturer. The current EMOSS system has met the project objective; however, the lack of available replacement parts and system support has made this system unsustainable and has increased the risk of an inoperable system. A new system that utilizes current technology and is maintainable is required to replace the aging EMOSS system. ABACC intends to replace one of the existing ABACC EMOSS systems by the Secure Video Surveillance System. SVSS utilizes commercial off-the shelf (COTS) technologies for all individual components. Sandia National Laboratories supported the system design for SVSS to meet Safeguards requirements, i.e. tamper indication, data authentication, etc. The SVSS consists of two video surveillance cameras linked securely to a data collection unit. The collection unit is capable of retaining historical surveillance data for at least three hours with picture intervals as short as 1sec. Images in .jpg format are available to inspectors using various software review tools. SNL has delivered two SVSS systems for test and evaluation at the ABACC Safeguards Laboratory. An additional 'proto-type' system remains at SNL for software and hardware testing. This paper will describe the capabilities of the new surveillance system, application and requirements, and the design approach.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Deep Cyber-Physical Situational Awareness for Energy Systems: A Secure Foundation for Next-Generation Energy Management

This document provides the final report for the CYPRES project. The purpose is (1) to highlight and summarize its major accomplishments and (2) to provide guidance on how its outcomes have informed and can inform important additional research and technology transfer. The goal of CYPRES was the research, development, and demonstration of a security-oriented next generation cyber-physical EMS for electric power systems that detects malicious and abnormal events through the fusion of cyber and physical data. To achieve this, the CYPRES project team researched, developed, and built a prototype of the solution, referred to as the CYPRES EMS. The CYPRES EMS is a proof-of-concept cyber-physical platform that demonstrates the management of the energy system, communications, security, and cyber-physical grid modeling and analytics. As part of the capabilities of the CYPRES EMS, the team designed and developed a suite of power system applications for monitoring, risk analyses, detection, and control that are inherently cyberaware. At its core, the project aimed to research, develop, and demonstrate a security-oriented next-generation cyber-physical Energy Management System (EMS) capable of detecting malicious and abnormal events through the innovative fusion of cyber and physical data. This approach represents a fundamental shift from traditional EMS, reimagining how critical infrastructure can be protected through unified cyber-aware and physics-aware secure data flow pipelines. The project’s cornerstone deliverable, the CYPRES EMS, serves as a proof-of-concept cyber-physical platform that revolutionizes the management of energy systems, communications, security, and cyber-physical grid modeling and analytics. This prototype implements a comprehensive suite of power system applications for monitoring, risk analyses, detection, and control, all designed with inherent cyber awareness. The system’s architecture extends from end-devices in the field through to control center applications, establishing a secure and resilient control framework that addresses the challenges posed by diverse devices of unknown trustworthiness connecting to modern power systems. Through this innovative approach to deep cyber-physical situational awareness, the CYPRES project not only advances the state-of-the-art in energy infrastructure protection but also establishes a new paradigm for how EMS can be designed, deployed, and operated in an increasingly complex threat landscape. The findings and developments from this project provide crucial insights for stakeholders across the energy sector, offering a blueprint for enhancing the reliability and resilience of our nation’s critical energy infrastructure in the face of evolving cyber threats.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Calendar Year 2011 Groundwater Monitoring Report, U.S. Department of Energy Y-12 National Security Complex

This report contains the groundwater and surface water monitoring data that were obtained during calendar year (CY) 2011 at the U.S. Department of Energy (DOE) Y-12 National Security Complex (Y-12) on the DOE Oak Ridge Reservation (ORR) in Oak Ridge, Tennessee. The CY 2011 monitoring data were obtained from wells, springs, and surface water sampling locations in three hydrogeologic regimes at Y-12. The Bear Creek Hydrogeologic Regime (Bear Creek Regime) encompasses a section of Bear Creek Valley (BCV) between the west end of Y-12 and the west end of the Bear Creek Watershed (directions are in reference to the Y-12 grid system). The Upper East Fork Poplar Creek Hydrogeologic Regime (East Fork Regime) encompasses the Y-12 industrial facilities and support structures in BCV. The Chestnut Ridge Hydrogeologic Regime (Chestnut Ridge Regime) encompasses a section of Chestnut Ridge directly south of Y-12. This report provides background information pertinent to groundwater and surface water quality monitoring in each hydrogeologic regime, including the topography and bedrock geology, surface water drainage, groundwater system, and known extent of groundwater contamination.

54 ENVIRONMENTAL SCIENCES↗

Los Alamos National Laboratory Input for the National TRU Program 5 Year Plan for National Nuclear Security Administration TRU Waste

On July 16, 2018, Los Alamos National Security (LANS) received a Corporate Board Data Call for Los Alamos National Laboratory (LANL) site input to the National TRU Program 5 Year Plan from the Office of the National TRU Program. This report summarizes LANL input for the National Nuclear Security Administration (NNSA) TRU waste and captures the shipping needs to support the LANL mission. Information is included for the current waste inventory and projected generation rates. The site’s location, characteristics, role and mission are described. Additional information for consideration and pictures for inclusion are provided. A separate submittal will be prepared by N3B for LANL legacy TRU waste managed by N3B under contract to EM-LA.

, LANL↗

A Hybrid Data-Driven and Model-Based Anomaly Detection Scheme for DER Operation

This paper proposes a hybrid data and model-based anomaly detection scheme to secure the operation of distributed energy resources (DERs) in distribution grids. Data-driven autoencoders are set up at the edge device level and they use local DER operational data as inputs. The abnormal statuses are detected by analyzing reconstruction errors. In parallel, modelbased state estimation (SE) is set up at the central level and it uses system-wide models and measurements as data inputs. The anomalies are identified by analyzing measurement residuals. The hybrid scheme preserves the benefits of both data-driven and model-based analyses and thus improves the robustness and the accuracy of anomaly detection. Numerical tests based on the model of a real distribution feeder in Southern California highlight the proposed scheme's effectiveness and benefits.

anomaly detection↗

Detection of Anomalies in Gamma Background Radiation Data with K-Means and Self-Organizing Map Clustering Algorithms (Consortium on Nuclear Security Technologies (CONNECT) Q1 Report)

Environmental screening of gamma radiation consists of detecting weak nuisance and anomaly signal in the presence of strong and highly varying background. In a typical scenario, a mobile detector-spectrometer continuously measures gamma radiation spectra in short, e.g., one-second, signal acquisition intervals. The measurement data is a 2D matrix, where one dimension is gamma ray energy, and the other dimension is the number of measurements or total time. In principle, gamma radiation sources can be detected and identified from the measured data by their unique spectral lines. Detecting sources from data measured in a search scenario is difficult due to the highly varying background because of naturally occurring radioactive material (NORM), and low signal-to-noise ratio (S/N) of spectral signal measured during one-second acquisition intervals. The objective of this work is to explore unsupervised machine learning (ML) algorithms for detection and identification of weak nuisances and anomalies events in the presence of highly fluctuating background. The challenge is that spectral lines of isotopes are difficult to observe in one-second measurements. Averaging over the entire measurement campaign data set reveals spectral lines of most common background isotopes. Spectral lines of orphan sources, which might appear only in a few measurements during the campaign, will be washed out if averaging is performed over the entire measurement data set. The approach we have explored consists of extracting one-second measurements containing weak spectral features through data clustering. Averaging one-second spectra in a cluster should reveal the presence of anomaly sources. We created two ML models using K-means clustering and Neural Network Self-organizing Map (SOM). Performance of these ML models was benchmarked using search data. One data set contained 137 Cs source, and another dataset contained 131 I source.

61 RADIATION PROTECTION AND DOSIMETRY↗

Big Data-Based Approach to Detect, Locate, and Enhance the Stability of an Unplanned Microgrid Islanding

In this paper, a big data-based approach is proposed for the security improvement of an unplanned microgrid islanding (UMI). The proposed approach contains two major steps: the first step is big data analysis of wide-area monitoring to detect a UMI and locate it; the second step is particle swarm optimization (PSO)-based stability enhancement for the UMI. First, an optimal synchrophasor measurement device selection (OSMDS) and matching pursuit decomposition (MPD)-based spatial-temporal analysis approach is proposed to significantly reduce the volume of data while keeping appropriate information from the synchrophasor measurements. Second, a random forest-based ensemble learning approach is trained to detect the UMI. When combined with grid topology, the UMI can be located. Then the stability problem of the UMI is formulated as an optimization problem and the PSO is used to find the optimal operational parameters of the UMI. An eigenvalue-based multiobjective function is proposed, which aims to improve the damping and dynamic characteristics of the UMI. Finally, the simulation results demonstrate the effectiveness and robustness of the proposed approach.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Testing the AVNG

An attribute measurement system (AMS) measures a number of unclassified attributes of potentially classified material. By only displaying these unclassified results as red or green lights, the AMS protects potentially classified information while still generating confidence in the measurement result. The A VNG implementation that we describe is an AMS built by RFNC - VNIIEF in Sarov, Russia. The AVNG detects neutron and gamma radiation signatures and displays the three unclassified attributes of 'plutonium presence,' 'plutonium mass > 2 kg,' and 'plutonium isotopic ratio ({sup 240}Pu to {sup 239}Pu) < 0.1.' Prior to demonstration to a joint US/Russian audience, the overall operation and thresholds of the AVNG were tested using a number of multi-kg plutonium reference material (RM) sources manufactured for this purpose. As the AVNG was designed to incorporate an open mode (where all radiation data was displayed) as well as a secure mode (where only the red/green attribute lights were visible), the AVNG was tested in both modes. We will present data from these tests using unclassified plutonium RM sources as well as detector test results obtained using smaller calibration sources.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

A Privacy-Preserving Strategy for the Trust Layer of the Energy Grid of Things Distributed Energy Resource Management System

Emergent from the shadows of the traditional grid flaws, the Smart Grid (SG) idea was born and led by government mandates toward cleaner energy production. The SG represents the next generation of electricity distribution systems that subsume recent technological innovations. It uses digital communication between its components and entities to attain more automation, self-sufficiency, and reliability. Unfortunately, this relatively new concept is not flawless; the intrinsic reliance on increased digital communication spreads open attack paths for adversaries. Therefore, finding solutions that address information exchange vulnerabilities has become imperative. The Energy Grid of Things (EGoT) is Portland State University’s (PSU’s) implementation of a Distributed Energy Resource Management System (DERMS). The EGoT DERMS requires access to customers’ information to achieve operational objectives. The system’s access to customers’ information needs to be restricted such that it does not violate customers’ privacy. Applying privacy protection models such as K-anonymity to EGoT DERMS sub-components safeguards that privacy. This thesis work proposes a strategy to ensure communication in the EGoT DERMS is privacy-preserving and secure. Specifically, it provides an approach to applying the Mondrian Algorithm to ensure data within the system excludes Personally Identifiable Information (PII) and provides means for securing the communication according to industry standards (IEEE 2030.5). Results suggest that the generalization hierarchy derived for the EGoT DERMS exhibits an Identical Generalization Hierarchy structure. Guarantees of sameness manifested in the test feeder topology would not hold in real-world scenarios. This thesis work proposes a strategy to ensure communication in the EGoT DERMS is privacy-preserving and secure. Specifically, it provides an approach to applying the Mondrian Algorithm to ensure data within the system excludes Personally Identifiable Information (PII) and provides means for securing the communication according to industry standards (IEEE 2030.5). Results suggest that the generalization hierarchy derived for the EGoT DERMS exhibits an Identical Generalization Hierarchy structure. Guarantees of sameness manifested in the test feeder topology would not hold in real-world scenarios.

Alsiad, Mohammed↗

Transactive Energy System Deployment Over Insecure Communication Links

Here, in this paper, the privacy and security issues associated with the transactive energy system (TES) deployment over insecure communication links are addressed. In particular, it is ensured that 1) individual agents’ bidding information is kept private throughout hierarchical market-based interactions; and 2) any extraneous data injection attack can be quickly and easily detected. An implementation framework is proposed to enable the cryptography-based enhancement of privacy and security for the deployment of any general hierarchical systems including TESs. Under the proposed framework, a unified cryptography-based approach is developed to achieve both privacy and security simultaneously. Specifically, privacy preservation is realized by an enhanced Paillier encryption scheme, where a block design is proposed to significantly improve computational efficiency. Attack detection is further achieved by an enhanced Paillier digital signature scheme, where a stamp-concatenation mechanism is proposed to enable detection of data replace and reorder attacks. Simulation results verify the effectiveness of the proposed cyber-resilient design for transactive energy systems. Note to Practitioners—This paper is motivated by addressing the issues of cyber resiliency for practically deploying transactive energy system (TES) but it is also applicable to the problem of enhancing the privacy and security for any general hierarchical control systems. TES is an emerging control approach that engages energy suppliers and customers through market operations and uses the price to optimally allocate energy resources. While it has been shown to be promising for power system applications, the underlying market-based interactions raise significant concerns of privacy (data leakage) and security (data tampering). However, existing TES works only focus on the coordination mechanism instead of privacy and security issues. This paper proposes a new cryptography-based TES design for practical deployment. Specifically, to protect privacy, individual supply and demand amounts to be exchanged are all encrypted in a particular way such that the original amounts cannot be inferred from the encrypted amounts, while the desired computation for setting the market clearing price can be carried out over the encrypted amounts, thus generating an encrypted result which, when decrypted, matches that of the same computation over the original amounts. To achieve security, for each exchanged data, its sender generates a particular digital signature which is exchanged together with the data. This enables the receiver to automatically detect the integrity by checking whether a mathematical relationship holds for the pair of data and signature. In our future research, we will investigate more challenging scenarios where some suppliers and customers themselves could be corrupted and purposely submit distorted amounts.

97 MATHEMATICS AND COMPUTING↗

Machine Learning in the Big Data Era: Are We There Yet?

In this paper, we discuss the machine learning challenges of the Big Data era. We observe that recent innovations in being able to collect, access, organize, integrate, and query massive amounts of data from a wide variety of data sources have brought statistical machine learning under more scrutiny and evaluation for gleaning insights from the data than ever before. In that context, we pose and debate the question - Are machine learning algorithms scaling with the ability to store and compute? If yes, how? If not, why not? We survey recent developments in the state-of-the-art to discuss emerging and outstanding challenges in the design and implementation of machine learning algorithms at scale. We leverage experience from real-world Big Data knowledge discovery projects across domains of national security and healthcare to suggest our efforts be focused along the following axes: (i) the data science challenge - designing scalable and flexible computational architectures for machine learning (beyond just data-retrieval); (ii) the science of data challenge the ability to understand characteristics of data before applying machine learning algorithms and tools; and (iii) the scalable predictive functions challenge the ability to construct, learn and infer with increasing sample size, dimensionality, and categories of labels. We conclude with a discussion of opportunities and directions for future research.

Big Data↗

A National Collaboratory to Advance the Science of High Temperature Plasma Physics for Magnetic Fusion

This report summarizes the work of the National Fusion Collaboratory (NFC) Project to develop a persistent infrastructure to enable scientific collaboration for magnetic fusion research. The original objective of the NFC project was to develop and deploy a national FES Grid (FusionGrid) that would be a system for secure sharing of computation, visualization, and data resources over the Internet. The goal of FusionGrid was to allow scientists at remote sites to participate as fully in experiments and computational activities as if they were working on site thereby creating a unified virtual organization of the geographically dispersed U.S. fusion community. The vision for FusionGrid was that experimental and simulation data, computer codes, analysis routines, visualization tools, and remote collaboration tools are to be thought of as network services. In this model, an application service provider (ASP provides and maintains software resources as well as the necessary hardware resources. The project would create a robust, user-friendly collaborative software environment and make it available to the US FES community. This Grid's resources would be protected by a shared security infrastructure including strong authentication to identify users and authorization to allow stakeholders to control their own resources. In this environment, access to services is stressed rather than data or software portability.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Trust-Enhancing Probabilistic Transfer Learning for Sparse and Noisy Data Environments

There is an increasing aspiration to utilize machine learning (ML) for various tasks of relevance to national security. ML models have thus far been mostly applied to tasks and domains that, while impactful, have sufficient volume of data. For predictive tasks of national security relevance, ML models of great capacity (ability to approximate nonlinear trends in input-output maps) are often needed to capture the complex underlying physics. However, scientific problems of relevance to national security are often accompanied by various sources of sparse and/or incomplete data, including experiments and simulations, across different regimes of operation, of varying degrees of fidelity, and include noise with different characteristics and/or intensity. State-of-the-art ML models, despite exhibiting superior performance on the task and domain they were trained on, may suffer detrimental loss in performance in such sparse data environments. This report summarizes the results of the Laboratory Directed Research and Development project entitled Trust-Enhancing Probabilistic Transfer Learning for Sparse and Noisy Data Environments. The objective of the project was to develop a new transfer learning (TL) framework that aims to adaptively blend the data across different sources in tackling one task of interest, resulting in enhanced trustworthiness of ML models for mission- and safety-critical systems. The proposed framework determines when it is worth applying TL and how much knowledge is to be transferred, despite uncontrollable uncertainties. The framework accomplishes this by leveraging concepts and techniques from the fields of Bayesian inverse modeling and uncertainty quantification, relying on strong mathematical foundations of probability and measure theories to devise new uncertainty-aware TL workflows.

97 MATHEMATICS AND COMPUTING↗

Nevada National Security Site 2011 Waste Management Monitoring Report, Area 3 and Area 5 Radioactive Waste Management Site

Environmental monitoring data are collected at and around the Area 3 and Area 5 Radioactive Waste Management Sites (RWMSs) at the Nevada National Security Site (NNSS). These data are associated with radiation exposure, air, groundwater, meteorology, and vadose zone. This report summarizes the 2011 environmental data to provide an overall evaluation of RWMS performance and to support environmental compliance and performance assessment (PA) activities. Some of these data (e.g., radiation exposure, air, and groundwater) are presented in other reports. Direct radiation monitoring data indicate exposure levels at the RWMSs are within the range of background levels measured at the NNSS. Slightly elevated exposure levels outside the Area 3 RWMS are attributed to nearby historical aboveground nuclear weapons tests. Air monitoring data show tritium concentrations in water vapor and americium and plutonium concentrations in air particles are only slightly above detection limits and background levels. The measured levels of radionuclides in air particulates and moisture are below derived concentration guides for these radionuclides. During the last 2 weeks of March 2011, gamma spectroscopy results for air particles showed measurable activities of iodine-131 ( 131 I), cesium-134 ( 134 Cs), and cesium-137 ( 137 Cs). These results are attributed to the release of fission products from the damaged Fukushima Daiichi power plant in Japan. The remaining gamma spectroscopy results for air particulates collected at the Area 3 and Area 5 RWMS were below minimum detectable concentrations. Groundwater monitoring data indicate the groundwater in the uppermost aquifer beneath the Area 5 RWMS is not impacted by RWMS operations. Results of groundwater analysis from wells around the Area 5 RWMS were all below established investigation levels. The 86.3 millimeters (mm) (3.40 inches [in.]) of precipitation at the Area 3 RWMS during 2011 is 44% below the average of 154.1 mm (6.07 in.), and the 64.8 mm (2.55 in.) of precipitation at the Area 5 RWMS during 2011 is 47% below the average of 122.4 mm (4.82 in.). Water balance measurements indicate that evapotranspiration from the vegetated weighing lysimeter dries the soil and prevents downward percolation of precipitation more effectively than evaporation from the bare-soil weighing lysimeter. Automated vadose zone monitoring on Area 5 RWMS operational waste covers was not done during 2011 due to construction of the final evapotranspiration cover at these monitoring locations. Moisture from precipitation did not percolate below 122 centimeters (4 feet) in the vegetated final mono-layer cover on the U-3ax/bl disposal unit at the Area 3 RWMS before being removed by evapotranspiration. During 2011, there was no drainage through 2.4 meters (8 feet) of soil from the Area 3 drainage lysimeters that received only natural precipitation. Ten percent of the applied precipitation and irrigation drained from the bare-soil drainage lysimeter that received 3 times natural precipitation. All 2011 monitoring data indicate that the Area 3 and Area 5 RWMSs are performing within expectations of the model and parameter assumptions for the facility PAs.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Finite-key security analysis of quantum key distribution with imperfect light sources

In recent years, the gap between theory and practice in quantum key distribution (QKD) has been significantly narrowed, particularly for QKD systems with arbitrarily flawed optical receivers. The status for QKD systems with imperfect light sources is however less satisfactory, in the sense that the resulting secure key rates are often overly dependent on the quality of state preparation. This is especially the case when the channel loss is high. Very recently, to overcome this limitation, Tamaki et al proposed a QKD protocol based on the so-called 'rejected data analysis', and showed that its security in the limit of infinitely long keys is almost independent of any encoding flaw in the qubit space, being this protocol compatible with the decoy state method. Here, as a step towards practical QKD, we show that a similar conclusion is reached in the finite-key regime, even when the intensity of the light source is unstable. More concretely, we derive security bounds for a wide class of realistic light sources and show that the bounds are also efficient in the presence of high channel loss. Our results strongly suggest the feasibility of long distance provably secure communication with imperfect light sources.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Open Research Challenges with Big Data - A Data-Scientist s Perspective

In this paper, we discuss data-driven discovery challenges of the Big Data era. We observe that recent innovations in being able to collect, access, organize, integrate, and query massive amounts of data from a wide variety of data sources have brought statistical data mining and machine learning under more scrutiny and evaluation for gleaning insights from the data than ever before. In that context, we pose and debate the question - Are data mining algorithms scaling with the ability to store and compute? If yes, how? If not, why not? We survey recent developments in the state-of-the-art to discuss emerging and outstanding challenges in the design and implementation of machine learning algorithms at scale. We leverage experience from real-world Big Data knowledge discovery projects across domains of national security, healthcare and manufacturing to suggest our efforts be focused along the following axes: (i) the data science challenge - designing scalable and flexible computational architectures for machine learning (beyond just data-retrieval); (ii) the science of data challenge the ability to understand characteristics of data before applying machine learning algorithms and tools; and (iii) the scalable predictive functions challenge the ability to construct, learn and infer with increasing sample size, dimensionality, and categories of labels. We conclude with a discussion of opportunities and directions for future research.

Big data↗