Toward On-line Data Mining and Event Detection in Petascale Data Streams
No abstract available
online data mining↗
SEARCH · Search NASA
Results for “online data mining”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
MOFX-DB: An Online Database of Computational Adsorption Data for Nanoporous Materials
Machine learning and data mining coupled with molecular modeling have become powerful tools for materials discovery. Metal-organic frameworks (MOFs) are a rich area for this due to their modular construction and numerous applications. Here, we make data from several previous large-scale studies in MOFs and zeolites from our groups (and new data for N 2 and Ar adsorption in MOFs) easily accessible in one place. The database includes over 3 million simulated adsorption data points for H 2 , CH 4 , CO 2 , Xe, Kr, Ar, and N 2 in over 160 000 MOFs and zeolites, textural properties like pore sizes and surface areas, and the structure file for each material. We include metadata about the Monte Carlo simulations to enable reproducibility. The database is searchable by MOF properties, and the data are stored in a standardized JSON format that that is interoperable with the NIST adsorption database. We also identify several MOFs that meet high performance targets for multiple applications, such as high storage capacity for both hydrogen and methane or high CO 2 capacity plus good Xe/Kr selectivity. Here, by providing this data publicly, we hope to facilitate machine learning studies on these materials, leading to new insights on adsorption in MOFs and zeolites.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
MERRA-2 Data and Analytic Services at NASA GES DISC for Climate Extremes Study
NASA's climate reanalysis datasets from the Modern Era Retrospective-analysis for Research and Applications, Version 2 (MERRA-2) contains numerous long-term atmosphere, land, and ocean data products from 1980-present. MERRA-2 datasets, such as precipitation, soil moisture, and temperature, have been used widely to study extreme events. The native archived MERRA-2 data files are day-file (hourly time interval) and month-file, containing up to 125 parameters in one file. Due to the large number of data files and volumes, it is challenging for users, especially the applications research community, to handle the original hourly data files for long time periods to analyze extreme events. In this presentation, we review MERRA-2 data for studies of extreme conditions, and demonstrate analytic services at the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). One of the current operational services, 'subsetter', allows users to download only specific data of interest, i.e. data selected by parameter, region, and time period. New services are under development that will provide more 'on-the-fly' statistical calculations when downloading data; improve efficiency when accessing long time-series data. We will provide additional "How-to" resources that include step-by-step instructions on data access and usage. We have tested restructuring of day-files in an optimized data cube, which has significantly improved system performance for accessing long time-series. Overall performance is associated with cube size and structure, data compression method, and how the data are accessed. The optimized data cube structure will enable better online analytic services for statistical analysis and extreme events mining. To demonstrate the service, we use an extreme drought associated with the anomalous 2016 monsoon over southern Asia. This prototype time-series service may be augmented in the cloud infrastructure in the future.
data access↗
Visual Analytics of Multivariate Networks With Representation Learning and Composite Variable Construction
Multivariate networks are commonly found in real-world data-driven applications. Uncovering and understanding the relations of interest in multivariate networks is not a trivial task. This article presents a visual analytics workflow for studying multivariate networks to extract associations between different structural and semantic characteristics of the networks (e.g., what are the combinations of attributes largely relating to the density of a social network?). The workflow consists of a neural-network-based learning phase to classify the data based on the chosen input and output attributes, a dimensionality reduction and optimization phase to produce a simplified set of results for examination, and finally an interpreting phase conducted by the user through an interactive visualization interface. A key part of our design is a composite variable construction step that remodels nonlinear features obtained by neural networks into linear features that are intuitive to interpret. We demonstrate the capabilities of this workflow with multiple case studies on networks derived from social media usage and also evaluate the workflow with qualitative feedback from experts.
97 MATHEMATICS AND COMPUTING↗
TripleGraph
RDF triplestores are great tools for online graph analytic processing (i.e., graph pattern query processing), but they do not provide graph mining capabilities (e.g., PageRank, connected-component analysis, node eccentricity, etc.). The software title “TripleGraph” is a graph analysis toolkit, which uses an RDF triplestore as its backend for creating, manipulating, mining, and programming with large scale property graphs. It allows users to run various graph mining algorithms easily. User can import edgelist-formatted (homogeneous graph) or JSON-formatted graph (property graph) into the RDF triplestore using the provided tool and perform various analysis such as (1) Node/edge retrieval and manipulation, (2) Pathfinding between two given nodes, (3) Running graph mining algorithms (PageRank/Personalized PageRank, Single Source Shortest Path/Multi-Source Shortest Path, Connected Component, Node Eccentricity, Peer Pressure Clustering). It supports standard graph data format and works with a standard SPARQL endpoint like Jena Fuseki. It allows users to perform online graph analytic processing and graph mining on the same platform (a triplestore).
Sangkeun, MattLee↗
Data Driven Approach to Public Opinion Mining on Autonomous Vehicles: Sentiment Analysis of Social Media Comments Using Large Language Models
In the realm of online identity, social media has emerged as a rich and dynamic source of user-generated content, making it an invaluable resource for understanding public sentiment on a wide range of topics. Individuals often share their raw emotions and candid opinions on these platforms without fear of judgment or backlash. In this study, we conduct a sentiment analysis on user comments collected from various online platforms, with a specific focus on discussions surrounding autonomous vehicles. Leveraging the capabilities of large language models (LLMs), we classify each comment into one of five sentiment categories: Very Negative, Negative, Neutral, Positive, and Very Positive. Our approach demonstrates the effectiveness of LLMs in capturing nuanced contextual sentiment, offering a scalable and state-of-the-art alternative to traditional manual annotation methods. The results reveal key trends and insights into public perception, enabling a deeper understanding of how autonomous vehicle technologies are received by the online community. Our findings underscore the dynamic nature of public sentiment, which is shaped not only by advances in autonomous vehicle technology but also by contextual events such as regulatory developments, political adjustment and safety incidents.
24 POWER TRANSMISSION AND DISTRIBUTION↗
MINE 2.0: enhanced biochemical coverage for peak identification in untargeted metabolomics
Abstract Summary Although advances in untargeted metabolomics have made it possible to gather data on thousands of cellular metabolites in parallel, identification of novel metabolites from these datasets remains challenging. To address this need, Metabolic in silico Network Expansions (MINEs) were developed. A MINE is an expansion of known biochemistry which can be used as a list of potential structures for unannotated metabolomics peaks. Here, we present MINE 2.0, which utilizes a new set of biochemical transformation rules that covers 93% of MetaCyc reactions (compared to 25% in MINE 1.0). This results in a 17-fold increase in database size and a 40% increase in MINE database compounds matching unannotated peaks from an untargeted metabolomics dataset. MINE 2.0 is thus a significant improvement to this community resource. Availability and implementation The MINE 2.0 website can be accessed at https://minedatabase.ci.northwestern.edu. The MINE 2.0 web API documentation can be accessed at https://mine-api.readthedocs.io/en/latest/. The data and code underlying this article are available in the MINE-2.0-Paper repository at https://github.com/tyo-nu/MINE-2.0-Paper. MINE 2.0 source code can be accessed at https://github.com/tyo-nu/MINE-Database (MINE construction), https://github.com/tyo-nu/MINE-Server (backend web API) and https://github.com/tyo-nu/MINE-app (web app). Supplementary information Supplementary data are available at Bioinformatics online.
59 BASIC BIOLOGICAL SCIENCES↗
Open Chemistry, JupyterLab, REST, and quantum chemistry
Quantum chemistry must evolve if it wants to fully leverage the benefits of the internet age, where the worldwide web offers a vast tapestry of tools that enable users to communicate and interact with complex data at the speed and convenience of a button press. The Open Chemistry project has developed an open-source framework that offers an end-to-end solution for producing, sharing, and visualizing quantum chemical data interactively on the web using an array of modern tools and approaches. These tools build on some of the best open-source community projects such as Jupyter for interactive online notebooks, coupled with 3D accelerated visualization, state-of-the-art computational chemistry codes including NWChem and Psi4, and emerging machine learning and data mining tools such as ChemML and ANI. They offer flexible formats to import and export data, along with approaches to compare computational and experimental data.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Amazonia Disasters: Assessing Methods for Gold Mining-related Deforestation Detection in Amazonia Using NASA Earth Observations
Artisanal and small-scale gold mining (ASGM) is responsible for a large fraction of deforestation and disturbance in Amazonia. These activities cause severe impacts on the rainforest ecosystem and socioeconomic state of the region. NASA DEVELOP partnered with the Asociación para la Conservación de la Cuenca Amazónica (ACCA), NASA SERVIR Science Coordination Office, and the Spatial Informatics Group to enhance ASGM-related deforestation detection methods. ACCA currently uses the Omnibus Q-test Change Point Detection Algorithm to identify changes in Synthetic Aperture Radar (SAR) monthly-aggregated temporal data from the Sentinel-1 satellite. The team determined the algorithm's accuracy by comparing a stratified random sample of change points against data from January 2019 to June 2020 identified using PlanetScope and Landsat 8 Operational Land Imager (OLI) Earth observations through Collect Earth Online. Our results indicated a users' accuracy of 55% for temporal change detection and producer's and user's accuracies of 99% and 97%, respectively, for detecting when change did not occur. Of the labeled change points, only 19% were due to mining activity. This research can help our partners have a more accurate understanding of where illegal gold mining may be taking place and inform decisions to remediate this activity.
DEVELOP Project Summary↗
Amazonia Disasters: Assessing Methods for Gold Mining-Related Deforestation Detection in Amazonia Using NASA Earth Observations
Artisanal and small-scale gold mining (ASGM) is responsible for a large fraction of deforestation and disturbance in Amazonia. These activities cause severe impacts on the rainforest ecosystem and socioeconomic state of the region. NASA DEVELOP partnered with the Asociación para la Conservación de la Cuenca Amazónica (ACCA), NASA SERVIR Science Coordination Office, and the Spatial Informatics Group to enhance ASGM-related deforestation detection methods. ACCA currently uses the Omnibus Q-test Change Point Detection Algorithm to identify changes in Synthetic Aperture Radar (SAR) monthly-aggregated temporal data from the Sentinel-1 satellite. The team determined the algorithm's accuracy by comparing a stratified random sample of change points against data from January 2019 to June 2020 identified using PlanetScope and Landsat 8 Operational Land Imager (OLI) Earth observations through Collect Earth Online. Our results indicated a users' accuracy of 55% for temporal change detection and producer's and user's accuracies of 99% and 97%, respectively, for detecting when change did not occur. Of the labeled change points, only 19% were due to mining activity. This research can help our partners have a more accurate understanding of where illegal gold mining may be taking place and inform decisions to remediate this activity.
DEVELOP Tech Paper↗
Two Trees: Migrating Fault Trees to Decision Trees for Real Time Fault Detection on International Space Station
We started from ISS fault trees example to migrate to decision trees, presented a method to convert fault trees to decision trees. The method shows that the visualizations of root cause of fault are easier and the tree manipulating becomes more programmatic via available decision tree programs. The visualization of decision trees for the diagnostic shows a format of straight forward and easy understands. For ISS real time fault diagnostic, the status of the systems could be shown by mining the signals through the trees and see where it stops at. The other advantage to use decision trees is that the trees can learn the fault patterns and predict the future fault from the historic data. The learning is not only on the static data sets but also can be online, through accumulating the real time data sets, the decision trees can gain and store faults patterns in the trees and recognize them when they come.
Lee, Charles↗
Intelligent Systems: Terrestrial Observation and Prediction Using Remote Sensing Data
NASA has made science and technology investments to better utilize its large space-borne remote sensing data holdings of the Earth. With the launch of Terra, NASA created a data-rich environment where the challenge is to fully utilize the data collected from EOS however, despite unprecedented amounts of observed data, there is a need for increasing the frequency, resolution, and diversity of observations. Current terrestrial models that use remote sensing data were constructed in a relatively data and compute limited era and do not take full advantage of on-line learning methods and assimilation techniques that can exploit these data. NASA has invested in visualization, data mining and knowledge discovery methods which have facilitated data exploitation, but these methods are insufficient for improving Earth science models that have extensive background knowledge nor do these methods refine understanding of complex processes. Investing in interdisciplinary teams that include computational scientists can lead to new models and systems for online operation and analysis of data that can autonomously improve in prediction skill over time.
Coughlan, Joseph C.↗
Online Tools for Astronomy and Cosmochemistry
Over the past year, the Webnucleo Group at Clemson University has been developing a web site with a number of interactive online tools for astronomy and cosmochemistry applications. The site uses SHP (Simplified Hypertext Preprocessor), which, because of its flexibility, allows us to embed almost any computer language into our web pages. For a description of SHP, please see http://www.joeldenny.com/ At our web site, an internet user may mine large and complex data sets, such as our stellar evolution models, and make graphs or tables of the results. The user may also run some of our detailed nuclear physics and astrophysics codes, such as our nuclear statistical equilibrium code, which is written in fortran and C. Again, the user may make graphs and tables and download the results.
Meyer, B. S.↗
National Energy Water Treatment & Speciation (NEWTS): A Water & Critical Mineral Database and Dashboard
The scarcity of water resources, the need for beneficial water reuse, and the challenges of wastewater treatment are becoming increasingly pressing in economic, social, and environmental domains. Addressing these concerns requires effective treatment strategies to manage wastewater streams and tackle environmental and economic issues. Furthermore, the recovery of critical minerals from the waste streams associated with energy production holds the promise of offsetting treatment costs and securing local sources of valuable minerals. However, relevant data on these waste streams are dispersed and challenging to locate. The process of ingesting such data into modeling software often involves multiple steps, requiring data restructuring to meet software-input requirements. The non-standardized reporting of water data makes data aggregation and reformatting a time-consuming process. Additionally, essential attributes necessary for modeling water treatment and mineral scale formation are frequently missing. Moreover, data gaps vary depending on the region of interest. Consequently, there is a pressing need for high-quality energy-water composition data that can be easily imported into water chemistry modeling software. To address this need, the National Energy Technology Laboratory has created the National Energy Water Treatment and Speciation (NEWTS) Database and Dashboard—a free online tool catering to community leaders and water researchers. NEWTS facilitates a comprehensive understanding of the composition of energy-related wastewater streams in the United States. The datasets provide detailed concentrations and speciation of major and minor aqueous compounds in energy-related wastewater streams, including power plant leachate, acid mine drainage, brackish water, and oil and gas produced water across the United States. Many of the aqueous species are critical minerals (Li, REEs) in high demand to modernize the world’s energy infrastructure. Many of the datasets also contain volumetric flow-rates needed to model the treatment and reuse scenarios in advanced aqueous chemistry software programs. The NEWTS Database and Dashboard offer public access to hitherto challenging-to-access datasets, presented in a standardized format that is tailored for easy input into aqueous chemistry modeling software. By performing the work needed to transform dispersed, disparate data sources into unified, model-ready datasets, NEWTS serves as an essential resource in advancing water treatment research and sustainable water resource management.
produced water management↗
Beyond Fair: Engagement, Data Usability, and Open Community Productivity through the NASA Open Science Data Repository
The FAIR principle (findable, accessible, interoperable, and reusable) governs the storage and sharing of NASA space biology and health data[1]. These guiding principles maximize reuse of data and the reproducibility of scientific findings. The NASA Open Science Data Repository (OSDR; an expansion of NASA GeneLab) was built on the FAIR principles and houses over 500 studies and close to 1000 datasets from decades of space life sciences experiments. OSDR embodies the FAIR principles through data governance that includes mediated, embargoed, and fully open access data. The FAIR data governance principles were recently proposed to be expanded to encompass a FAIREST framework for assessing research data repositories (FAIR + Engagement, Social connections, and Trust)[2]. FAIREST emphasizes the importance of data repositories engaging with the scientific community and gaining the trust of researchers regarding data quality. Trust also refers to the TRUST principles developed for assessment of digital repositories: Transparency, Responsibility, User Focus, Sustainability, Technology[3]. We present the “Open Science for Life in Space” Analysis Working Groups (AWGs) as evidence regarding the power of engagement, social connections, and trust which has enhanced OSDR’s capabilities and productivity. AWG members engage in two main activities. One, members provide feedback on OSDR scientific standards for data ingestion, curation, and reuse (study, subject and assay metadata; processing pipelines; dataset formats and uniformed structures for machine-readability). Two, AWG members collaborate to mine-reuse OSDR data to conduct scientific analysis. With nearly 800 active members, the AWGs have resulted in 32 publications re-using OSDR data and contributed many papers in two major special issues in Cell (2020) and Nature (2024). AWGs also serve as networking groups, facilitate social connections between researchers at all levels of experience, and also have a social online ‘Forum’ used to keep members informed on projects and opportunities. This community-centric, productive, and trustworthy data culture has resulted in a broader effect with international space agencies, academics, and the commercial space sector wanting to submit their data to OSDR. Ten studies of Inspiration 4 data were recently publicly released by OSDR, as were some JAXA human data. Coming up soon in OSDR are data submissions from the European Space Agency, Virgin Galactic PIs, and SpaceX Polaris Dawn. A major benefit of OSDR is the array of standardized and uniformly formatted data (which was developed through AWG member consensus), from which visualization tools, analysis tools, and machine learning models can be built or trained. This talk will cover the Multi-Study Visualization Tool, the Environmental Data Application, RadLab, and a UCSF-NSF funded knowledge graph biomedical health discovery tool ‘SPOKE’ currently being integrated with OSDR. OSDR also provides training programs in bioinformatics and machine learning to improve the scientific community’s awareness of data availability and to boost their ability to perform data analysis. The increasing engagement of the scientific community and the public with technologies powered by artificial intelligence (AI) heightens the need for data analysis to be transparent. The AI for Life in Space initiative leverages the data products provided in OSDR to train AI models, with an emphasis on explainable and trustworthy AI, which would not be possible without FAIR data and metadata. Overall, here we will demonstrate the importance for NASA life sciences data repositories to adhere to the FAIREST framework, by providing examples and success stories from different aspects of OSDR.
data↗
MIBiG 3.0: a community-driven effort to annotate experimentally validated biosynthetic gene clusters
Abstract With an ever-increasing amount of (meta)genomic data being deposited in sequence databases, (meta)genome mining for natural product biosynthetic pathways occupies a critical role in the discovery of novel pharmaceutical drugs, crop protection agents and biomaterials. The genes that encode these pathways are often organised into biosynthetic gene clusters (BGCs). In 2015, we defined the Minimum Information about a Biosynthetic Gene cluster (MIBiG): a standardised data format that describes the minimally required information to uniquely characterise a BGC. We simultaneously constructed an accompanying online database of BGCs, which has since been widely used by the community as a reference dataset for BGCs and was expanded to 2021 entries in 2019 (MIBiG 2.0). Here, we describe MIBiG 3.0, a database update comprising large-scale validation and re-annotation of existing entries and 661 new entries. Particular attention was paid to the annotation of compound structures and biological activities, as well as protein domain selectivities. Together, these new features keep the database up-to-date, and will provide new opportunities for the scientific community to use its freely available data, e.g. for the training of new machine learning models to predict sequence-structure-function relationships for diverse natural products. MIBiG 3.0 is accessible online at https://mibig.secondarymetabolites.org/.
59 BASIC BIOLOGICAL SCIENCES↗
Enabling Power at Sea: Opportunities for Expanded Ocean Observations through Marine Renewable Energy Integration
The blue economy is a dynamic and rapidly growing movement that captures the interplay between economic, social, and ecological sustainability of the ocean and encompasses numerous maritime sectors and activities (e.g., commerce and trade; living resources; renewable energy; minerals, materials, and freshwater; and ocean health and data). The demand for ocean data to inform scientific, risk reduction, and national security needs is leading to a large increase in the number of deployed ocean observation and monitoring systems, most of which require increased power. Because ocean observation systems are often placed in remote locations, they primarily rely on energy storage (or in some cases in situ energy generation) to power instruments and equipment, which imposes limits on sampling rates, deployment times, and spatiotemporal resolution of data. The U.S. Department of Energy Water Power Technologies Office is exploring the potential for marine renewable energy (MRE) devices (largely wave and tidal energy converters) to provide power to support multiple blue economy opportunities. A portion of these opportunities focus on power at sea markets for providing power in off-grid and offshore locations to support a variety of ocean-based activities, including ocean observation and navigation, underwater vehicle charging, marine aquaculture, marine algae farming, and seawater mining. Initially, research has focused on better understanding how and where MRE can provide a consistent source of reliable power to extend ocean observing missions, including operation of autonomous underwater vehicles. Online surveys as well as phone and in-person interviews were conducted with experts in the field of ocean observing systems and observatories to gather end-user requirements, determine energy needs, identify opportunities for codevelopment, and pinpoint constraints for MRE to meet those needs. The surveys and interviews provided feedback on the potential for powering devices and vehicles using MRE, including identifying common themes and challenges that will inform foundational research and development steps needed to advance the integration of MRE with ocean observing systems. In most cases, additional power generation on the order of watts was identified as significantly beneficial to enhancing ocean observations capabilities.
16 TIDAL AND WAVE POWER↗
Powering the Blue Economy: Progress Exploring Marine Renewable Energy Integration With Ocean Observations
The blue economy is a dynamic and rapidly growing movement that captures the interplay between economic, social, and ecological sustainability of the ocean and encompasses numerous maritime sectors and activities (e.g., commerce and trade; living resources; renewable energy; minerals, materials, and freshwater; and ocean health and data). The demand for ocean data to inform scientific, risk reduction, and national security needs is leading to a large increase in the number of deployed ocean observation and monitoring systems, most of which require increased power. Because ocean observation systems are often placed in remote locations, they primarily rely on energy storage (or in some cases in situ energy generation) to power instruments and equipment, which imposes limits on sampling rates, deployment times, and spatiotemporal resolution of data. The U.S. Department of Energy Water Power Technologies Office is exploring the potential for marine renewable energy (MRE) devices (largely wave and tidal energy converters) to provide power to support multiple blue economy opportunities. A portion of these opportunities focus on power at sea markets for providing power in off-grid and offshore locations to support a variety of ocean-based activities, including ocean observation and navigation, underwater vehicle charging, marine aquaculture, marine algae farming, and seawater mining. Initially, research has focused on better understanding how and where MRE can provide a consistent source of reliable power to extend ocean observing missions, including operation of autonomous underwater vehicles. Online surveys as well as phone and inperson interviews were conducted with experts in the field of ocean observing systems and observatories to gather end-user requirements, determine energy needs, identify opportunities for codevelopment, and pinpoint constraints for MRE to meet those needs. The surveys and interviews provided feedback on the potential for powering devices and vehicles using MRE, including identifying common themes and challenges that will inform foundational research and development steps needed to advance the integration of MRE with ocean observing systems. In most cases, additional power generation on the order of watts was identified as significantly beneficial to enhancing ocean observations capabilities.
marine renewable energy, powering the blue economy↗