Toward On-line Data Mining and Event Detection in Petascale Data Streams
No abstract available
online data mining↗
SEARCH · Search NASA
Results for “online data mining”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
MERRA-2 Data and Analytic Services at NASA GES DISC for Climate Extremes Study
NASA's climate reanalysis datasets from the Modern Era Retrospective-analysis for Research and Applications, Version 2 (MERRA-2) contains numerous long-term atmosphere, land, and ocean data products from 1980-present. MERRA-2 datasets, such as precipitation, soil moisture, and temperature, have been used widely to study extreme events. The native archived MERRA-2 data files are day-file (hourly time interval) and month-file, containing up to 125 parameters in one file. Due to the large number of data files and volumes, it is challenging for users, especially the applications research community, to handle the original hourly data files for long time periods to analyze extreme events. In this presentation, we review MERRA-2 data for studies of extreme conditions, and demonstrate analytic services at the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). One of the current operational services, 'subsetter', allows users to download only specific data of interest, i.e. data selected by parameter, region, and time period. New services are under development that will provide more 'on-the-fly' statistical calculations when downloading data; improve efficiency when accessing long time-series data. We will provide additional "How-to" resources that include step-by-step instructions on data access and usage. We have tested restructuring of day-files in an optimized data cube, which has significantly improved system performance for accessing long time-series. Overall performance is associated with cube size and structure, data compression method, and how the data are accessed. The optimized data cube structure will enable better online analytic services for statistical analysis and extreme events mining. To demonstrate the service, we use an extreme drought associated with the anomalous 2016 monsoon over southern Asia. This prototype time-series service may be augmented in the cloud infrastructure in the future.
data access↗
Amazonia Disasters: Assessing Methods for Gold Mining-related Deforestation Detection in Amazonia Using NASA Earth Observations
Artisanal and small-scale gold mining (ASGM) is responsible for a large fraction of deforestation and disturbance in Amazonia. These activities cause severe impacts on the rainforest ecosystem and socioeconomic state of the region. NASA DEVELOP partnered with the Asociación para la Conservación de la Cuenca Amazónica (ACCA), NASA SERVIR Science Coordination Office, and the Spatial Informatics Group to enhance ASGM-related deforestation detection methods. ACCA currently uses the Omnibus Q-test Change Point Detection Algorithm to identify changes in Synthetic Aperture Radar (SAR) monthly-aggregated temporal data from the Sentinel-1 satellite. The team determined the algorithm's accuracy by comparing a stratified random sample of change points against data from January 2019 to June 2020 identified using PlanetScope and Landsat 8 Operational Land Imager (OLI) Earth observations through Collect Earth Online. Our results indicated a users' accuracy of 55% for temporal change detection and producer's and user's accuracies of 99% and 97%, respectively, for detecting when change did not occur. Of the labeled change points, only 19% were due to mining activity. This research can help our partners have a more accurate understanding of where illegal gold mining may be taking place and inform decisions to remediate this activity.
DEVELOP Project Summary↗
Amazonia Disasters: Assessing Methods for Gold Mining-Related Deforestation Detection in Amazonia Using NASA Earth Observations
Artisanal and small-scale gold mining (ASGM) is responsible for a large fraction of deforestation and disturbance in Amazonia. These activities cause severe impacts on the rainforest ecosystem and socioeconomic state of the region. NASA DEVELOP partnered with the Asociación para la Conservación de la Cuenca Amazónica (ACCA), NASA SERVIR Science Coordination Office, and the Spatial Informatics Group to enhance ASGM-related deforestation detection methods. ACCA currently uses the Omnibus Q-test Change Point Detection Algorithm to identify changes in Synthetic Aperture Radar (SAR) monthly-aggregated temporal data from the Sentinel-1 satellite. The team determined the algorithm's accuracy by comparing a stratified random sample of change points against data from January 2019 to June 2020 identified using PlanetScope and Landsat 8 Operational Land Imager (OLI) Earth observations through Collect Earth Online. Our results indicated a users' accuracy of 55% for temporal change detection and producer's and user's accuracies of 99% and 97%, respectively, for detecting when change did not occur. Of the labeled change points, only 19% were due to mining activity. This research can help our partners have a more accurate understanding of where illegal gold mining may be taking place and inform decisions to remediate this activity.
DEVELOP Tech Paper↗
Two Trees: Migrating Fault Trees to Decision Trees for Real Time Fault Detection on International Space Station
We started from ISS fault trees example to migrate to decision trees, presented a method to convert fault trees to decision trees. The method shows that the visualizations of root cause of fault are easier and the tree manipulating becomes more programmatic via available decision tree programs. The visualization of decision trees for the diagnostic shows a format of straight forward and easy understands. For ISS real time fault diagnostic, the status of the systems could be shown by mining the signals through the trees and see where it stops at. The other advantage to use decision trees is that the trees can learn the fault patterns and predict the future fault from the historic data. The learning is not only on the static data sets but also can be online, through accumulating the real time data sets, the decision trees can gain and store faults patterns in the trees and recognize them when they come.
Lee, Charles↗
Intelligent Systems: Terrestrial Observation and Prediction Using Remote Sensing Data
NASA has made science and technology investments to better utilize its large space-borne remote sensing data holdings of the Earth. With the launch of Terra, NASA created a data-rich environment where the challenge is to fully utilize the data collected from EOS however, despite unprecedented amounts of observed data, there is a need for increasing the frequency, resolution, and diversity of observations. Current terrestrial models that use remote sensing data were constructed in a relatively data and compute limited era and do not take full advantage of on-line learning methods and assimilation techniques that can exploit these data. NASA has invested in visualization, data mining and knowledge discovery methods which have facilitated data exploitation, but these methods are insufficient for improving Earth science models that have extensive background knowledge nor do these methods refine understanding of complex processes. Investing in interdisciplinary teams that include computational scientists can lead to new models and systems for online operation and analysis of data that can autonomously improve in prediction skill over time.
Coughlan, Joseph C.↗
Online Tools for Astronomy and Cosmochemistry
Over the past year, the Webnucleo Group at Clemson University has been developing a web site with a number of interactive online tools for astronomy and cosmochemistry applications. The site uses SHP (Simplified Hypertext Preprocessor), which, because of its flexibility, allows us to embed almost any computer language into our web pages. For a description of SHP, please see http://www.joeldenny.com/ At our web site, an internet user may mine large and complex data sets, such as our stellar evolution models, and make graphs or tables of the results. The user may also run some of our detailed nuclear physics and astrophysics codes, such as our nuclear statistical equilibrium code, which is written in fortran and C. Again, the user may make graphs and tables and download the results.
Meyer, B. S.↗
Beyond Fair: Engagement, Data Usability, and Open Community Productivity through the NASA Open Science Data Repository
The FAIR principle (findable, accessible, interoperable, and reusable) governs the storage and sharing of NASA space biology and health data[1]. These guiding principles maximize reuse of data and the reproducibility of scientific findings. The NASA Open Science Data Repository (OSDR; an expansion of NASA GeneLab) was built on the FAIR principles and houses over 500 studies and close to 1000 datasets from decades of space life sciences experiments. OSDR embodies the FAIR principles through data governance that includes mediated, embargoed, and fully open access data. The FAIR data governance principles were recently proposed to be expanded to encompass a FAIREST framework for assessing research data repositories (FAIR + Engagement, Social connections, and Trust)[2]. FAIREST emphasizes the importance of data repositories engaging with the scientific community and gaining the trust of researchers regarding data quality. Trust also refers to the TRUST principles developed for assessment of digital repositories: Transparency, Responsibility, User Focus, Sustainability, Technology[3]. We present the “Open Science for Life in Space” Analysis Working Groups (AWGs) as evidence regarding the power of engagement, social connections, and trust which has enhanced OSDR’s capabilities and productivity. AWG members engage in two main activities. One, members provide feedback on OSDR scientific standards for data ingestion, curation, and reuse (study, subject and assay metadata; processing pipelines; dataset formats and uniformed structures for machine-readability). Two, AWG members collaborate to mine-reuse OSDR data to conduct scientific analysis. With nearly 800 active members, the AWGs have resulted in 32 publications re-using OSDR data and contributed many papers in two major special issues in Cell (2020) and Nature (2024). AWGs also serve as networking groups, facilitate social connections between researchers at all levels of experience, and also have a social online ‘Forum’ used to keep members informed on projects and opportunities. This community-centric, productive, and trustworthy data culture has resulted in a broader effect with international space agencies, academics, and the commercial space sector wanting to submit their data to OSDR. Ten studies of Inspiration 4 data were recently publicly released by OSDR, as were some JAXA human data. Coming up soon in OSDR are data submissions from the European Space Agency, Virgin Galactic PIs, and SpaceX Polaris Dawn. A major benefit of OSDR is the array of standardized and uniformly formatted data (which was developed through AWG member consensus), from which visualization tools, analysis tools, and machine learning models can be built or trained. This talk will cover the Multi-Study Visualization Tool, the Environmental Data Application, RadLab, and a UCSF-NSF funded knowledge graph biomedical health discovery tool ‘SPOKE’ currently being integrated with OSDR. OSDR also provides training programs in bioinformatics and machine learning to improve the scientific community’s awareness of data availability and to boost their ability to perform data analysis. The increasing engagement of the scientific community and the public with technologies powered by artificial intelligence (AI) heightens the need for data analysis to be transparent. The AI for Life in Space initiative leverages the data products provided in OSDR to train AI models, with an emphasis on explainable and trustworthy AI, which would not be possible without FAIR data and metadata. Overall, here we will demonstrate the importance for NASA life sciences data repositories to adhere to the FAIREST framework, by providing examples and success stories from different aspects of OSDR.
data↗
Value-added Data Services at the Goddard Earth Sciences Data and Information Services Center
The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), in addition to serving the Earth Science community as one of the major Distributed Active Archives Centers (DAACs), provides much more than just data. Among the value-added services available to general users are subsetting data spatially and/or by parameter, online analysis (to avoid downloading unnecessarily all the data), and assistance in obtaining data from other centers. Services available to data producers and high-volume users include consulting on building new products with standard formats and metadata and construction of data management systems. A particularly useful service is data processing at the DISC (i.e., close to the input data) with the users algorithm. This can take a number of different forms: as a configuration-managed algorithm within the main processing stream; as a stand-alone program next to the on-line data storage; as build-it-yourself code within the Near-Archive Data Mining (NADM) system; or as an on-the-fly analysis with simple algorithms embedded into the web-based tools. Partnerships between the GES DISC and scientists, both producers and users, allow the scientists to concentrate on science, while the GES DISC handles the data management, e.g., formats, integration, and data processing. The existing data management infrastructure at the GES DISC supports a wide spectrum of options: from simple data support to sophisticated on-line analysis tools, producing economies of scale and rapid time-to-deploy. At the same time, such partnerships allow the GES DISC to serve the user community more efficiently and to better prioritize on-line holdings. Several examples of successful partnerships are described in the presentation.
Leptoukh, Gregory G.↗
Building a Massive Volcano Archive and the Development of a Tool for the Science Community
The Jet Propulsion Laboratory has traditionally housed one of the world's largest databases of volcanic satellite imagery, the ASTER Volcano Archive (~10Tb), making these data accessible online for public and scientific use. However, a series of changes in how satellite imagery is housed by the Earth Observing System (EOS) Data Information System has meant that JPL has been unable to systematically maintain its database for the last several years. We have provided a fast, transparent, machine-to-machine client that has updated JPL's database and will keep it current in near real-time. The development of this client has also given us the capability to retrieve any data provided by NASA's Earth Observing System Clearinghouse (ECHO) that covers a volcanic event reported by U.S. Air Force Weather Agency (AFWA). We will also provide a publicly available tool that interfaces with ECHO that can provide functionality not available in any of ECHO's Earth science discovery tools.
remote sensing↗
Scraping Unstructured Data to Explore the Relationship between Rainfall Anomalies and Vector-Borne Disease Outbreaks
According to the World Health Organization (WHO), vector-borne diseases such as malaria and dengue account for 17% of all infectious disease cases and lead to more than 700,000 deaths per year. Tracking and predicting the spread of vector-borne diseases is a vital task that could save hundreds of thousands of lives annually. Oftentimes, the first reports of vector-borne disease outbreaks occur through emails and online reporting systems long before they are officially documented. Tracking and predicting the emergence and spread of vector-borne disease outbreaks requires extracting data from these unstructured sources in combination with historical weather and climate data to understand the underlying background triggers and disease dynamics. In this work, we develop a data extraction pipeline for the online outbreak reporting website ProMED-mail that utilizes a web scraper, transformer neural network summarizer, and named entity recognizer to obtain a dataset of malaria, dengue, zika, and chikungunya outbreaks over the last 30 years. This scraped dataset was further analyzed in association with global rainfall anomalies derived from NASA’s Integrated Multi-satellitE Retrievals for GPM [Global Precipitation Mission] (IMERG) dataset. This preliminary analysis was to understand the effect of global rainfall patterns on the spread of vector-borne diseases. Analysis of the ProMED-mail and GPM data shows that vector-borne disease outbreaks are clustered towards the tropics and outbreaks are often amplified during the rainy seasons. Our scraped dataset can be a valuable tool in creating comprehensive georeferenced disease records for modeling and predicting future outbreaks.
Web scraping↗
Apollo Crew Loads Data Mining
INTRODUCTION Human exposure data to dynamic loads in spaceflight are sparse not only because very few crewmembers have flown in space, but also because much of the historic data is no longer readily available. Earth and Moon landings of the Apollo missions are unique and valuable for understanding human tolerance in water landing conditions for both short-duration and long-duration crewmembers as well as informing crew tolerance in standing postures during planetary landings. The goal of this task was to find relevant Apollo data to better understand human tolerance in these scenarios, in order to apply this knowledge to future missions. METHODS The goal of this task was to recover time history data from dynamic phases of flight from crewed Apollo missions. The first phase of the task was to search available records at JSC and other centers to locate the appropriate records at the National Archives in Fort Worth, TX. Once the possible data locations were located, it was planned that one researcher would travel to the National Archives in Fort Worth, TX to collect the relevant Apollo data. Once the data was retrieved, they would be digitized for future use and evaluation, and archived within the Human Physiology, Performance, Protection, and Operations (H-3PO) laboratory within the Human Health and Performance Directorate. These data would be used to assess the feasibility and acceptability of current standards and vehicle design requirements for lunar and Mars missions. Additionally, any medical monitoring required, or physiological changes noted in crewmembers based on dynamic loads would be documented for future use. RESULTS Relevant records were found at Ft. Worth, TX and College Park, MD using the National Archives Online Catalog. In addition to the online catalog, an Archives staff member was able to share an additional inventory list of NASA JSC records held in Ft. Worth, TX. Due to COVID-19, NASA has only recently allowed non-mission essential travel. In addition, the National Archives has also been closed to the public since March 2020. Because of this, travel to the National Archives to search records there has not been possible. CONCLUSION The findings include collections at the National Archives at both Ft. Worth, TX and College Park, MD that could possibly hold the data we are looking for. Unfortunately, the descriptions of each collection are very broad. Though we are hopeful that relevant data is stored in one of these collections, we cannot be sure that we will find what we are looking for. At this time, it is not clear when the Archives will reopen to the public or when NASA will allow travel. Travel to the Archives has been postponed, and will be reassessed at a future date.
T Reiber↗