SEARCH · Search NASA
Results for “big data analysis”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Anticipated Changes in Conducting Scientific Data-Analysis Research in the Big-Data Era
A Big-Data environment is one that is capable of orchestrating quick-turnaround analyses involving large volumes of data for numerous simultaneous users. Based on our experiences with a prototype Big-Data analysis environment, we anticipate some important changes in research behaviors and processes while conducting scientific data-analysis research in the near future as such Big-Data environments become the mainstream. The first anticipated change will be the reduced effort and difficulty in most parts of the data management process. A Big-Data analysis environment is likely to house most of the data required for a particular research discipline along with appropriate analysis capabilities. This will reduce the need for researchers to download local copies of data. In turn, this also reduces the need for compute and storage procurement by individual researchers or groups, as well as associated maintenance and management afterwards. It is almost certain that Big-Data environments will require a different "programming language" to fully exploit the latent potential. In addition, the process of extending the environment to provide new analysis capabilities will likely be more involved than, say, compiling a piece of new or revised code.We thus anticipate that researchers will require support from dedicated organizations associated with the environment that are composed of professional software engineers and data scientists. A major benefit will likely be that such extensions are of higherquality and broader applicability than ad hoc changes by physical scientists. Another anticipated significant change is improved collaboration among the researchers using the same environment. Since the environment is homogeneous within itself, many barriers to collaboration are minimized or eliminated. For example, data and analysis algorithms can be seamlessly shared, reused and re-purposed. In conclusion, we will be able to achieve a new level of scientific productivity in the Big-Data analysis environments.
Re-Organizing Earth Observation Data Storage to Support Temporal Analysis of Big Data
The Earth Observing System Data and Information System archives many datasets that are critical to understanding long-term variations in Earth science properties. Thus, some of these are large, multi-decadal datasets. Yet the challenge in long time series analysis comes less from the sheer volume than the data organization, which is typically one (or a small number of) time steps per file. The overhead of opening and inventorying complex, API-driven data formats such as Hierarchical Data Format introduces a small latency at each time step, which nonetheless adds up for datasets with O(10^6) single-timestep files. Several approaches to reorganizing the data can mitigate this overhead by an order of magnitude: pre-aggregating data along the time axis (time-chunking); storing the data in a highly distributed file system; or storing data in distributed columnar databases. Storing a second copy of the data incurs extra costs, so some selection criteria must be employed, which would be driven by expected or actual usage by the end user community, balanced against the extra cost.
Analysis Ready Data in Analytics Optimized Data Stores for Analysis of Big Earth Data in the Cloud
Cloud computing offers the possibility of making the analysis of Big Data approachable for a wider community due to affordable access to computing power, an ecosystem of usable tools for parallel processing, and migration of many large datasets to archives in the cloud, allowing data-proximal computing. Generally, data analysis acceleration in the cloud comes from running multiple nodes in a split-combine-apply strategy. Data systems such as the Earth Observing System Data and Information System are in a position to "pre-split" the data by storing them in a data store that is optimized for data parallel computing, i.e., an Analytics-Optimized Data Store (AODS). A variety of approaches to AODS are possible, from highly scalable databases to scalable filesystems to data formats optimized for cloud access (e.g., zarr and cloud-optimized datasets), with the optimal choice dependent on both the types of analysis and the geospatial structure of the data. A key question is how much preprocessing of the data to do, both before splitting and as the first part of the apply step. Again, the geospatial structure of the data and the analysis type influence the decision, with the added complexity of the user type. Trans-disciplinary users who are not well-versed in the nuances of quality-filtering and georeferencing of remote sensing orbit/swath/scene data tend to ask for more highly processed data, relying on the data provider to make sensible decisions on preprocessing parameters. (This accounts for the popularity of "Level 3" gridded data, despite the lower spatial resolution it provides.) In this case, data can be preprocessed before the split, resulting in higher performance in the rest of the "apply" step, which can be transformative for use cases such as interactive data exploration at scale. Discipline researchers who are experienced with remote sensing data often prefer more flexibility in customizing the preprocessing data into Analysis Ready Data, resulting in more need for on-the-fly preprocessing.
Earth Science Data Analysis in the Era of Big Data
Anyone with even a cursory interest in information technology cannot help but recognize that "Big Data" is one of the most fashionable catchphrases of late. From accurate voice and facial recognition, language translation, and airfare prediction and comparison, to monitoring the real-time spread of flu, Big Data techniques have been applied to many seemingly intractable problems with spectacular successes. They appear to be a rewarding way to approach many currently unsolved problems. Few fields of research can claim a longer history with problems involving voluminous data than Earth science. The problems we are facing today with our Earth's future are more complex and carry potentially graver consequences than the examples given above. How has our climate changed? Beside natural variations, what is causing these changes? What are the processes involved and through what mechanisms are these connected? How will they impact life as we know it? In attempts to answer these questions, we have resorted to observations and numerical simulations with ever-finer resolutions, which continue to feed the "data deluge." Plausibly, many Earth scientists are wondering: How will Big Data technologies benefit Earth science research? As an example from the global water cycle, one subdomain among many in Earth science, how would these technologies accelerate the analysis of decades of global precipitation to ascertain the changes in its characteristics, to validate these changes in predictive climate models, and to infer the implications of these changes to ecosystems, economies, and public health? Earth science researchers need a viable way to harness the power of Big Data technologies to analyze large volumes and varieties of data with velocity and veracity. Beyond providing speedy data analysis capabilities, Big Data technologies can also play a crucial, albeit indirect, role in boosting scientific productivity by facilitating effective collaboration within an analysis environment. To illustrate the effects of combining a Big Data technology with an effective means of collaboration, we relate the (fictitious) experience of an early-career Earth science researcher a few years beyond the present, interlaced and contrasted with reminiscences of its recent past (i.e., the present).
Taming Big Data Variety in the Earth Observing System Data and Information System
Although the volume of the remote sensing data managed by the Earth Observing System Data and Information System is formidable, an oft-overlooked challenge is the variety of data. The diversity in satellite instruments, science disciplines and user communities drives cost as much or more as the data volume. Several strategies are used to tame this variety: data allocation to distinct centers of expertise; a common metadata repository for discovery, data format standards and conventions; and services that further abstract the variations in data.
Earth Science Technology Office (ESTO) New Observing Strategies (NOS) and NOS-Testbed (NOS-T)
With the advancement of space hardware technologies such as smaller spacecraft, component and instrument miniaturization and high performance space processors, and with the advancement of software technologies in artificial intelligence, big data analysis and autonomous decision making, Earth Science is looking at novel ways to observe phenomena that previously could not have been studied or would have been too expensive to study with traditional missions. In particular, the New Observing Strategies (NOS) component of the NASA Earth Science Technology Office (ESTO) Advanced Information Systems Technology (AIST) Program aims at leveraging these novel technologies as well as low cost and easy access to space to acquire multi-temporal or simultaneous multi-angular, multi-locations, multi-resolution and multi-spectral observations that will provide better multi-source measurements and will build a more dynamic and comprehensive picture of Earth Science phenomena that need to be studied and analyzed. For applications such as water resources management, air quality monitoring, biodiversity studies or disaster management, NOS will integrate the use of small instruments, small spacecraft, constellations of spacecraft and networks of sensors to design new missions that will provide the necessary measurements to improve future forecast and science modeling systems.Measurement acquisition will therefore be approached as a system of systems rather than on a mission basis, and a system of this complexity should not be expected to work without full integration and experimental characterization. Although most of the individual technologies enabling to link and coordinate multi-source observations are more or less mature, a few technologies need to be developed and all of them need to be integrated and tested as a system. In order for this validation to occur, the AIST Program is developing the NOS Testbed that includes 3 main goals:1.Validate novel NOS technologies, independently and as a system2.Demonstrate novel distributed operations concepts3.Socialize new Distributed Spacecraft Mission (DSM) and SensorWeb (SW) technologies and concepts to the science community by significantly retiring the risk of integrating these new technologies.The NOS Testbed will consist of multiple sensing nodes, simulated or actual, representing space, air and/or ground measurements, that are interconnected by a communications fabric (infrastructure that permits nodes to transmit and receive data between one another and interact with each other). Each node will be supported by hardware capabilities required to perform nodes monitoring and command & control, as well as intelligent "onboard" computing. The nodes will work together in a collaborative manner to demonstrate optimal science capabilities. The testbed will enable to validate technologies such as inter-node communication models, techniques and protocols; inter-node coordination; real-time data fusion and understanding; planning; sensor re-targeting; etc. Additionally, the testbed will have the capability to interact with various mission design tools, OSSEs and one or several forecast models. More details about the NOS Testbed will be presented at the confererence.
Big lunar data visualization and analysis
NASA's earth and planetary spacecraft return large amounts of remote sensing data, such as imagery and raw science measurements, in support of remarkable research. Not only does the data lead to new scientific discoveries about our planet and the solar system, it provides a wealth of information to educate, inspire, and engage the public at large. To leverage this rich data for mission planning, scientific research, public outreach and education, it is essential to make it accessible and understandable, analyzable, all while appealing to their interests. This presentation will highlight web-based capabilities that showcase NASA's large volume of lunar data collected from past and current Moon missions. It is particularly relevant as the new Administration has more plans for the Moon. We will illustrate big data visualization and analysis in easy-touse and interactive mediums for diverse use.
High Resolution Nature Runs and the Big Data Challenge
NASA's Global Modeling and Assimilation Office at Goddard Space Flight Center is undertaking a series of very computationally intensive Nature Runs and a downscaled reanalysis. The nature runs use the GEOS-5 as an Atmospheric General Circulation Model (AGCM) while the reanalysis uses the GEOS-5 in Data Assimilation mode. This paper will present computational challenges from three runs, two of which are AGCM and one is downscaled reanalysis using the full DAS. The nature runs will be completed at two surface grid resolutions, 7 and 3 kilometers and 72 vertical levels. The 7 km run spanned 2 years (2005-2006) and produced 4 PB of data while the 3 km run will span one year and generate 4 BP of data. The downscaled reanalysis (MERRA-II Modern-Era Reanalysis for Research and Applications) will cover 15 years and generate 1 PB of data. Our efforts to address the big data challenges of climate science, we are moving toward a notion of Climate Analytics-as-a-Service (CAaaS), a specialization of the concept of business process-as-a-service that is an evolving extension of IaaS, PaaS, and SaaS enabled by cloud computing. In this presentation, we will describe two projects that demonstrate this shift. MERRA Analytic Services (MERRA/AS) is an example of cloud-enabled CAaaS. MERRA/AS enables MapReduce analytics over MERRA reanalysis data collection by bringing together the high-performance computing, scalable data management, and a domain-specific climate data services API. NASA's High-Performance Science Cloud (HPSC) is an example of the type of compute-storage fabric required to support CAaaS. The HPSC comprises a high speed Infinib and network, high performance file systems and object storage, and a virtual system environments specific for data intensive, science applications. These technologies are providing a new tier in the data and analytic services stack that helps connect earthbound, enterprise-level data and computational resources to new customers and new mobility-driven applications and modes of work. In our experience, CAaaS lowers the barriers and risk to organizational change, fosters innovation and experimentation, and provides the agility required to meet our customers' increasing and changing needs
Introduction to Analysis Methods for Big Earth Data
Big Earth Data are too big to be tractable to simple data inspection and require models to make sense of all the data. Useful models for Big Earth Data may be physical, statistical, or machine learning based. In many cases, hybrid models combine attributes of two or more of these types.
Introduction to Analysis Methods for Big Earth Data
Big Earth Data are too big to be tractable to simple data inspection. Thus, they typically require models to make sense of all the data. Useful models for Big Earth Data may be physical, statistical, or machine learning based. While physical models are ideal for understanding the data, they are not always feasible, particularly when our ability to observe at finer scales exceeds our ability to incorporate the physics. Statistical models are more generalized, but computationally intensive for many Earth Observation datasets. Machine Learning models generally scale well but are sometimes limited in the physical understanding they can offer. Hybrid models combine attributes—and advantages—of two or more of these types.
Self-Aware Vehicles: Mission and Performance Adaptation to System Health
Advances in sensing (miniaturization, distributed sensor networks) combined with improvements in computational power leading to significant gains in perception, real-time decision making/reasoning and dynamic planning under uncertainty as well as big data predictive analysis have set the stage for realization of autonomous system capability. These advances open the design and operating space for self-aware vehicles that are able to assess their own capabilities and adjust their behavior to either complete the assigned mission or to modify the mission to reflect their current capabilities. This paper discusses the self-aware vehicle concept and associated technologies necessary for full exploitation of the concept. A self-aware aircraft, spacecraft or system is one that is aware of its internal state, has situational awareness of its environment, can assess its capabilities currently and project them into the future, understands its mission objectives, and can make decisions under uncertainty regarding its ability to achieve its mission objectives.
NASA Pilot-Engaged Expert Response Using IBM Watson Technology: Prototype Evaluation of Knowledge Retrieval System
NASA Langley Research Center and IBM have been investigating the use of IBM Watson technology in aerospace research and development. One application of Watson technology is the Pilot-Engaged Expert Response (PEER) use case. The PEER system is envisioned as an in-cockpit advisor that will act as a source of situationally-relevant information for pilots and other flight crew members to assist in decision making about real-time events and situations that arise in the course of aircraft operations. PEER will make available vast stores of knowledge and information quickly and directly, putting important informational resources where they are needed most. IBM has worked with NASA to develop an architecture and articulate a roadmap for the development of the PEER system. That vision is built around Watson Discovery Advisor (WDA) software solution, derived from IBM's Jeopardy!-winning automatic question answering system. PEER makes use of WDA's sophisticated question-answering capabilities as its core, adding important User Interface components and other customizations for the cockpit environment, including communication with flight systems and other external data sources. The development plan for PEER includes four development stages, with the current project constituting the first phase. In this project, a prototype instance of PEER was successfully adapted to the aviation domain, enabling users to ask questions about aviation topics and receive useful and accurate answers to these questions. Major tasks accomplished include the development of procedures for domain adaptation through automatic lexicon extraction from domain glossaries; generation of question-answer training data which was used to train the system; and assessment of the effectiveness of domain adaptation, which showed a dramatic improvement in the ability of the PEER system to answer domain-relevant questions. In addition, the vision for the PEER system was pushed forward by the articulation of a plan for the automatic enhancement of question-answering with contextual information. This initial phase focused on two main goals: 1) the targeted domain adaptation of the underlying WDA system to the aviation domain; and, 2) the design of the software systems needed to leverage flight-contextual data. Domain adaptation of the WDA system proceeds via three main activities: Domain data ingestion, lexical customization and model training. A textual corpus consisting of 1,147 individual documents with more than 7.5 million words of text was ingested into the system and this served as the basis of all further development. A domain lexicon of over 3,500 aviation-domain terms was semi-automatically generated from domain documents and used to train the system. In addition, a set of over 500 question-answer (QA) pairs relevant to the PEER use case was developed; these were used to train and assess the system. These important first steps established the basis for the PEER system. In addition, steps were taken towards the integration of the PEER system into the cockpit environment with the development of a functional design for the Contextual Data Augmentation (CDA) subsystem. This subsystem brings to bear contextual data to improve system responses. It has three main submodules: the Contextual Data Collection module, the Contextual Data Selection module, and the Contextual QA Augmentation module. These modules form a processing pipeline that addresses the problems associated with automatically integrating information from external resources into the knowledge-retrieval mechanism.
2015 Space Radiation Standing Review Panel
The 2015 Space Radiation Standing Review Panel (from here on referred to as the SRP) met for a site visit in Houston, TX on December 8 - 9, 2015. The SRP met with representatives from the Space Radiation Element and members of the Human Research Program (HRP) to review the updated research plan for the Risk of Radiation Carcinogenesis Cancer Risk. The SRP also reviewed the newly revised Evidence Reports for the Risk of Acute Radiation Syndromes Due to Solar Particle Events (SPEs) (Acute Risk), the Risk of Acute (In-flight) and Late Central Nervous System Effects from Radiation Exposure (CNS Risk), and the Risk of Cardiovascular Disease and Other Degenerative Tissue Effects from Radiation (Degen Risk), as well as a status update on these Risks. The SRP would like to commend Dr. Simonsen, Dr. Huff, Dr. Nelson, and Dr. Patel for their detailed presentations. The Space Radiation Element did a great job presenting a very large volume of material. The SRP considers it to be a strong program that is well-organized, well-coordinated and generates valuable data. The SRP commended the tissue sharing protocols, working groups, systems biology analysis, and standardization of models. In several of the discussed areas the SRP suggested improvements of the research plans in the future. These include the following: It is important that the team has expanded efforts examining immunology and inflammation as important components of the space radiation biological response. This is an overarching and important focus that is likely to apply to all aspects of the program including acute, CVD, CNS, cancer and others. Given that the area of immunology/inflammation is highly complex (and especially so as it relates to radiation), it warrants the expansion of investigators expertise in immunology and inflammation to work with the individual research projects and also the NASA Specialized Center of Research (NSCORs). Historical data on radiation injury to be entered into the Watson “big data” study must be used with caution. The general scientific issues of reproducibility, details of experimental methods and data analysis from preclinical and basic research laboratories have been raised broadly over the last few years (not specific to this work) and indicate that caution must be applied in the ways these data are used. This pertains to preclinical data and also to phase 3 clinical trials in radiation oncology and medical oncology. Of course, appropriate use and analysis of these “big-data” sets also offer the potential of pinpointing limitations and extracting remaining useful information. Emphasis should be placed on the latter possibility. A key target is risk reduction from radiation exposure. Progress of the entire space program, now moving towards the Mars mission, requires timely answers to key components of human risk, which are known to be complex. Periodic review of progress should be conducted with additional resources directed into achieving critical milestones. Turning the long red bars to yellow and green (or for some risks such as CNS possibly to grey) must be high priority. That such progress will require new science and not engineering means that it should be viewed in a knowledge-based light. The technology-based aspects of engineering issues are certainly as important, however, science and knowledge-based problems are solved in a different way than engineering. Timelines for engineering are more predictable, while for science, progress can be methodical with occasional major incremental findings that can rapidly change the rate of progress. As opportunities for rapid incremental changes arise, periodic enhancement of investment is strongly recommended to enable such new knowledge to be quickly and efficiently exploited. Collaborations and linkages with National Institute of Allergy and Infectious Diseases (NIAID), the Biomedical Advanced Research and Development Authority (BARDA) and the Department of Defense (DoD) are in place and more are encouraged, where possible, with the radiation injury and medical countermeasure studies. This could include utilizing some of their animal model testing contracts to facilitate obtaining results using common platforms. Such approach will facilitate the comparison of results among laboratories, and will facilitate and accelerate the development of medical countermeasures. It is particularly noteworthy that the NASA Space Radiation Element is reaching out to the Multidisciplinary European Low Dose Initiative (MELODI) platform coordinating low dose radiation risk research, and to other international agencies that are studying low dose radiation effects in an effort to fill the void generated by the cancelation of the Department of Energy (DOE) low dose radiation program. While NASA is working actively with NIAID and BARDA to integrate their relevant findings of radiation mitigator investigations to NASA programs, the committee notes its disappointment that the United States currently lacks a dedicated low dose radiation program with clear mechanistic orientation and aimed at the quantification and mitigation of human radiation risk on Earth. This void gives to the NASA Space Radiation Program Element special societal value, but also makes its overall design more challenging.
Experiment to Evaluate the Feasibility of Utilizing Skylab-EREP Remote Sensing Data for Tectonic Analysis Through a Study of the Big Horn Mountain Region, Wyoming, South Dakota and Wyoming
The author has identified the following significant results. S190B imagery was the best single product from which fairly detailed structural and some lithologic mapping could be accomplished in the Big Horn basin, the Owl Creek Mountains, and the northern Big Horn Mountains. The Nye-Bowler lineament could not be extended east of its presently mapped location although a linear (fault or monocline) was noted that may be part of the lineament, but north of postulated extensions. Much more structure was discernible in the Big Horn basin than could be seen on LANDSAT-1 imagery; RB-57 color IR photography, in turn, revealed additional folds and faults. A number of linears, several of which could be identified as faults and one a monocline, cut obliquely the east-west trending Owl Creek uplift. The heavy forest cover of the Black Hills makes direct lithologic delineation impossible. However, drainage and linear overlays revealed differences in pattern between the areas of exposed Precambrian crystalline core and the flanking Paleozoic rocks. S192 data, even precision corrected segments, were not of much use.
Scalable Adaptive Graphics Environment (SAGE) Software for the Visualization of Large Data Sets on a Video Wall
The use of collaborative scientific visualization systems for the analysis, visualization, and sharing of "big data" available from new high resolution remote sensing satellite sensors or four‐dimensional numerical model simulations is propelling the wider adoption of ultra‐resolution tiled display walls interconnected by high speed networks. These systems require a globally connected and well‐integrated operating environment that provides persistent visualization and collaboration services. This abstract and subsequent presentation describes a new collaborative visualization system installed for NASA's Shortterm Prediction Research and Transition (SPoRT) program at Marshall Space Flight Center and its use for Earth science applications. The system consists of a 3 x 4 array of 1920 x 1080 pixel thin bezel video monitors mounted on a wall in a scientific collaboration lab. The monitors are physically and virtually integrated into a 14' x 7' for video display. The display of scientific data on the video wall is controlled by a single Alienware Aurora PC with a 2nd Generation Intel Core 4.1 GHz processor, 32 GB memory, and an AMD Fire Pro W600 video card with 6 mini display port connections. Six mini display‐to‐dual DVI cables are used to connect the 12 individual video monitors. The open source Scalable Adaptive Graphics Environment (SAGE) windowing and media control framework, running on top of the Ubuntu 12 Linux operating system, allows several users to simultaneously control the display and storage of high resolution still and moving graphics in a variety of formats, on tiled display walls of any size. The Ubuntu operating system supports the open source Scalable Adaptive Graphics Environment (SAGE) software which provides a common environment, or framework, enabling its users to access, display and share a variety of data‐intensive information. This information can be digital‐cinema animations, high‐resolution images, high‐definition video‐teleconferences, presentation slides, documents, spreadsheets or laptop screens. SAGE is cross‐platform, community‐driven, open‐source visualization and collaboration middleware that utilizes shared national and international cyberinfrastructure for the advancement of scientific research and education.