SEARCH · Search NASA

Results for “Big data transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

17 records

All sources NASA NTRS DOE OSTI

NASA NTRSDate not supplied

Big-data Efficient Automated Science Transfer (BEAST): an open-source software architecture for arc jet data management, modeling, and automation

Big-data Efficient and Automated Science Transfer (BEAST) was conceived to address the existing ground testing data management of the NASA Ames arc jet facilities (e.g., manually entered Excel files and USB drive data transfers). These data management practices were seen as a choke point for future thermal protection system (TPS) development as they limit statistical tracking, resolution of diagnostics, coordination between video/time series, data throughput, and data processing speed/efficiency. Consequently, BEAST was developed to provide a new data infrastructure with streamlined data collection, processing, transfer, and analysis. This new framework also seeks to implement the FAIR principles of data stewardship: Findable, Accessible, Interoperable, and Reusable. The BEAST framework is based on a combination of the Python Django web framework and the Python data stack to provide a monolithic, open-source platform for data management, automation, and machine learning. This architecture was chosen for maintainability and scalability for a small, in-house development team. This paper will describe the application framework, deployment, and discuss the benefits and future plans for the system.

Data management↗

NASA NTRSDate not supplied

Big-data Efficient and Automated Science Transfer (BEAST): An Open-Source Software Architecture for Arc Jet Data Management, Modeling, and Automation

Data management↗

NASA NTRSDate not supplied

Big-data Efficient and Automated Science Transfer (BEAST): An Open-Source Software Architecture for Arc Jet Data Management, Modeling, and Automation

Big-data Efficient and Automated Science Transfer (BEAST) is a facility data management application developed for the NASA Ames arc jet facilities. The current decentralized data management practices limit statistical tracking, synchronization between video/time series, search capability, data throughput, and data processing speed/efficiency. Consequently, BEAST was developed to provide a new data infrastructure with streamlined data collection, processing, transfer, and analysis. This new framework also seeks to implement the FAIR principles of data stewardship: Findable, Accessible, Interoperable, and Reusable. The BEAST framework is based on a combination of the Python Django web framework and the Python data stack to provide a monolithic, open-source platform for data management, automation, and machine learning. This architecture was chosen for maintainability and scalability for a small, in-house development team. This paper will describe the application framework, deployment, and discuss the benefits and future plans for the system.

Data management↗

NASA NTRSDate not supplied

SWIPE: Spectral Water Inversion Processor and Emulator

Degradation of Earth’s inland water resources due to anthropogenic perturbations and climate anomalies at both local and global scales continues to place human health at substantial risk. There is now a growing necessity to develop pragmatic approaches that allow timely and effective extrapolation of local processes, to spatially resolved global products, and to promote operational and sustainable resource policy management. This presentation will be discussing the progress made developing SWIPE: Spectral Water Inversion Processor and Emulator. SWIPE is a platform for advanced modeling of coastal and inland aquatic habitats. The goal is create a comprehensive and cohesive system to leverage recent advancements in computation and machine learning to develop a synthetic training ground for sensitivity studies and algorithm development. The four principal facets of SWIPE include: 1. Advanced two-layer coated sphere bio-optical modeling and GPU radiative transfer modeling, 2. Big Data involving massive synthetic spectral libraries of optical properties of various global aquatic particles, surface reflectance, and top-of-atmosphere reflectance, all at hyperspectral resolution leveraging high-end computing systems at NASA Ames Research Center, 3. Deep Learning for algorithm development for water quality inversion of concentrations of common biogeophysical variables as well as optics, full uncertainty characterization by water type, and forward emulation, and lastly, 4. Image Processing for application of developed retrieval algorithms for both hyperspectral and multispectral sensors with experimental corrections for global adjacency, noise, sunglint, and benthic reflectance. This presentation will demonstrate the Equivalent Algal Populations (EAP) two-layer coated sphere scattering model which has been used develop spectral libraries of hyperspectral inherent optical properties of roughly 80 species of phytoplankton, covering 15 different classes and nine taxonomic functional types. The EAP model was also used to derive spectral properties of 10 different non-algal particle functional types. Examples of how the SMART-G (Speed-up Monte-carlo Advanced Radiative Transfer using GPU) radiative transfer code is used to model optically complex aquatic signals will be presented and discussed in the context of creating a massive synthetic database which can leverage the full power of next generation machine learning techniques and high end computing for water quality inversion. We will discuss our active investigation in things like appropriate model architectures, dimensionality reduction techniques such as PCA and autoencoders, uncertainty quantification and abstaining, and which variables actually benefit most from hyperspectral information versus multispectral resolution. We are also curious about questions relating to cost/benefit analysis in terms of computation resources, neural network complexity, and data volumes. Answers to these questions will hopefully elaborate on cost efficiency for potential future sensor design considerations.

SWIPE↗

NASA NTRS1992

Progress towards understanding and predicting convection heat transfer in the turbine gas path

A new era is drawing in the ability to predict convection heat transfer in the turbine gas path. We feel that the technical community now has the capability to mount a major assault on this problem, which has eluded significant progress for a long time. We hope to make a case for this bold statement by reviewing the state of the art in three major heat transfer, configuration-specific experiments, whose data have provided the big picture and guided both the fundamental modeling research and the code development. Following that, we review progress and directions in the development of computer codes to predict turbine gas path heat transfer. Finally, we cite examples and make observations on the more recent efforts to do all this work in a simultaneous, interactive, and more synergistic manner. We conclude with an assessment of progress, suggestions for how to use the current state of the art, and recommendations for the future.

Simoneau, Robert J.↗

NASA NTRS1993

Progress towards understanding and predicting heat transfer in the turbine gas path

A new era is dawning in the ability to predict convection heat transfer in the turbine gas path. We feel that the technical community now has the capability to mount a major assault on this problem, which has eluded significant progress for a long time. In this paper we hope to make a case for this bold statement by reviewing the state of the art in three major and related areas, which we believe are indispensable to the understanding and accurate prediction of turbine gas path heat transfer: configuration-specific experiments, fundamental physics and model development, and code development. We begin our review with the configuration-specific experiments, whose data have provided the big picture and guided both the fundamental modeling research and the code development. Following that, we examine key modeling efforts and comment on what will be needed to incorporate them into the codes. In this region we concentrate on bypass transition, 3D endwalls, and film cooling. We then review progress and directions in the development of computer codes to predict turbine gas path heat transfer. Finally, we cite examples and make observations on the more recent efforts to do all this work in a simultaneous, interactive, and more synergistic manner. We conclude with an assessment of progress, suggestions for how to use the current state of the art, and recommendations for the future.

Simoneau, Robert J.↗

NASA NTRS1993

RAID 7 disk array

Each RAID level reflects a different design architecture. Associated with each is a backdrop of imposed limitations, as well as possibilities which may be exploited within the architectural constraints of that level. There are three unique features that differentiate RAID 7 from all other levels. RAID 7 is asynchronous with respect to usage of I/O data paths. Each I/O drive (includes all data and one parity drives) as well as each host interface (there may be multiple host interfaces) has independent control and data paths. This means that each can be accessed completely, independently, of the other. This is facilitated by a separate device cache for each device/interface as well. RAID 7 is asynchronous with respect to device hierarchy and data bus utilization. Each drive and each interface is connected to a high speed data bus controlled by the embedded operating system to make independent transfers to and from central cache. RAID 7 is asynchronous with respect to the operation of an embedded real time process oriented operating system. This means that exclusive and independent of the host, or multiple host paths, the embedded OS manages all I/O transfers asynchronously across the data and parity drives. A key factor to consider is that of the RAID 7's ability to anticipate and match host I/O usage patterns. This yields the following benefits over RAID's built around micro-code based architectures. RAID 7 appears to the host as a normally connected Big Fast Disk (BFD). RAID 7 appears, from the perspective of the individual disk devices, to minimize the total number of accesses and optimize read/write transfer requests. RAID 7 smoothly integrates the random demands of independent users with the principles of spatial and temporal locality. This optimizes small, large, and time sequenced I/O requests which results in users having an I/O performance which approaches performance to that of main memory.

Stout, Lloyd↗

NASA NTRSDate not supplied

NASA’s Prototype Spectral Water Inversion Processor and Emulator (SWIPE): Towards Global Coastal and Inland Water Quality and Algal Biodiversity Monitoring

Degradation of Earth’s inland water resources due to anthropogenic perturbations and climate anomalies at both local and global scales continues to place human health at substantial risk. There is now a growing necessity to develop pragmatic approaches that allow timely and effective extrapolation of local processes, to spatially resolved global products, and to promote operational and sustainable resource policy management. This presentation will provide updates on NASA’s prototype open-source aquatic modeling platform, Spectral Water Inversion Processor and Emulator (SWIPE), which is a comprehensive, multi-faceted modeling platform for both forward and inverse modeling of diverse aquatic ecosystems from the benthos to top-of-atmosphere (TOA). SWIPE provides a cohesive application which leverages recent advancements in particle modeling, Big Data analytics, and machine learning to develop a high-fidelity synthetic training ground for sensitivity studies and algorithm development for multispectral or upcoming hyperspectral missions. Some of the prominent features of SWIPE to be discussed include: 1. Advanced hyperspectral modeling of globally diverse algal and non-algal particles using a novel two-layer coated sphere scattering model and radiative transfer modeling, 2. Massive, highly detailed synthetic spectral libraries of Analysis-Ready-Data (ARD) which include spectral libraries of particle microphysics, water biogeophysical and optical properties, as well as surface and TOA reflectances at 1 nm resolution, 3. An ensemble of pre-built analytic, machine learning, and deep learning inversion algorithms for various water quality and biodiversity related retrieval parameters and uncertainty quantification, 4. Sensor-agnostic water quality inversion at wide ranging spatial and spectral resolutions including a codebase for seamless application in the Google Earth Engine and NASA Earth Exchange (NEX) for planetary scale analysis. SWIPE will be a fully open-source platform based in python with comprehensive documentation, tutorials, and options for distributed computing on high performance computing clusters or on single, local machines. Further, we will discuss how we envision SWIPE contributing towards a global analysis of coastal and inland water quality dynamics.

top-of-atmosphere (TOA)↗

NASA NTRS2014

MERRA Analytic Services: Meeting the Big Data Challenges of Climate Science Through Cloud-enabled Climate Analytics-as-a-service

Climate science is a Big Data domain that is experiencing unprecedented growth. In our efforts to address the Big Data challenges of climate science, we are moving toward a notion of Climate Analytics-as-a-Service (CAaaS). We focus on analytics, because it is the knowledge gained from our interactions with Big Data that ultimately produce societal benefits. We focus on CAaaS because we believe it provides a useful way of thinking about the problem: a specialization of the concept of business process-as-a-service, which is an evolving extension of IaaS, PaaS, and SaaS enabled by Cloud Computing. Within this framework, Cloud Computing plays an important role; however, we it see it as only one element in a constellation of capabilities that are essential to delivering climate analytics as a service. These elements are essential because in the aggregate they lead to generativity, a capacity for self-assembly that we feel is the key to solving many of the Big Data challenges in this domain. MERRA Analytic Services (MERRAAS) is an example of cloud-enabled CAaaS built on this principle. MERRAAS enables MapReduce analytics over NASAs Modern-Era Retrospective Analysis for Research and Applications (MERRA) data collection. The MERRA reanalysis integrates observational data with numerical models to produce a global temporally and spatially consistent synthesis of 26 key climate variables. It represents a type of data product that is of growing importance to scientists doing climate change research and a wide range of decision support applications. MERRAAS brings together the following generative elements in a full, end-to-end demonstration of CAaaS capabilities: (1) high-performance, data proximal analytics, (2) scalable data management, (3) software appliance virtualization, (4) adaptive analytics, and (5) a domain-harmonized API. The effectiveness of MERRAAS has been demonstrated in several applications. In our experience, Cloud Computing lowers the barriers and risk to organizational change, fosters innovation and experimentation, facilitates technology transfer, and provides the agility required to meet our customers' increasing and changing needs. Cloud Computing is providing a new tier in the data services stack that helps connect earthbound, enterprise-level data and computational resources to new customers and new mobility-driven applications and modes of work. For climate science, Cloud Computing's capacity to engage communities in the construction of new capabilies is perhaps the most important link between Cloud Computing and Big Data.

Data Analytics↗

NASA NTRSDate not supplied

Improvements on Low-Density Parity-Check (LDPC) Codes and High-Performance Neuromorphic Engineering for Communication Systems

Belief propagation (BP) on LDPC codes is an iterative decoding algorithm that performs information transfer on the Tanner graph, which represents the code. In each iteration, the algorithm exchanges information (LLR) between variable nodes and check nodes through the edges of the graph. LLR values represent the probability that a given bit in a transmitted codeword equals 0 or 1, given a received word. In the hardware part, recent advancements in intelligent technologies, such as artificial intelligence, big data analytics, autonomous vehicles, and speech/image recognition, have heightened the demand for faster calculations and reduced energy consumption.

Danilo Barrionuevo↗

NASA NTRSDate not supplied

Big Data Challenges at CCMC

Like other research centers, the Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) at NASA Goddard Space Flight Center (GSFC) is also experiencing the big data challenges. CCMC hosts over 80 space weather models for Runs On Request (ROR), Continuous Runs and Instant Runs simulation services for the research community. In addition, CCMC has started to support simulation output onboarding in response to the Open Science initiative. Overall, we have accumulated over petabytes of simulation output data and are rapidly growing. In this presentation, we will discuss our data and storage challenges. We will present our attempts to address our challenges and any associated lessons learned. CCMC uses Apache Airflow to ensure data transfer is consistent. We will give a brief overview on how we leverage Apache Airflow to enhance our environment.

space weather↗

NASA NTRSDate not supplied

Big Data Challenges at the Community Coordinated Modeling Center (CCMC)

space weather↗

NASA NTRS2022

Cosmological constraints on dark matter interactions with ordinary matter

Dark matter interactions with electrons or protons during the early Universe leave imprints on the cosmic microwave background and the matter power spectrum, and can be probed through cosmological and astrophysical observations. These interactions lead to momentum and heat exchange between the ordinary and dark matter components, which in turn results in a transfer of pressure from the ordinary to the dark matter. We explore these interactions using a diverse suite of data: cosmic microwave background anisotropies, baryon acoustic oscillations, the Lyman-α forest, and the abundance of Milky-Way subhalos. We derive constraints using model-independent parameterizations of the dark matter–electron and dark matter–proton interaction cross sections and map these constraints onto concrete dark matter models. Our constraints are complementary to other probes of dark matter interactions with ordinary matter, such as direct detection, big bang nucleosynthesis, various astrophysical systems, and accelerator based experiments. They exclude sufficiently large cross sections for a large range of dark matter masses, which cannot be accessed by direct-detection experiments due to the overburden from the Earth’s atmosphere or crust.

Manuel A. Buen-Abad↗

NASA NTRS2001

Telemetry Tracking & Control (TT&C) - First TDRSS, then Commercial GEO & Big LEO and Now Through LEO

The advent of low earth orbit (LEO) commercial communication satellites provides an opportunity to dramatically reduce Telemetry, Tracking and Control (TT&C) costs of launch vehicles, Unpiloted Aerial Vehicles (UAVs), Research Balloons and spacecraft by reducing or eliminating ground infrastructure. Personnel from the Goddard Space Flight Center's Wallops Flight Facility (GSFC\WFF) have successfully used commercial Geostationary Earth Orbit (GEO) and Big LEO communications satellites for Long Duration Balloon Flight TT&C. The Flight Modem is a GSFC\WFF Advanced Range Technology initiative (ARTI) designed to streamline TT&C capability in the user community of these scientific data gathering platforms at low cost. Making use of existing LEO satellites and adapting and ruggedized commercially available components; two-way, over the horizon communications may be established with these vehicles at great savings due to reduced infrastructure. Initially planned as a means for permitting GPS data for tracking and recovery of sounding rocket and balloon payloads, expectations are that the bandwidth can soon be expanded to allow more comprehensive data transfer. The system architecture which integrates antennas, GPS receiver, commercial satellite packet data modem and a single board computer with custom software is described and technical challenges are discussed along with the plan for their resolution. A three-phase testing and development plan is outlined and the current results are reported. Results and status of ongoing flight tests on aircraft and sounding rockets are reported. Future applications on these platforms and the potential for satellite support are discussed along with an analysis of cost effectiveness of this method vs. other tracking and data transmission schemes.

Morgan, Dwayne R.↗

NASA NTRSDate not supplied

Assessment of Sensor Footprint Size and Comparison of Commercial Smallsat Images

Science users of commercial satellite data build their studies on the properties of the satellite data they work with, pixel size being one of the determining factors. However, pixel size and nominal image footprint size can be different, impacting the scale of features discernable in the satellite images. Here, we assess geometric properties such as nominal image footprint size for Planet Labs' SuperDove series, MAXAR's WorldView series, and potentially BlackSky. Additionally, we assess temporal change in nominal footprint size for the SuperDove series. Nominal Image footprint size is assessed over the CalVal sites in Baotou, China; Shadnagar, India; and Big Spring, TX. Edge spread functions are constructed along the black/white transitions, from which the line spread functions are constructed and modulation transfer functions of the scene are estimated. Scene full width half maximum (FWHM), which represents sensor footprint size, is estimated from the line spread function. The average nominal footprint size is 3.3 pixels for SuperDove series, 1.5 pixels for WorldView-2, and 1.3 pixels for WorldView-3. The SuperDove series nominal footprint size improves with time in orbit, from an average of 3.4 pixels soon after launch to an average of 3.2 pixels one or more years after launch.

PlanetScope↗

NASA NTRS1982

Contrast reduction by the atmosphere and retrieval of nonuniform surface reflectance

A radiative transfer model is developed which gives the upward radiance at nadir for any 1-D Lambertian surface reflectance. This model is used to depict the atmospheric effect on the transmittance of contrast for any 1-D surface reflectance. Here by contrast we mean a general variation of the radiation field across the image. With the aid of this model an inversion algorithm is developed for retrieval of true surface reflectance from high resolution satellite data (e.g., Landsat). This inversion technique can be a useful tool for extraction of surface reflectance from satellite data in the case of a surface reflectance variable in one dimension only (e.g., seashore or near borders of big fields). A sensitivity study of the inversion procedure on the knowledge of atmospheric parameters and sensor calibration was performed. It is shown that this inversion technique is stable even in the presence of errors in the sensor calibration and the atmospheric parameters. The method was applied to Landsat data in two wavelengths. The results show reasonable dependence of the derived surface reflectance on the distance from the seashore.

Mekler, Y.↗

NASA NTRS2017

Smashing the Stovepipe: Leveraging the GMSEC Open Architecture and Advanced IT Automation to Rapidly Prototype, Develop and Deploy Next-Generation Multi-Mission Ground Systems

Satellite/Payload Ground Systems - Typically highly-customized to a specific mission's use cases - Utilize hundreds (or thousands!) of specialized point-to-point interfaces for data flows / file transfers Documentation and tracking of these complex interfaces requires extensive time to develop and extremely high staffing costs Implementation and testing of these interfaces are even more cost-prohibitive, and documentation often lags behind implementation resulting in inconsistencies down the road With expanding threat vectors, IT Security, Information Assurance and Operational Security have become key Ground System architecture drivers New Federal security-related directives are generated on a daily basis, imposing new requirements on current / existing ground systems - These mandated activities and data calls typically carry little or no additional funding for implementation As a result, Ground System Sustaining Engineering groups and Information Technology staff continually struggle to keep up with the rolling tide of security Advancing security concerns and shrinking budgets are pushing these large stove-piped ground systems to begin sharing resources - I.e. Operational / SysAdmin staff, IT security baselines, architecture decisions or even networks / hosting infrastructure Refactoring these existing ground systems into multi-mission assets proves extremely challenging due to what is typically very tight coupling between legacy components As a result, many "Multi-Mission" ops. environments end up simply sharing compute resources and networks due to the difficulty of refactoring into true multi-mission systems Utilizing continuous integration / rapid system deployment technologies in conjunction with an open architecture messaging approach allows System Engineers and Architects to worry less about the low-level details of interfaces between components and configuration of systems GMSEC messaging is inherently designed to support multi-mission requirements, and allows components to aggregate data across multiple homogeneous or heterogeneous satellites or payloads - The highly-successful Goddard Science and Planetary Operations Control Center (SPOCC) utilizes GMSEC as the hub for it's automation and situational awareness capability Shifts focus towards getting GS to a final configuration-managed baseline, as well as multi-mission / big-picture capabilities that help increase situational awareness, promote cross-mission sharing and establish enhanced fleet management capabilities across all levels of the enterprise.

GMSEC↗