Search NASA⌕ Search

SEARCH · Search NASA

Results for “data infrastructures”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Developing a Vision for Heliophysics Infrastructure: The LIKED Resource and the DIARieS Ecosystem

Heliophysics data and computational infrastracture are not equipped for 21st science, suffering from holes in the know-how to build better systems. Without a clear vision, efforts to improve the infrastructure have been incremental and incoherent. This poster presents both the vision and the technology required: an online LIbrary KnowledgE and Discovery (LIKED) resource for discovering and implementing knowledge, data, and infrastructure resources; and an online analysis ecosystem to simplify Discovery, Implementation, Analysis, Reproducibility, and Sharing (DIARieS) of scientific results and environments. The LIKED and DIARieS solutions adopt FAIR data principles and the best practices from the budding field of open science. The proposed new infrastructure components will close many of the current gaps in heliophysics’ infrastructure, such as the ability to search for data and knowledge by phenomenon across domains, and to find software and examples relevant to the desired data set (including model data). Further, these components will enable community members to more efficiently use the resources already present and improve upon the content via a community-curated and trusted library. Combining these solutions lowers the barriers to heliophysics resources for all, increasing the return on our investments. Finally, the structure behind these ideas are topic-agnostic, so they are fully extensible to other fields, leading to invaluable connections to other disciplines. Just as with the development and construction of a long-term satellite mission, we must work together as a community to build a vision of the infrastructure that will most benefit the community, and then collaborate to construct, assemble, and test all the necessary pieces individually and as a unit. Our purpose in presenting this work is to not only describe the proposed vision, but also to gather feedback from the community on this topic.

infrastructure↗
NASA NTRS2001

A Space Based Internet Protocol System for Sub-Orbital Tracking and Control

Personnel from the Goddard Space Flight Center Wallops Flight Facility (GSFC/WFF) in Virginia are responsible for the overall management of the NASA Sounding Rocket Program. Payloads are generally in support of NASA's Space Science Enterprise's missions and return a variety of scientific data as well as providing a reasonably economical means of conducting engineering tests for instruments and devices used on satellites and other spacecraft. The fifteen types of sounding rockets used by NASA can carry payloads of various weights to altitudes from 50 km to more than 1,300 km. Launch activities are conducted not only from established missile ranges, but also from remote locations worldwide requiring mobile tracking and command equipment to be transported and set up at considerable expense. The advent of low earth orbit (LEO) commercial communications satellites provides an opportunity to dramatically reduce tracking and control costs of launch vehicles and Unpiloted Aerial Vehicles (UAVs) by reducing or eliminating this ground infrastructure. Additionally, since data transmission is by packetized Internet Protocol (IP), data can be received and commands initiated from practically any location. A low cost Commercial Off The Shelf (COTS) system is currently under development for sounding rockets which also has application to UAVs and scientific balloons. Due to relatively low data rate (9600 baud) currently available, the system will first be used to provide GPS data for tracking and vehicle recovery. Range safety requirements for launch vehicles usually stipulate at least two independent tracking sources. Most sounding rockets flown by NASA now carry GPS receivers that output position data via the payload telemetry system to the ground station. The Flight Modem can be configured as a completely separate link thereby eliminating requirement for tracking radar. The system architecture which integrates antennas, GPS receiver, commercial satellite packet data modem, and a single board computer with custom software is described along with the technical challenges and the plan for their resolution. These include antenna development, high Doppler rates, reliability, environmental ruggedness, hand over between satellites and data security. An aggressive test plan is included which in addition to environmental testing measures bit error rate, latency and antenna patterns. Actual flight tests are planned for the near future on aircraft, long duration balloons and sounding rockets and these results as well as the current status of the project are reported.

Bull, Barton↗
NASA NTRS2001

A Space Based Internet Protocol System for Launch Vehicle Tracking and Control

Personnel from the Goddard Space Flight Center Wallops Flight Facility (GSFC/WFF) in Virginia are responsible for the overall management of the NASA Sounding Rocket and Scientific Balloon Programs. Payloads are generally in support of NASA's Space Science Enterprise's missions and return a variety of scientific data as well as providing a reasonably economical means of conducting engineering tests for instruments and devices used on satellites and other spacecraft. Sounding rockets used by NASA can carry payloads of various weights to altitudes from 50 km to more than 1,300 km. Scientific balloons can carry a payload weighing as much as 3,630 Kg to an altitude of 42 km. Launch activities for both are conducted not only from established ranges, but also from remote locations worldwide requiring mobile tracking and command equipment to be transported and set up at considerable expense. The advent of low earth orbit (LEO) commercial communications satellites provides an opportunity to dramatically reduce tracking and control costs of these launch vehicles and Unpiloted Aerial Vehicles (UAVs) by reducing or eliminating this ground infrastructure. Additionally, since data transmission is by packetized Internet Protocol (IP), data can be received and commands initiated from practically any location. A low cost Commercial Off The Shelf (COTS) system is currently under development for sounding rockets that also has application to UAVs and scientific balloons. Due to relatively low data rate (9600 baud) currently available, the system will first be used to provide GPS data for tracking and vehicle recovery. Range safety requirements for launch vehicles usually stipulate at least two independent tracking sources. Most sounding rockets flown by NASA now carry GP~ receivers that output position data via the payload telemetry system to the ground station. The Flight Modem can be configured as a completely separate link thereby eliminating the requirement for tracking radar. The system architecture that integrates antennas, GPS receiver, commercial satellite packet data modem, and a single board computer with custom software is described along with the technical challenges and the plan for their resolution. These include antenna development, high Doppler rates, reliability, environmental ruggedness, hand over between satellites, and data security. An aggressive test plan is included which, in addition to environmental testing, measures bit error rate, latency and antenna patterns. Actual launches on a sounding rocket and various aircraft flights have taken place. Flight tests are planned for the near future on aircraft, long duration balloons and sounding rockets. These results, as well as the current status of the project, are reported.

Bull, Barton↗
NASA NTRSDate not supplied

Real-Time Science Decisioning During High Tempo-High Intensity Mission Operations and the Role of Analogs

Introduction: NASA’s VIPER mission presents a unique operational paradigm within the history of robotic spaceflight. The proximity of the Moon to the Earth and the terrain elements (surface characteristics, light/shadow dynamics, communication links) of the lunar South Polar landing site create unprecedented operational conditions between these two planetary bodies. Apollo era lunar science and exploration included humans in situ to operate instruments and assimilate observational inputs in real-time. Previous lunar orbital missions have worked to operational timescales, e.g., decisional timelines and communication exchanges, that were weeks in length. Mars rover missions have worked to operational timescales, e.g., decisional timelines and communication exchanges between Mars and Earth, that were hours, days, and weeks in length. In the case of the VIPER mission, our operational decisioning for rover driving and instrument commanding will be compressed to minute-scale timeframes. These operational conditions directly impact the manner and speed with which the VIPER Science Team (VST) is required to synthesize and analyze data and produce timely science-driven decisions throughout surface mission operations. The VST shall provide mission enhancing scientific input to guide rover traverse planning and drill site confirmation and selection throughout surface operations. Further, the VST input will be of vital importance to the mission’s ability to maximize science return and to meet broader NASA objectives for future lunar in-situ resource utilization (ISRU)and exploration activities. The VST co-located in the Mission Science Center (MSC) will be responsive to the tactical operational cadence of the Mission Operations Center (MOC) and will provide further strategic and Long-Term Planning (LTP) guidance to the mission. The VIPER Science Operations & Integration(SO&I)team has developed an architecture that is focused on the infusion of science-decisioning into the operational framework and execution cadence of VIPER. NASA analog research has played a significant role in the construction of the VIPER science operations systems. As an example, the SO&I team has led analog missions that have focused on bringing together expertise in the sciences (natural, applied and social) and in operations in service of learning how to build and hold together interdisciplinary work environments and what tools are needed to support high tempo, high intensity integrated decisioning. These experiences have provided an essential foundation of knowledge to the VIPER team. Those analogs that specifically influenced the VIPER science operations construct were identified through a process of comparative analysis to prioritize those that offered relevance in whole or in part, and those that did not. The analog research output that provided extensibility to the VIPER science operations architecture included remote teams of humans and robots in cooperation (synchronous and asynchronous) with simulated earthbound systems, engineering and science teams, and the integrated assembly of tools that supported scientific analysis and data synthesis and provided infrastructure for the remote testing framework. Analogs which included real-time data monitoring, synthesis, visualization and access in a democratized and operationalized manner were of particular interest to the development of the VIPER MSC toolset both in terms of the technology and the processes used to develop the supporting infrastructure. We anticipate that each subsequent mission to the lunar south pole, whether with robots or humans, will be able to optimize science and exploration return by evolving strategies to infuse real-time collaborative science-decisioning. Furthermore, these efforts will result in a foundation for science operations development in support of human-robotic exploration of deep space and Mars. NASA analogs can continue to provide the opportunity to prepare, test and iterate on the operational concepts and tools that will support these ever-expanding space exploration efforts. Our presentation will include an overview of the VIPER Science Operations & Integration development process and specifics on what aspects of analog research have had a significant impact on our work systems.

D S S Lim↗
NASA NTRSDate not supplied

Risk Characterization Research for Artemis II: Human Factors and Behavioral Performance

BACKGROUND Artemis II will be the first time NASA astronauts go beyond low-Earth orbit (LEO) since the Apollo era, and the first astronauts heading into space in the Orion vehicle. As such, it provides a critical opportunity to refine our understanding of the likelihood and consequences associated with the Behavioral Medicine (BMed), Team, Human System Integration Architecture (HSIA), and Sleep Risks, and prepare for future Moon and Mars missions. However, Artemis II research efforts are uniquely shaped by in-mission data collection constraints. There is currently no in-mission crew time available to complete measures. In-mission data will need to be collected unobtrusively from available data streams (e.g., audiovisual, existing records such as schedules, and actigraphy). Accordingly, the overarching goal of our research is to utilize Artemis II data to further define the likelihood and consequences of these risks, and to create an unobtrusive research infrastructure that can be expanded to include future Artemis missions. This goal spans four aims across three research phases: (1) identify and operationally define key performances metrics and constructs across the four aforementioned risks, (2) develop an unobtrusive methodology and coding scheme for in-mission data collection, (3) characterize performance decrements due to Bmed, Team, HSIA, and Sleep Risks, and (4) develop a data infrastructure for future Artemis missions. The following details results of Phase I efforts in which we address Aims 1 and 2 to develop an unobtrusive measurement plan and coding scheme to capture key constructs, contributing factors, and performance decrements across each risk area. METHOD As part of Phase I, we conducted an interdisciplinary literature review and consulted with SMEs to identify unobtrusive methodologies that leverage text, audio, and/or video data as well as conceptualize key performance metrics, contributing factors, and BMed, Team, HSIA, and Sleep risk constructs related to performance decrements. The Phase I effort resulted in a finalized pre- and post-mission protocol for Artemis II, along with a measurement and coding scheme for in-mission Artemis II data. Phase II will involve data collection from the upcoming Artemis II mission. Phase III will include data processing, coding, depiction, analysis, and report writing of the Artemis II data. RESULTS & DISCUSSION To date, we have completed Phase I efforts. Specifically, we identified BMed, Team, HSIA, and Sleep risk constructs related to performance metrics, summarized how these constructs can be measured using audiovisual data collected during the mission, and worked with NASA’s HFBP Element to finalize a data collection protocol that leverages audiovisual input from the Orion spacecraft system. Our protocol includes novel unobtrusive methodologies that adhere to in-mission data streams and subsequent constraints (e.g., limited storage space on GoPro cameras, ambient noise impeding audio files) to best capture in-mission phenomena across each risk area. We will present our results from Phase I efforts, namely best practices for unobtrusive measurement as identified through literature reviews and SME consultation as well as codebook excerpts for use in Artemis II. We will include a description of planned work as we prepare for Phase II and Phase III of this research plan and the Artemis II mission itself. SUMMARY We describe progress on our Human Factors and Behavioral Performance Research for Artemis II study.

behavioral health↗
NASA NTRS1991

SHARP: Spacecraft Health Automated Reasoning Prototype

The planetary spacecraft mission OPS as applied to SHARP is studied. Knowledge systems involved in this study are detailed. SHARP development task and Voyager telecom link analysis were examined. It was concluded that artificial intelligence has a proven capability to deliver useful functions in a real time space flight operations environment. SHARP has precipitated major change in acceptance of automation at JPL. The potential payoff from automation using AI is substantial. SHARP, and other AI technology is being transferred into systems in development including mission operations automation, science data systems, and infrastructure applications.

Atkinson, David J.↗
NASA NTRS2017

Aeronautical Situational Awareness - Airport Surface

This paper advocates for a specific design approach, based on simple principals, yet addresses challenges faced by the system engineers when designing complex data and information infrastructure. The document provides guidance for breaking out various work elements in the overall network architecture design, so that communication systems are conceived and effectively realized regardless of their location, size and local specifics. Although targeted at the Global Airspace System (GAS) and National Airspace System (NAS), this framework can be applied to any network-centric architecture.

networking↗
NASA NTRS2016

NASA Operational Simulator for Small Satellites: Tools for Software Based Validation and Verification of Small Satellites

The NASA Operational Simulator for Small Satellites (NOS3) is a suite of tools to aid in areas such as software development, integration test (IT), mission operations training, verification and validation (VV), and software systems check-out. NOS3 provides a software development environment, a multi-target build system, an operator interface-ground station, dynamics and environment simulations, and software-based hardware models. NOS3 enables the development of flight software (FSW) early in the project life cycle, when access to hardware is typically not available. For small satellites there are extensive lead times on many of the commercial-off-the-shelf (COTS) components as well as limited funding for engineering test units (ETU). Considering the difficulty of providing a hardware test-bed to each developer tester, hardware models are modeled based upon characteristic data or manufacturers data sheets for each individual component. The fidelity of each hardware models is such that FSW executes unaware that physical hardware is not present. This allows binaries to be compiled for both the simulation environment, and the flight computer, without changing the FSW source code. For hardware models that provide data dependent on the environment, such as a GPS receiver or magnetometer, an open-source tool from NASA GSFC (42 Spacecraft Simulation) is used to provide the necessary data. The underlying infrastructure used to transfer messages between FSW and the hardware models can also be used to monitor, intercept, and inject messages, which has proven to be beneficial for VV of larger missions such as James Webb Space Telescope (JWST). As hardware is procured, drivers can be added to the environment to enable hardware-in-the-loop (HWIL) testing. When strict time synchronization is not vital, any number of combinations of hardware components and software-based models can be tested. The open-source operator interface used in NOS3 is COSMOS from Ball Aerospace. For testing, plug-ins are implemented in COSMOS to control the NOS3 simulations, while the command and telemetry tools available in COSMOS are used to communicate with FSW. NOS3 is actively being used for FSW development and component testing of the Simulation-to-Flight 1 (STF-1) CubeSat. As NOS3 matures, hardware models have been added for common CubeSat components such as Novatel GPS receivers, ClydeSpace electrical power systems and batteries, ISISpace antenna systems, etc. In the future, NASA IVV plans to distribute NOS3 to other CubeSat developers and release the suite to the open-source community.

Verification↗
NASA NTRS2018

NOS3: NASA Operational Simulator for Small Satellites

The NASA Operational Simulator for Small Satellites (NOS3) is a suite of open-source software tools to aid in areas such as software development, integration & test (I&T), mission operations/training, verification and validation (V&V), and software systems check-out. NOS3 provides a software development environment, a multi-target build system, operational interface/ground software, dynamics and environment simulations, and software-based hardware models. NOS3 has just recently been open-sourced by NASA and is available for immediate use. It enables the development of flight software (FSW) early in the project life cycle when hardware availability is limited. Small satellite development suffers from extensive lead times on many of the commercial-off-the-shelf (COTS) components as well as limited funding for engineering test units (ETUs). To alleviate the need to provide a hardware test-bed for each developer/tester, NOS3 hardware models are based upon characteristic data or manufacturer's data sheets for each individual component. The NOS3 hardware models' fidelity is such that FSW executes unaware that physical hardware is not present. This allows FSW binaries to be compiled for both the simulation environment and the flight computer without changing the FSW source code. For hardware models that provide data which is dependent upon the environment and spacecraft dynamics, such as a GPS receiver or magnetometer, an open-source tool from NASA GSFC (42 Spacecraft Simulator) is used to provide the necessary data. The underlying infrastructure used to transfer messages between FSW and the hardware models can also be used to monitor, intercept, and inject messages, which has proven to be beneficial for V&V of larger missions such as James Webb Space Telescope (JWST). As hardware is selected and becomes available, drivers can be added to the NOS3 environment to enable hardware-in-the-loop (HWIL) testing. When strict time synchronization is not vital, any number of combinations of hardware components and software-based models can be tested. NOS3 was actively used for FSW development and component testing of the Simulation-to-Flight 1 (STF-1) CubeSat and the Lunar IceCube CubeSat. As NOS3 matures, hardware models have been added for common small satellite components such as GPS receivers, electrical power systems and batteries, and antenna systems.

Suder, Mark↗
NASA NTRSDate not supplied

Zero Trust and Identity Access Management in Support of Service-Based Urban Air Mobility Applications

Urban Air Mobility environments will contain of a collection of service-based services, which will be typically hosted within cloud infrastructures. The underlying data for these UAM services will need to be secured. One approach to securing these UAM services would be to leverage the Zero Trust framework, that focuses on securing services and associated data, instead of securing the network. An early step in moving towards a Zero Trust framework is to standardize identity access manage support for an ever-widening set of services, where users must explicitly be granted access to each service.

UAM↗
NASA NTRS1991

The Earth Observing System Data and Information System

The system is described in terms of the broad context of scientific cooperation and research which are facilitated by the architecture designed and the nature of the generated data. The Earth Observing System Data and Information System (EOSDIS) is composed of earth-science and applications data systems which archive global-change data and distribute the information to system users. Several archive centers catalogue selected Pathfinder data sets and generate new data products (such as maps of sea ice, cloud studies, etc.) with consistent calibration of all data. The schedule of three versions of EOSDIS is discussed with reference to the system architecture, programmatic plans, and data policy. The critical issue for EOSDIS is international cooperation to sponsor an infrastructure of global change data of this magnitude for public benefit and scientific research.

Maiden, Martha E.↗
NASA NTRS1998

AXAF: Current Developments and Future Plans for the Scientific Data Analysis

AXAF, the Advanced X-ray Astrophysics Facility, is one of NASA's great observatories scheduled for launch in January of 1999 by a Space Shuttle. This facility will be available to scientists in the United States and to the international astronomical community over an anticipated mission lifetime of at least 5 years. The objective of AXAF is to make astrophysical observations in the 0.09 to 10.0 keV energy range with two imaging detectors, and two sets of transmission gratings giving high spatial (0.5 arcsec) and spectral (E/(Delta)E=100-2000) resolution. The AXAF Science Center (ASC) in Cambridge, MA, USA is responsible for obtaining and reviewing observation proposals submitted by users, providing long range science planning and performing all science data calibration, data analysis, science instrument trend analysis, and science data archiving. The ASC will distribute science data products to the users. We will review here the current developments and future plans for the AXAF scientific data analysis system, its infrastructure and new key features, including the global modelling and fitting environment, integrated data analysis GUIs and new analysis tools.

Fruscione, Antonella↗
NASA NTRS2010

Updates of Land Surface and Air Quality Products in NASA MAIRS and NEESPI Data Portals

Following successful support of the Northern Eurasia Earth Sciences Partner Initiative (NEESPI) project with NASA satellite remote sensing data, from Spring 2009 the NASA GES DISC (Goddard Earth Sciences Data and Information Services Center) has been working on collecting more satellite and model data to support the Monsoon Asia Integrated Regional Study (MAIRS) project. The established data management and service infrastructure developed for NEESPI has been used and improved for MAIRS support.Data search, subsetting, and download functions are available through a single system. A customized Giovanni system has been created for MAIRS.The Web-based on line data analysis and visualization system, Giovanni (Goddard Interactive Online Visualization ANd aNalysis Infrastructure) allows scientists to explore, quickly analyze, and download data easily without learning the original data structure and format. Giovanni MAIRS includes satellite observations from multiple sensors and model output from the NASA Global Land Data Assimilation System (GLDAS), and from the NASA atmospheric reanalysis project, MERRA. Currently, we are working on processing and integrating higher resolution land data in to Giovanni, such as vegetation index, land surface temperature, and active fire at 5km or 1km from the standard MODIS products. For data that are not archived at the GESDISC,a product metadata portal is under development to serve as a gateway for providing product level information and data access links, which include both satellite, model products and ground-based measurements information collected from MAIRS scientists.Due to the large overlap of geographic coverage and many similar scientific interests of NEESPI and MAIRS, these data and tools will serve both projects.

Shen, Suhung↗
NASA NTRS2000

SHADOZ (Southern Hemisphere ADditional OZonesondes): An Ozonesonde Network and Resource for Remote Sensing Research and Education

Balloon-borne ozone instrumentation (ozonesondes), launched at fixed sites, is used to study local patterns in stratospheric and tropospheric ozone and to provide validation for satellite ozone products and model calculations of ozone. A paucity of coordinated ozonesonde data in the southern hemisphere tropics is being remedied in a 3-year project of coordinated ozonesondes launches at 10 sites. The data are available to the scientific community at the SHADOZ website at NASA/Goddard. Stations and their operational characteristics, with examples of ozone observations, are given. One expectation of SHADOZ is that wide dissemination of data and interaction with users and field projects will leverage local funding to maintain infrastructure and operations. SHADOZ data are well-suited for educational projects in which students learn about regional ozone patterns.

Thompson, Anne M.↗
NASA NTRS1999

A Science Information Infrastructure for Access to Earth and Space Science Data through the Nation's Science Museums

In this project, we worked with the University of California at Berkeley/Center for Extreme Ultraviolet Astrophysics and five science museums (the National Air and Space Museum, the Science Museum of Virginia, the Lawrence Hall of Science, the Exploratorium., and the New York Hall of Science) to formulate plans for computer-based laboratories located at these museums. These Science Learning Laboratories would be networked and provided with real Earth and space science observations, as well as appropriate lesson plans, that would allow the general public to directly access and manipulate the actual remote sensing data, much as a scientist would.

Murray, S.↗
NASA NTRS2014

Robust Informatics Infrastructure Required For ICME: Combining Virtual and Experimental Data

With the increased emphasis on reducing the cost and time to market of new materials, the need for robust automated materials information management system(s) enabling sophisticated data mining tools is increasing, as evidenced by the emphasis on Integrated Computational Materials Engineering (ICME) and the recent establishment of the Materials Genome Initiative (MGI). This need is also fueled by the demands for higher efficiency in material testing; consistency, quality and traceability of data; product design; engineering analysis; as well as control of access to proprietary or sensitive information. Further, the use of increasingly sophisticated nonlinear, anisotropic and or multi-scale models requires both the processing of large volumes of test data and complex materials data necessary to establish processing-microstructure-property-performance relationships. Fortunately, material information management systems have kept pace with the growing user demands and evolved to enable: (i) the capture of both point wise data and full spectra of raw data curves, (ii) data management functions such as access, version, and quality controls;(iii) a wide range of data import, export and analysis capabilities; (iv) data pedigree traceability mechanisms; (v) data searching, reporting and viewing tools; and (vi) access to the information via a wide range of interfaces. This paper discusses key principles for the development of a robust materials information management system to enable the connections at various length scales to be made between experimental data and corresponding multiscale modeling toolsets to enable ICME. In particular, NASA Glenn's efforts towards establishing such a database for capturing constitutive modeling behavior for both monolithic and composites materials

Mutli-scale models↗
NASA NTRSDate not supplied

Selecting Approaches for Enabling Enterprise Data Search: NASA’s Science Mission Directorate (SMD) Catalog

NASA’s Science Mission Directorate (SMD) is working to build an open-source science infrastructure to accelerate open, collaborative and interdisciplinary science. One key component in the open-source science infrastructure is the SMD data catalog. In this paper, we present our process for selecting a technical approach to building a NASA SMD enterprise-wide integrated search capability for science users across multiple science disciplines to support discovery and access to complex scientific data.

Kaylin Bugbee↗
NASA NTRSDate not supplied

Open-Source Data Engineering at NASA: CCMC's Approach to Managing Petabyte-Scale Heliophysics Data

The Community Coordinated Modeling Center (CCMC) at NASA Goddard Space Flight Center (GSFC) leads heliophysics research by providing open access to numerous models and their outputs. Our resources are available on-demand and continuously updated with real-time data, covering sun-earth interactions across multiple domains. These domains include coronal, heliosphere, inner and global magnetosphere, ionosphere, thermosphere, and lower atmosphere interactions. Operating in a hybrid environment, CCMC utilizes both self-owned hardware and Amazon Web Services (AWS) cloud infrastructure. Managing petabytes of data across multiple locations necessitates robust data engineering solutions. To address this challenge, CCMC has adopted industry-standard and open-source tools. We use Apache Airflow as our primary data engineering platform, Python for scripting and data processing, and GitLab for version control and CI/CD. Additionally, we employ Kubernetes for containerized services, Grafana and Prometheus for metrics and monitoring, and Terraform and Puppet for reproducible infrastructure as code. This presentation will discuss lessons learned from our data engineering experiences, platforms evaluated but found unsuitable for our scientific data requirements, and specific techniques developed to enhance data transfer speed and reliability. By using these technologies effectively, CCMC continues to advance heliophysics research through efficient data management and open-access modeling.

space weather↗