Engineering topics

Polymnia Manessis

Publications and source records attributed to Polymnia Manessis.

4 records

NASA NTRSDate not supplied

Reducing Barriers in Space Weather Research and Operations with Next-Generation Simulation Services at the Community Coordinated Modeling Center (CCMC)

Space weather forecasting capabilities are becoming increasingly important to the health of advanced technological infrastructure. The Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) serves as a key liaison in the US space weather program between the research and operations communities by providing a wide range of tools and capabilities that help to evaluate, compare, exercise, and archive the results of simulations of a growing list of space weather models. With its unique toolset, CCMC supports space weather research and model development that advances our understanding of space weather phenomena and improves forecasting skill, while also facilitating development of space weather applications and deployment of operational capabilities. Guided by experience from over 20 years of providing simulation services, feedback from its research, operational and educational users world-wide, recommendations from CCMC Advisory Group and Programmatic Review Panel, the CCMC has begun work on the next generation system for its simulation services and model output archives. The new system has been envisioned to employ state-of-the-art technologies and standards to provide a user-oriented experience while improving ease of access, transparency, interoperability with partner systems, and enhancing reliability by incorporating advanced automation for performance monitoring and intelligent failover. In the presentation, we will give an overview of the current CCMC ecosystem and discuss updates to some of the key services of the system, including Runs-on-Request, Instant Runs, and Continuous Runs. We will also describe how a planned expansion and standardization of data archival activities will enhance the role of CCMC as a world-class provider of heliophysics information for the research and analysis of space weather. It is our hope that this evolution of the services can further reduce the barriers and burdens on researchers, forecasters and decision makers who rely on CCMC for their daily research and operations.

Space Weather↗

NASA NTRSDate not supplied

Lowering Barriers to Science and Space Weather Research at the Community Coordinated Modeling Center (CCMC)

The Space Weather and Heliophysics research and modeling community has been pushing the limits of our ability to understand and predict space weather events. The Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) supports the community by providing a convenient collaborative platform hosting space weather models, model simulation data, curated datasets of solar events, and associated value-added services. Using these services, researchers and other end-users may exercise, evaluate, and intercompare contributed models, triage designated R2O models, as well as collaborate on a continuously updated archive of model run results. We will focus on CCMC’s ongoing commitment to the principles and guidelines of the Open Science initiative. Particularly, we will discuss our work towards making our services more transparent and our library of model simulations more accessible, open, and reproducible. We will introduce our recent tools for data discovery and correlative analysis designed to further increase the value of the user-generated data and metadata. We will also present our recent work on making heliophysical models more accessible and open to the community, particularly through simplified user experience and expert domain support. We will report on our progress in establishing an inter-center infrastructure with the ESA Virtual Space Weather Modelling Centre (VSWMC), designed to cross organizational boundaries and provide streamlined access to a joint palette of the models.

space weather↗

NASA NTRSDate not supplied

Open-Source Data Engineering at NASA: CCMC's Approach to Managing Petabyte-Scale Heliophysics Data

The Community Coordinated Modeling Center (CCMC) at NASA Goddard Space Flight Center (GSFC) leads heliophysics research by providing open access to numerous models and their outputs. Our resources are available on-demand and continuously updated with real-time data, covering sun-earth interactions across multiple domains. These domains include coronal, heliosphere, inner and global magnetosphere, ionosphere, thermosphere, and lower atmosphere interactions. Operating in a hybrid environment, CCMC utilizes both self-owned hardware and Amazon Web Services (AWS) cloud infrastructure. Managing petabytes of data across multiple locations necessitates robust data engineering solutions. To address this challenge, CCMC has adopted industry-standard and open-source tools. We use Apache Airflow as our primary data engineering platform, Python for scripting and data processing, and GitLab for version control and CI/CD. Additionally, we employ Kubernetes for containerized services, Grafana and Prometheus for metrics and monitoring, and Terraform and Puppet for reproducible infrastructure as code. This presentation will discuss lessons learned from our data engineering experiences, platforms evaluated but found unsuitable for our scientific data requirements, and specific techniques developed to enhance data transfer speed and reliability. By using these technologies effectively, CCMC continues to advance heliophysics research through efficient data management and open-access modeling.

space weather↗

NASA NTRSDate not supplied

Big Data Challenges at the Community Coordinated Modeling Center (CCMC)

Like other research centers, the Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) at NASA Goddard Space Flight Center (GSFC) is also experiencing the big data challenges. CCMC hosts over 80 space weather models for Runs On Request (ROR), Continuous Runs and Instant Runs simulation services for the research community. In addition, CCMC has started to support simulation output onboarding in response to the Open Science initiative. Overall, we have accumulated over petabytes of simulation output data and are rapidly growing. In this presentation, we will discuss our data and storage challenges. We will present our attempts to address our challenges and any associated lessons learned. CCMC uses Apache Airflow to ensure data transfer is consistent. We will give a brief overview on how we leverage Apache Airflow to enhance our environment.

space weather↗