Search NASA⌕ Search

SEARCH · Search NASA

Results for “kubernetes”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Smart connected worker edge platform for smart manufacturing: Part 1—Architecture and platform design

Abstract The challenge of sustainably producing goods and services for healthy living on a healthy planet requires simultaneous consideration of economic, societal, and environmental dimensions in manufacturing. Enabling technology for data driven manufacturing paradigms like Smart Manufacturing (a.k.a. Industry 4.0) serve as the technological backbone from which sustainable approaches to manufacturing can be implemented. Unfortunately, these technologies are typically associated with broader and deeper factory automation that is often too expensive and complex for the small and medium sized manufacturers (SMMs) that comprise the majority of manufacturing business in the USA and for whom their most valuable asset are the people whose jobs automation while replace. This paper describes an edge intelligent platform to integrate internet‐of‐things technologies with computing hardware, software, computational workflows for machine learning, and data ingestion, enabling SMMs to transition into smart manufacturing paradigms by leveraging the intelligence of their people. The platform leverages consumer grade electronics and sensors (affordable and portable), customized software with open source software packages (accessible), and existing communication network infrastructures (scalable). The software systems are implemented via Kubernetes orchestration of Docker containerization to ensure scalability and programmability. The platform is adaptive via computational workflow engines that produce information from data by processing with low‐cost edge computing devices while efficiently accessing resources of cloud servers as needed. The proposed edge platform connects workers to technological resources that provide computational intelligence (i.e., silicon‐based sensing and computation for data collection and contextualization) to enable decision making at the edge of advanced manufacturing.

Kim, Yoon G.↗
DOE OSTI2025

Hydra: An AI-Based Framework for Interpretable and Portable Data Quality Monitoring

Hydra is an advanced framework designed for training and managing AI models for near real time data quality monitoring at Jefferson Lab. Deployed in all four experimental halls, Hydra has analyzed over 2 million images and has extended its capabilities to offline monitoring and validation. Hydra utilizes computer vision to continually analyze sets of images of monitoring plots generated 24/7 during experiments. Generally, these sets of images are produced at a rate and quantity that is exceedingly difficult for shift crews to effectively monitor. Significant effort has been devoted to enhancing Hydra’s user interface, to ensure that it provides clear, actionable insights for shift workers and other users. Gradient Weighted Class Activation Maps (GradCAM) provide added interpretability, allowing users to visualize important regions of the image for classification. Hydra has been containerized to enable the creation of portable demos and seamless integration with container-based technologies such as Kubernetes and Docker. With the user interface enhancements and containerization, Hydra can be rapidly deployed for new use cases and experiments. This talk will describe the Hydra framework, its user interface and experience, and the challenges inherent in its design and deployment.

Britton, Thomas [Thomas Jefferson National Acceler↗
DOE OSTI2022

Real-time Multi-granular Analytics Framework for HIT Systems

Streaming analytics is the process of ingesting and digesting live data from multiple data sources. In the healthcare domain, as the importance of extracting immediate insights while data are streaming into the system grows, the focus is shifting from batch processing to streaming analytics. With data increasing dramatically at high speeds, many informatics designs have been proposed to adapt healthcare domain into this new environment. In our previous work, we introduced a prototype of health informatics technology (HIT) framework that aims to address challenges in adopting state-of-the-art technologies to enable advanced healthcare analytic tasks in new streaming environments. We recently made major updates to the framework so that anomaly from multiple streaming data sources at different granularity levels can be detected in near real-time. In this paper, we detail the implementation and deployment of the framework in Kubernetes clusters and report its performances when tested on electronic health record (EHR) data of Veterans Affairs.

Park, Byung↗
DOE OSTI2025

Network Anomaly Detection in Distributed Edge Computing Infrastructure

As networks continue to grow in complexity and scale, detecting anomalies has become increasingly challenging, particularly in diverse and geographically dispersed environments. Traditional approaches often struggle with managing the computational burden associated with analyzing large-scale network traffic to identify anomalies. This paper introduces a distributed edge computing framework that integrates federated learning with Apache Spark and Kubernetes to address these challenges. We hypothesize that our approach, which enables collaborative model training across distributed nodes, significantly enhances the detection accuracy of network anomalies across different network types. We show that by leveraging distributed computing and containerization technologies, our framework not only improves scalability and fault tolerance but also achieves superior detection performance compared to state-of-the-art methods. Extensive experiments on the UNSW-NB15 and ROAD datasets validate the effectiveness of our approach, demonstrating statistically significant improvements in detection accuracy and training efficiency over baseline models, as confirmed by MannWhitney U and Kolmogorov-Smirnov tests (p<0.05).

Marfo, William [University of Texas at El Paso,Dep↗
DOE OSTI2024

DICER: Data Intensive Computing Environment and Runtime for Evaluating Unprecedented Scale of Geospatial-Temporal Human Mobility Data

With the significant increase in sources and volume of human mobility data through commercial data vendors as well as microsimulation of cities, the scale of geospatial-temporal data to analyze and assess for mobility characterization has grown to the level of Big Data. There are mobility related commercial organizations deploying scalable computing, but often the system architecture, workflow, and intermediate processing components are not fully disclosed in relevant scope. Current research literature has a notable lack of studies demonstrating architectures and workflows for human mobility analytics that are implemented on a TeraByte scale of geospatial-temporal data. In this context, this paper presents a hyperscale-level system solution named DICER (Data Intensive Computing Environment and Runtime) for processing and analytics of geospatial-temporal data at big data scale. Although the cluster computing architecture of DICER with Apache Spark job running on Kubernetes cluster is not new, there are innovations in the workflow, hierarchical processing logic, and a wide range of intermediate preprocessing and mobility metrics calculation. We have performed case studies to validate the effectiveness of DICER system solution by performing detailed analytics and assessment of human mobility microsimulation output at three different scopes and scale, including a usecase with 16.97 TeraByte and 259.2 Billion rows of data. In addition, we have presented another case study of utilizing DICER to perform the same mobility processing and comparative analytics on large-scale commercially available geospatial-temporal data. All these case studies validate the efficiency and usefulness of DICER in computing population mobility characteristics from geospatial-temporal trajectory data at an unprecedented scale (not only just data volume, but also combination of: number of user entities, temporal frequency, spatial resolution, data duration).

De, Debraj↗
DOE OSTI2022

Elastic Resource Management for Deep Learning Applications in a Container Cluster

The increasing demand for learning from massive datasets is restructuring our economy. Effective learning, however, involves nontrivial computing resources. Most businesses utilize commercial infrastructure providers (e.g., AWS) to host their computing clusters in the cloud, where various jobs compete for available resources. While cloud resource management is a fruitful research field that has made many advances in production, such as Kubernetes and YARN, few efforts have been invested to further optimize the system performance, especially for deep learning (DL) training jobs in a container cluster. This work introduces FlowCon, a system that is able to monitor the individual evaluation functions of DL jobs at runtime, and thus to make placement decisions on resource allocations elastically. Here, we present a detailed design and implementation of FlowCon and conduct intensive experiments over various DL models. The results demonstrate that FlowCon significantly improves DL job completion time and resource utilization efficiency, compared to default systems. According to the results, FlowCon is able to improve the completion time by up to 68.8% and meanwhile, reduce the makespan by 18.0%, in the presence of various DL job workloads.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

HyperKube

A kubernetes based pipeline for analyzing hyperspectral data

Yoon, Hong Jun↗
DOE OSTI2023

Foldy v1

We created a cloud-based application for running AlphaFold2 and related structural programs called Foldy. Foldy is built on Helm / Kubernetes, which enables a facile production deployment and a scalable backend. Once set up by an institution, Foldy requires no software expertise from its users. It can be used to predict the structure of large proteins (up to 3000 amino acids), to visualize Pfam and antiSMASH annotations, and to perform ligand docking with Autodock Vina.

Roberts, Jacob↗
DOE OSTI2026

Janus v1.0

Janus provides a software framework for lightweight container management and orchestration. It's primary use cases are around deploying containerized services for high-performance data movement needs. Thus, Janus differentiates itself from systems like Kubernetes by tailoring the deployment of containers around network, storage, and host tuning optimizations. Janus uses the concept of profiles to capture repeatable deployment patterns and applies them to container execution across one or more endpoints. A Janus Agent component provides remote host tuning and monitoring capabilities.

Essiari, Abdelilah [Lawrence Berkeley National Lab↗
DOE OSTI2023

Orchestration of materials science workflows for heterogeneous resources at large scale

In the era of big data, materials science workflows need to handle large-scale data distribution, storage, and computation. Any of these areas can become a performance bottleneck. We present a framework for analyzing internal material structures (e.g., cracks) to mitigate these bottlenecks. We demonstrate the effectiveness of our framework for a workflow performing synchrotron X-ray computed tomography reconstruction and segmentation of a silica-based structure. Our framework provides a cloud-based, cutting-edge solution to challenges such as growing intermediate and output data and heavy resource demands during image reconstruction and segmentation. Specifically, our framework efficiently manages data storage, scaling up compute resources on the cloud. The multi-layer software structure of our framework includes three layers. A top layer uses Jupyter notebooks and serves as the user interface. A middle layer uses Ansible for resource deployment and managing the execution environment. A low layer is dedicated to resource management and provides resource management and job scheduling on heterogeneous nodes (i.e., GPU and CPU). At the core of this layer, Kubernetes supports resource management, and Dask enables large-scale job scheduling for heterogeneous resources. The broader impact of our work is four-fold: through our framework, we hide the complexity of the cloud’s software stack to the user who otherwise is required to have expertise in cloud technologies; we manage job scheduling efficiently and in a scalable manner; we enable resource elasticity and workflow orchestration at a large scale; and we facilitate moving the study of nonporous structures, which has wide applications in engineering and scientific fields, to the cloud. While we demonstrate the capability of our framework for a specific materials science application, it can be adapted for other applications and domains because of its modular, multi-layer architecture.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

ESGF project plan for ramping down development activities and transition the project and data to a new team

This document summarizes the current software development activities carried by the ESGF team with planned delivery dates before the transition of the project to a new team. The work we started on the publishing, search and retrieval, and the backend for indexing and data management that can be deployed using Kubernetes are planned to be delivered within the next three months; however, this assumes that the current team members will continue to work on the project. The activities are listed in Table 1 by priority and in case we lose team members, development of lower priority tasks can be paused and passed to the new team with its current status. Team members can be re-assigned based on their availability and transition work needed. Table 2 below describe activities necessary to help project transition to the new team. Deadlines for these activities are before October 1 st of 2022 or the completion of a successful transition. Support during the transition period assumes that we will have 3 key ESGF members and the ESGF PI supported at the 50% level or less for the duration of the transition. The level of support is an estimate, and it depends on several factors that cannot be decided at this time. The total cost of the proposed work and support activities for fiscal year 2021 is $1,270K with $230K carry over to fiscal year 2022. The cost for the support work listed in Table 2 is the sum of the two rows “HW support” and “project support, maintenance, and transition efforts”, colored blue, in Table1 which adds up to $720K. This cost is included in the $1,270K cost in table 1.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Distributed Resources for the Earth System Grid Advanced Management (DREAM). Final Report

The DREAM project was funded more than 3 years ago to design and implement a next generation ESGF (Earth System Grid Federation) architecture which would be suitable for managing and accessing data and services resources on a distributed and scalable environment. In particular, the project intended to focus on the computing and visualization capabilities of the stack, which at the time were rather primitive. At the beginning, the team had the general notion that a better ESGF architecture could be built by modularizing each component, and redefining its interaction with other components by defining and exposing a well defined API. Although this was still the high level principle that guided the work, the DREAM project was able to accomplish its goals by leveraging new practices in IT that started just about 3 or 4 years ago: the advent of containerization technologies (specifically, Docker), the development of frameworks to manage containers at scale (Docker Swarm and Kubernetes), and their application to the commercial Cloud. Thanks to these new technologies, DREAM was able to improve the ESGF architecture (including its computing and visualization services) to a level of deployability and scalability beyond the original expectations.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

FTS3: Data Movement Service in containers deployed in OKD [Slides]

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider's data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed a couple of FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. During this talk, we are going to present the two different configurations currently running at Fermilab, comparing and contrasting a Docker-based OKD deployment against a traditional RPM-based deployment and giving an overview of the possible issues encountered. In addition, we discuss our method of certificate management and maintenance utilizing Kubernetes cronjobs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

FTS3: Data Movement Service in containers deployed in OKD [Slides]

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider's data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. In this article we summarize our experience building docker images based on work from the SLATE project (slateci.io) and deployed in OKD, the community distribution of Red Hat OpenShift. Additionally, we discuss our method of certificate management and maintenance utilizing Kubernetes CronJobs. Finally, we also report on the two different configurations currently running at Fermilab, comparing and contrasting a Docker-based OKD deployment against a traditional RPM-based deployment.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Charliecloud is not affected by CVE-2024-21626 or related vulnerabilities

As you may be aware, four vulnerabilities in popular open-source container implementations were announced on January 21. Nicknamed “Leaky Vessels” by the Snyk Security Labs team that discovered them [1], these vulnerabilities in runC (CVE-2024-21626) and Moby BuildKit (CVE-2024-23651, CVE-2024-23652, and CVE-2024-23653) allow malicious container images or builds to execute arbitrary code on the container host with the privileges of the container runtime, i.e., a “container breakout”. Often, including typical configurations of Docker and/or Kubernetes, that means full root access.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Docker Containers for MCNP ® Development

Containers are a revolutionary technology in software development and deployment that provides a lightweight, portable environment for ensuring consistency across multiple computing environments. In anticipation of the MCNP 6.3.1 release, two Docker container images have been released on DockerHub for general use. The MCNP source code is not included in the images, and users are still required to obtain it through RSICC. The images produced by Docker are compliant with the OCI (Open Container Initiative) standards, ensuring compatibility with other container engines such as Podman or Kubernetes’ CRI-O. Initially, the images are stored under the author’s personal space on DockerHub (docker.io/azukaitis), but they will be relocated to a dedicated MCNP group space once approved. In the future, they will also be available through the registry feature of the https://github.com/lanl/mcnp-containers project. The use of Docker provides a pre-configured environment for building and running MCNP, ensuring reproducibility of results across various host architectures. This significantly improves consistency when running MCNP on different systems. Notably, executables and installers from the Docker images have successfully passed the MCNP development branch testing suite on x86-64 architectures, including Windows, macOS, and Linux operating systems. Furthermore, testing has demonstrated compatibility with macOS Docker in emulation mode on the latest Apple Mac M2 Ultra hardware, ensuring robust support even on the latest platforms. In this document, we will provide a step-by-step guide to using the Docker images across multiple platforms. Additionally, we will present performance numbers for building and running the MCNP test suite.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Geospatial Data Workflow Orchestration and Architecture

In an era characterized by explosive growth in geospatial data, the selection of appropriate technologies for data storage, processing, and orchestration is critical for organizations aiming to maintain competitive advantages. This white paper provides a comprehensive analysis of how Oak Ridge National Laboratory (ORNL) has effectively employed various cloud technologies, including containerized applications, container orchestrators, and workflow orchestrators, to develop robust geospatial data processing solutions. We explore the fundamental concepts behind these technologies and compare multiple deployment models tailored to diverse use cases. Our findings conclude that while Kubernetes has emerged as the preferred platform for truly scalable and fault-tolerant production workflows, the choice of workflow orchestration tool requires careful consideration of team needs, pipeline complexity, and deployment environments. This paper aims to serve as a strategic guide for organizations leveraging geospatial data, articulating the balance between technology choices and practical implementation to enhance workflow efficacy and scalability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

The Astronomy Commons Platform: A Deployable Cloud-based Analysis Platform for Astronomy

Abstract We present a scalable, cloud-based science platform solution designed to enable next-to-the-data analyses of terabyte-scale astronomical tabular data sets. The presented platform is built on Amazon Web Services (over Kubernetes and S3 abstraction layers), utilizes Apache Spark and the Astronomy eXtensions for Spark for parallel data analysis and manipulation, and provides the familiar JupyterHub web-accessible front end for user access. We outline the architecture of the analysis platform, provide implementation details and rationale for (and against) technology choices, verify scalability through strong and weak scaling tests, and demonstrate usability through an example science analysis of data from the Zwicky Transient Facility’s 1Bn+ light-curve catalog. Furthermore, we show how this system enables an end user to iteratively build analyses (in Python) that transparently scale processing with no need for end-user interaction. The system is designed to be deployable by astronomers with moderate cloud engineering knowledge, or (ideally) IT groups. Over the past 3 yr, it has been utilized to build science platforms for the DiRAC Institute, the ZTF partnership, the LSST Solar System Science Collaboration, and the LSST Interdisciplinary Network for Collaboration and Computing, as well as for numerous short-term events (with over 100 simultaneous users). In a live demo instance, the deployment scripts, source code, and cost calculators are accessible. 4 4 http://hub.astronomycommons.org/

79 ASTRONOMY AND ASTROPHYSICS↗