A mini grand challenge: build a verifiable filesystem
Explore the source record for details and available documents.
filesystems↗
SEARCH · Search NASA
Results for “filesystems”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Reducing Risk of InSight Surface Operations Through High-Fidelity Command Sequence Modeling
Simulating spacecraft behavior is crucial for the success of deep space missions, and failure to do so may result in damages to or the loss of the spacecraft. Many previous deep space missions have made use of ground-simulation of sequenced commanding, at speeds far greater than real time, to predict spacecraft state over time through the execution of onboard sequences. This type of modeling can be done at any fidelity, and most missions have opted to decrease fidelity to reduce cost and complexity. However, NASA’s Interior Exploration using Seismic Investigations, Geodesy and Heat Transport (InSight) mission expanded the scope of ground modeling considerably, which has led to numerous benefits over past implementations. This paper will discuss the process and products that InSight created, as well as the lessons learned from successfully operating the spacecraft on Mars. InSight is the first JPL mission to expand the scope of ground modeling to include the uplink of files from Earth to the spacecraft, rather than making the simplification that any command sequences already exist onboard the spacecraft. The advantages of modeling the uplink of files are numerous. First, it allows for accurate modeling of the onboard filesystem of the spacecraft at all points in time, meaning that all file loads and deletions throughout the mission are modeled at the exact moment they are predicted to actually happen. Second, operators can be more certain that dependencies between sequences are not broken due to the dynamic nature of the filesystem as files are deleted, copied, and uplinked. Lastly, spacecraft filesystem tracking allows for management of sequences prior to uplink, limiting the uplink to only new sequences. The onboard filesystem model became crucial to mission success, emphasizing the importance of investing in accurate models before the need for them arises. During daily tactical operations of a spacecraft on Mars, a model is only useful if the results can be interpreted quickly. In this fast-paced environment, it is essential that command products are modeled and reviewed, errors are found and diagnosed, and new command products are redelivered, remodeled, re-reviewed in a timely manner. It is impossible to review the entire model and therefore the results of the model must be condensed and presented in a fashion that is intuitive, easy-to-navigate, complete, and trustworthy. InSight developed a number of innovative sequence review products that are designed to provide operators with the information required to quickly assess the validity of command products and diagnose potential issues. Together, these products provide a complete, yet succinct picture of the command and sequence model to the operators and facilitate a quick assessment of all sequence command products. This paper will cover planning and sequencing innovations made during InSight surface operations, and will compare the tools, processes, and results to those on other missions. Additionally, the paper will cover the flexible, yet robust nature of the planning and sequencing system architecture and how that flexibility allowed for rapid development and response to the unpredictability of Mars.
Cloutier, Kyle↗
Vnode Layering Changes in NetBSD Version 1.5
In this paper, I describe changes to the vnode layer in NetBSD between versions 1.4 and 1.5 to assist layered filesystems. The major feature of these chano,es on the whole is that layered filesystems are now sufficiently robust for production applications. These changes divide readily into three groups: modifications to vnode locking along the lines suggested by Heidemann, changes to the vnode locking protocol, and the introduction of an overlay filesystem. In addition to describing these changes, I will describe the testing we have performed.
Studenmund, William↗
Reducing Backups by Utilizing DMF
Although a filesystem may be migratable, for a period of time the data blocks are on disk only. When performing system dumps, these data blocks are backed up to tape. If the data blocks are offline or dual resident, then only the inode is backed up. If all online files are made dual resident prior to performing system dumps, the dump time and the amount of resources required can be significantly reduced. The High Speed Processors group at the Numerical Aerodynamics Simulation (NAS) Facility at NASA Ames Research Center developed a tool to make all online files dual resident. The result is that a file whose data blocks are on DMF tape and still assigned to the original inode. Our 150GB filesystem used to take 8 to 12 hours to backup and used 50 to 60 tapes. Now the backup is typically under 10 tapes and completes in under 2 hours. This paper discusses this new tool and advantages gained by using it.
Cardo, Nicholas P.↗
Designing the User Experience for Earth Observation Data Services in the Cloud
NASA's Earth Observation (EO) inventory is projected to grow by an order of magnitude over the next 5-6 years. The current mode for working with EO data of downloading the data to a local machine (laptop, desktop or server) will be difficult to sustain for these upcoming volumes. Therefore, NASA is in the process of developing a capability to host large volume data in commercial cloud, with an eye toward encouraging data analysis in the cloud. However, in order for the user community to take advantage of this new mode of data interaction, the user experience must be redesigned. Cloud-hosted data brings new challenges, such as managing the costs of data egress and working with data in Web Object Storage instead of a Posix filesystem. However, it also brings new opportunities. Scaling data transformation processes may permit more synchronous data services with near-immediate response vs. cumbersome ordering systems with latencies of hours or days. Data co-location in the cloud can facilitate data integration and fusion. Highly scalable filesystems and databases in the cloud support data reorganization to facilitate analysis at scale. In the course of NASA's reimagining of the User Experience for EO data usage relies on end user input gathered through surveys, workshops and meetings (such as this). At the same time, we have embarked on a course of pedagogy and capacity building to help the user community evolve to cloud-based analysis.
Lynnes, Christopher↗
Programming your way out of the past: ISIS and the META Project
The ISIS distributed programming system and the META Project are described. The ISIS programming toolkit is an aid to low-level programming that makes it easy to build fault-tolerant distributed applications that exploit replication and concurrent execution. The META Project is reexamining high-level mechanisms such as the filesystem, shell language, and administration tools in distributed systems.
Birman, Kenneth P.↗
Managing People's Data
Just imagine a mass storage system that consists of a machine with 2 CPUs, 1 Gigabyte (GB) of memory, 400 GB of disk space, 16800 cartridge tapes in the automated tape silos, 88,000 tapes located in the vault, and the software to manage the system. This system is designed to be a data repository; it will always have disk space to store all the incoming data. Currently 9.14 GB of new data per day enters the system with this rate doubling each year. To assure there is always disk space available for new data, the system. has to move data reside from the expensive disk to a much less expensive medium such as the 3480 cartridge tapes. Once the data is archived to tape, it should be able to move back to disk when someone wants to access it and the data movement should be transparent to the user. Now imagine all the tasks that a system administrator must perform to keep this system running 24 hour a day, 7 days a week. Since the filesystem maintains the illusion of unlimited disk space, data that comes to the system must get moved to tapes in an efficient manner. This paper will describe the mass storage system running at the Numerical Aerodynamic Simulation (NAS) at NASA Ames Research Center in both software and hardware aspects, then it will describe all of the tasks the system administrator has to perform on this system.
Le, Diana↗
NASA Advanced Supercomputing Facility Expansion
The NASA Advanced Supercomputing (NAS) Division enables advances in high-end computing technologies and in modeling and simulation methods to tackle some of the toughest science and engineering challenges facing NASA today. The name "NAS" has long been associated with leadership and innovation throughout the high-end computing (HEC) community. We play a significant role in shaping HEC standards and paradigms, and provide leadership in the areas of large-scale InfiniBand fabrics, Lustre open-source filesystems, and hyperwall technologies. We provide an integrated high-end computing environment to accelerate NASA missions and make revolutionary advances in science. Pleiades, a petaflop-scale supercomputer, is used by scientists throughout the U.S. to support NASA missions, and is ranked among the most powerful systems in the world. One of our key focus areas is in modeling and simulation to support NASA's real-world engineering applications and make fundamental advances in modeling and simulation methods.
NASA↗
Methodology and Application of HPC I/O Characterization with MPIProf and IOT
Combining the strengths of MPIProf and IOT, an efficient and systematic method is devised for I/O characterization at the per-job, per-rank, per-file and per-call levels of HPC programs running on the NASA Advanced Supercomputing Center. This method is applied to answer four I/O questions in this paper. A total of 13 MPI programs and 15 cases, ranging from 24 to 5968 ranks, are analyzed to establish the I/O landscape from answers to the four questions. Four of the 13 programs use MPI I/O and the behavior of their collective writes depends on the specific implementation of the MPI library used. The SGI MPT library, the prevailing MPI library for our systems, was found to gather small writes from a large number of ranks to perform larger writes by a small subset of collective buffering ranks. The number of collective buffering ranks invoked by MPT depends on the Lustre stripe count and the number of nodes used for the run. A demonstration of varying the stripe count to achieve double-digit speedup of one program's I/O was presented. Another program, which concurrently opens private files by all ranks and could potentially create a heavy load on the Lustre servers, was identified. The ability to systematically characterize I/O for a large number of programs running on a supercomputer, seek I/O optimization opportunity and identify programs that could cause a high load and instability on the filesystems is important for pursuing exascale in a real production environment.
Characterization↗
Analysis Ready Data in Analytics Optimized Data Stores for Analysis of Big Earth Data in the Cloud
Cloud computing offers the possibility of making the analysis of Big Data approachable for a wider community due to affordable access to computing power, an ecosystem of usable tools for parallel processing, and migration of many large datasets to archives in the cloud, allowing data-proximal computing. Generally, data analysis acceleration in the cloud comes from running multiple nodes in a split-combine-apply strategy. Data systems such as the Earth Observing System Data and Information System are in a position to "pre-split" the data by storing them in a data store that is optimized for data parallel computing, i.e., an Analytics-Optimized Data Store (AODS). A variety of approaches to AODS are possible, from highly scalable databases to scalable filesystems to data formats optimized for cloud access (e.g., zarr and cloud-optimized datasets), with the optimal choice dependent on both the types of analysis and the geospatial structure of the data. A key question is how much preprocessing of the data to do, both before splitting and as the first part of the apply step. Again, the geospatial structure of the data and the analysis type influence the decision, with the added complexity of the user type. Trans-disciplinary users who are not well-versed in the nuances of quality-filtering and georeferencing of remote sensing orbit/swath/scene data tend to ask for more highly processed data, relying on the data provider to make sensible decisions on preprocessing parameters. (This accounts for the popularity of "Level 3" gridded data, despite the lower spatial resolution it provides.) In this case, data can be preprocessed before the split, resulting in higher performance in the rest of the "apply" step, which can be transformative for use cases such as interactive data exploration at scale. Discipline researchers who are experienced with remote sensing data often prefer more flexibility in customizing the preprocessing data into Analysis Ready Data, resulting in more need for on-the-fly preprocessing.
Lynnes, Christopher↗
Automatic variable selection in ecological niche modeling: A case study using Cassin’s Sparrow (Peucaea cassinii)
MERRA/Max provides a feature selection approach to dimensionality reduction that enables direct use of global climate model outputs in ecological niche modeling. The system accomplishes this reduction through a Monte Carlo optimization in which many independent MaxEnt runs, operating on a species occurrence file and a small set of randomly selected variables in a large collection of variables, converge on an estimate of the top contributing predictors in the larger collection. These top predictors can be viewed as potential candidates in the variable selection step of the ecological niche modeling process. MERRA/Max’s Monte Carlo algorithm operates on files stored in the underlying filesystem, making it scalable to large data sets. Its software components can run as parallel processes in a high-performance cloud computing environment to yield near real-time performance. In tests using Cassin’s Sparrow (Peucaea cassinii) as the target species, MERRA/Max selected a set of predictors from Worldclim’s Bioclim collection of 19 environmental variables that have been shown to be important determinants of the species’ bioclimatic niche. It also selected biologically and ecologically plausible predictors from a more diverse set of 86 environmental variables derived from NASA’s Modern-Era Retrospective Analysis for Research and Applications Version 2 (MERRA-2) reanalysis, an output product of the Goddard Earth Observing System Version 5 (GEOS-5) modeling system. We believe these results point to a technological approach that could expand the use global climate model outputs in ecological niche modeling, foster exploratory experimentation with otherwise difficult-to-use climate data sets, streamline the modeling process, and, eventually, enable automated bioclimatic modeling as a practical, readily accessible, low-cost, commercial cloud service.
John L. Schnase↗