Search NASA⌕ Search

SEARCH · Search NASA

Results for “file format”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

TOAD Gateway

TOAD Gateway is interactive software tool for converting data files to and from variety of file formats. Currently reads and writes following file formats: TOAD; Standard Interface File (SIF); Program to Optimize Simulated Trajectories (POST) input; Comma Separated Value and Tab Separated Value, common in PC and Macintosh spreadsheet and database packages; and general free format. Additional modules for accommodating other formats easily developed and installed. Companion program, TOAD Editor (LAR-14423), manipulates contents of TOAD files and extracts selected subsets of data. TOAD Gateway written in FORTRAN 77.

Bingel, Bradford D.↗

TADPLOT program, version 2.0: User's guide

The TADPLOT Program, Version 2.0 is described. The TADPLOT program is a software package coordinated by a single, easy-to-use interface, enabling the researcher to access several standard file formats, selectively collect specific subsets of data, and create full-featured publication and viewgraph quality plots. The user-interface was designed to be independent from any file format, yet provide capabilities to accommodate highly specialized data queries. Integrated with an applications software network, data can be assessed, collected, and viewed quickly and easily. Since the commands are data independent, subsequent modifications to the file format will be transparent, while additional file formats can be integrated with minimal impact on the user-interface. The graphical capabilities are independent of the method of data collection; thus, the data specification and subsequent plotting can be modified and upgraded as separate functional components. The graphics kernel selected adheres to the full functional specifications of the CORE standard. Both interface and postprocessing capabilities are fully integrated into TADPLOT.

Hammond, Dana P.↗

Adaptable Metadata Rich IO Methods for Portable High Performance IO

Since IO performance on HPC machines strongly depends on machine characteristics and configuration, it is important to carefully tune IO libraries and make good use of appropriate library APIs. For instance, on current petascale machines, independent IO tends to outperform collective IO, in part due to bottlenecks at the metadata server. The problem is exacerbated by scaling issues, since each IO library scales differently on each machine, and typically, operates efficiently to different levels of scaling on different machines. With scientific codes being run on a variety of HPC resources, efficient code execution requires us to address three important issues: (1) end users should be able to select the most efficient IO methods for their codes, with minimal effort in terms of code updates or alterations; (2) such performance-driven choices should not prevent data from being stored in the desired file formats, since those are crucial for later data analysis; and (3) it is important to have efficient ways of identifying and selecting certain data for analysis, to help end users cope with the flood of data produced by high end codes. This paper employs ADIOS, the ADaptable IO System, as an IO API to address (1)-(3) above. Concerning (1), ADIOS makes it possible to independently select the IO methods being used by each grouping of data in an application, so that end users can use those IO methods that exhibit best performance based on both IO patterns and the underlying hardware. In this paper, we also use this facility of ADIOS to experimentally evaluate on petascale machines alternative methods for high performance IO. Specific examples studied include methods that use strong file consistency vs. delayed parallel data consistency, as that provided by MPI-IO or POSIX IO. Concerning (2), to avoid linking IO methods to specific file formats and attain high IO performance, ADIOS introduces an efficient intermediate file format, termed BP, which can be converted, at small cost, to the standard file formats used by analysis tools, such as NetCDF and HDF-5. Concerning (3), associated with BP are efficient methods for data characterization, which compute attributes that can be used to identify data sets without having to inspect or analyze the entire data contents of large files.

97 MATHEMATICS AND COMPUTING↗

Characterizing Impacts of Storage Faults on HPC Applications: A methodology and insights

In recent years, the increasing complexity in scientific simulations and emerging demands for training heavy artificial intelligence models require massive and fast data accesses, which urges high-performance computing (HPC) platforms to equip with more advanced storage infrastructures such as solid-state disks (SSDs). While SSDs offer high-performance I/O, it remains unclear about the reliability challenges faced by the HPC applications under the SSD-related failures, in particular, failures resulting in data corruptions. The goal of this paper is to understand the impact of SSD-related data corruptions on the behaviors of complex HPC applications. To this end, we propose FFIS, a FUSE-based fault injection framework that systematically introduces storage faults into the application layer to model the errors originated from SSDs. FFIS is able to plant different I/O related faults into the data returned from underlying file systems, which also enables the investigation on the error resilience characteristics of the scientific file format for the first time. We demonstrate the use of FFIS with three representative real HPC applications, show how each application reacts to the data corruptions, and provide insights on the error resilience of the widely-adopted HDF5 file format for the HPC applications.

Fang, Bo↗

ArrayBridge: Interweaving declarative array processing with high-performance computing

Scientists are increasingly turning to datacenter-scale computers to produce and analyze massive arrays. Despite decades of database research that extols the virtues of declarative query processing, scientists still write, debug and parallelize imperative HPC kernels even for the most mundane queries. This impedance mismatch has been partly attributed to the cumbersome data loading process; in response, the database community has proposed in situ mechanisms to access data in scientific file formats. Scientists, however, desire more than a passive access method that reads arrays from files. This paper describes ArrayBridge, a bi-directional array view mechanism for scientific file formats, that aims to make declarative array manipulations interoperable with imperative file-centric analyses. Our prototype implementation of ArrayBridge uses HDF5 as the underlying array storage library and seamlessly integrates into the SciDB open-source array database system. In addition to fast querying over external array objects, ArrayBridge produces arrays in the HDF5 file format just as easily as it can read from it. ArrayBridge also supports time travel queries from imperative kernels through the unmodified HDF5 API, and automatically deduplicates between array versions for space efficiency. Our extensive performance evaluation in NERSC, a large-scale scientific computing facility, shows that ArrayBridge exhibits statistically indistinguishable performance and I/O scalability to the native SciDB storage engine.

97 MATHEMATICS AND COMPUTING↗

Vulcan Digital Engineering Application

The application is being developed with the goal of preventing the entropy of all the different single purpose applications. This application utilizes a framework that enables different file formats to be read and understood through plugins dynamically loaded to the main application containing the framework. The main application provides interoperability to all the different plugins, this allows each file format to convert to every other file format that has been read. There is no theoretical limit to the number of plugins that can be used by the application. Even though the plugins may be individually written and serve a purpose like single use scripts, the plugins can only be used with the main application, this forces the application and plugins to be linked. In theory this will force better data management, or at least the bundling of the main application with the plugins.

Lehmer, JacobP↗

Thrifty Array Format (TAF) file specifications

Thrifty Array Foram (TAF) files store numeric data in a binary format, minimizing storage requirements while preserving quick read access. Real data of any size and dimensionality can be stored in this format at varying degrees of numeric precision. Implicit array are associated with each dimension, eliminating the need to explicitly store uniformly-spaced grid vectors. Unlimited text comments may be included with the array for user documentation, and every file begins with a text synopsis of the binary structure. The format is deliberately designed for memory mapping, where portions of the array can be read without loading the entire file at once.

97 MATHEMATICS AND COMPUTING↗

Tussock tundra surface temperatures, ambient air and incoming photosynthetically active radiation measured at the NGEE Arctic Council site, 2021 - 2023

This dataset contains temperature measurements carried out along two fiber optics cables/lines (150 m each) laid out along the ground at the Next Generation Ecosystem Experiment (NGEE) Arctic site near Council, Alaska. The lines traverse an heterogeneous part of the tussock tundra site including thermokarst features and lichen dominated sections of the tundra. Measurements were done using a Sensornet Oryx DTS, a Distributed Temperature sensor that was installed in September 2021 and taken down in August 2023. The sensor was powered by solar power with data being collected every 30 minutes at 1 m resolution. In addition to these measurements air temperature and incoming photosynthetically active radiation (PAR) are provided. These measurements are co-located with the NGEE Arctic Council eddy flux and meteorological station (AmeriFlux ID US-NGC). Included are six *.csv files (four data files and two reporting format files) and two *.kml files. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

Air temperature↗

Converting from DDOR SASF to APF

A computer program called ddor_sasf2apf converts delta-door (delta differential one-way range) request from an SASF (spacecraft activity sequence file) format to an APF (apgen plan file) format for use in the Mars Reconnaissance Orbiter (MRO) missionplanning- and-sequencing process. The APF is used as an input to APGEN/AUTOGEN in the MRO activity- planning and command-sequencegenerating process to sequence the delta-door (DDOR) activity. The DDOR activity is a spacecraft tracking technique for determining spacecraft location. The input to ddor_sasf2apf is an input request SASF provided by an observation team that utilizes DDOR. ddor_sasf2apf parses this DDOR SASF input, rearranging parameters and reformatting the request to produce an APF file for use in AUTOGEN and/or APGEN. The benefit afforded by ddor_sasf2apf is to enable the use of the DDOR SASF file earlier in the planning stage of the command-sequence-generating process and to produce sequences, optimized for DDOR operations, that are more accurate and more robust than would otherwise be possible.

Gladden, Roy E.↗

An optimal user-interface for EPIMS database conversions and SSQ 25002 EEE parts screening

The Electrical, Electronic, and Electromechanical (EEE) Parts Information Management System (EPIMS) database was selected by the International Space Station Parts Control Board for providing parts information to NASA managers and contractors. Parts data is transferred to the EPIMS database by converting parts list data to the EP1MS Data Exchange File Format. In general, parts list information received from contractors and suppliers does not convert directly into the EPIMS Data Exchange File Format. Often parts lists use different variable and record field assignments. Many of the EPES variables are not defined in the parts lists received. The objective of this work was to develop an automated system for translating parts lists into the EPIMS Data Exchange File Format for upload into the EPIMS database. Once EEE parts information has been transferred to the EPIMS database it is necessary to screen parts data in accordance with the provisions of the SSQ 25002 Supplemental List of Qualified Electrical, Electronic, and Electromechanical Parts, Manufacturers, and Laboratories (QEPM&L). The SSQ 2S002 standards are used to identify parts which satisfy the requirements for spacecraft applications. An additional objective for this work was to develop an automated system which would screen EEE parts information against the SSQ 2S002 to inform managers of the qualification status of parts used in spacecraft applications. The EPIMS Database Conversion and SSQ 25002 User Interfaces are designed to interface through the World-Wide-Web(WWW)/Internet to provide accessibility by NASA managers and contractors.

Watson, John C.↗

Biomass sorghum production in Iowa, USA

This dataset includes field measurement and model simulation data from Josh Bendorf's (former MS student with Drs. VanLoocke & Heaton, ISU Dept. of Agronomy) thesis project focusing on the simulation of biomass sorghum ( Sorghum bicolor L. Moench) in the state of Iowa. Measured field data are provided in CSV file format for 2019 & 2020. Model output files are provided as NetCDF file format for the state of Iowa (1998-2007) & the larger US region (1969-2018).

Agricultural production systems simulation↗

Data from: "Lab and Field Warming Similarly Advance Germination Date and Limit Germination Rate for High and Low Elevation Provenances of Two Widespread Subalpine Conifers"

This archive contains data used to draw conclusions in “Lab and Field Warming Similarly Advance Germination Date and Limit Germination Rate for High and Low Elevation Provenances of Two Widespread Subalpine Conifers”, by Kueppers et al. 2017. Cone collection and field experiments were conducted on Niwot Ridge, Colorado, with the latter completed within the Alpine Treeline Warming Experiment (ATWE). Laboratory germination experiments were conducted at the University of California Merced. Geospatial files are also provided for additional geographical context.There are four data file formats in this package: comma-separated values (.csv), Microsoft Excel (.xlsx), keyhole markup language (.kml), and ESRI shapefile (.shp). The latter two are geospatial file formats. .csv files can be opened using any simple text editor program such as TextEdit (Mac) and Notepad (Windows), and .xlsx files can be opened using Microsoft Excel or Google Sheets. The .kml file contains coordinate data, and is compatible with Google Earth and Google Maps. ESRI shapefiles (.shp) are compatible with any geospatial software able to read the file type, such as QGIS and ESRI’s ArcGIS Suite. This data user’s guide is available in as a .pdf, and can be opened using software such as Adobe Acrobat Reader.-----------------------------------------------------------------------------------------------------------------------------------Accurately predicting upslope shifts in subalpine tree ranges with warming requires understanding how future forest populations will be affected by climate change, as these are the seed sources for new tree line and alpine populations. Early life history stages are particularly sensitive to climate and are also influenced by genetic variation among populations. We tested the climate sensitivity of germination and initial development for two widely distributed subalpine conifers, using controlled-environment growth chambers with one temperature regime from subalpine forest in the Colorado Rocky Mountains and one 5 °C warmer, and two soil moisture levels. We tracked germination rate and timing, rate of seedling development, and seedling morphology for two seed provenances separated by ~300 m elevation. Warming advanced germination timing and initial seedling development by a total of ~2 weeks, advances comparable to mean differences between provenances. Advances were similar for both provenances and species; however, warming reduced the overall germination rate, as did low soil moisture, only for Picea engelmannii. A three-year field warming and watering experiment planted with the same species and provenances yielded responses qualitatively consistent with the lab trials. Together these experiments indicate that in a warmer, drier climate, P. engelmannii germination, and thus regeneration, could decline, which could lead to declining subalpine forest populations, while Pinus flexilis forest populations could remain robust as a seed source for upslope range shifts.

54 ENVIRONMENTAL SCIENCES↗

JSATS Decoder Software Manual

The Juvenile Salmon Acoustic Telemetry System (JSATS) Decoder is a software application that converts a digitized acoustic signal (a waveform stored in the .bwm file format) into a list of potential JSATS Acoustic MicroTransmitter (AMT) tagcodes along with other data about the signal including time of arrival and signal to noise ratios (SNR). This software is capable of decoding single files, directories, and viewing raw acoustic waveforms. When coupled with the JSATS Detector, the Decoder is capable of decoding in ‘real-time’ and can also provide statistical information about acoustic beacons placed within receive range of hydrophones within a JSATS array. This document details the features and functionality of the software. The document begins with software installation instructions (section 2), followed in order by instructions for decoder setup (section 3), decoding process initiation (section 4), then monitoring of beacons (section 5) using real-time decoding features. The last section in the manual describes the beacon, beacon statistics, and the results file formats. This document does not consider the raw binary waveform file format.

Flory, Adam E.↗

ASDC’s Python-Based Metadata Extraction Pipeline for Suborbital Campaigns

The FAIRness of data products, especially findability and accessibility depend on rich metadata which, when extracted, can allow for proper curation. Over the past few years, the Atmospheric Science Data Center (ASDC) suborbital science support team has developed a metadata extraction pipeline to ensure the required metadata can be retrieved systematically, effectively, and efficiently to ensure the data can be used by a broad community. The development of a pipeline has presented many, but necessary, challenges to support archival and distribution of ASDC’s 30+ suborbital missions. Though sufficient metadata is provided by instrument scientists, the metadata may not be readily machine actionable due to different formats and templates. Further complicating metadata extraction, our team has found that the nature of metadata can be quite diverse given the difference in measurement types, instruments, and measurement platforms. A metadata extraction pipeline has been developed to provide an efficient, plugin-in based, method for adding new parsers, a configuration system that lets non-developers customize how files are processed, and a system for identifying and logging metadata quality issues to ensure they are readily found and addressed. The metadata extraction pipeline identifies critical pieces of metadata that are needed to promote data FAIRness, including location, file revision, measurement start/end datetime and can be easily modified to extract further information (such as variables). Given the wide-ranging datasets, the pipeline has been modified to accommodate multiple file formats, including multiple versions of ICARTT (International Consortium for Atmospheric Research on Transport and Transformation), HDF (Hierarchical Data Format), netCDF (network Common Data Form), and multiple versions of the Ames File Format. The pipeline also supports building metadata for file formats that cannot have metadata easily extracted from them, such as PDF (Portable Document Format) and GIF (Graphics Interchange Format). The pipeline has allowed our team to maintain a consistent flow of data and metadata to archival and distribution services, ensuring the ASDC meets the needs of the suborbital science community. This presentation will highlight the ASDC’s suborbital metadata extraction pipeline, its development, how it’s been modified to support data FAIRness, and plans for maintaining the pipeline and adding new features.

Abraham Porter↗

Deep PDF parsing to extract features for detecting embedded malware.

The number of PDF files with embedded malicious code has risen significantly in the past few years. This is due to the portability of the file format, the ways Adobe Reader recovers from corrupt PDF files, the addition of many multimedia and scripting extensions to the file format, and many format properties the malware author may use to disguise the presence of malware. Current research focuses on executable, MS Office, and HTML formats. In this paper, several features and properties of PDF Files are identified. Features are extracted using an instrumented open source PDF viewer. The feature descriptions of benign and malicious PDFs can be used to construct a machine learning model for detecting possible malware in future PDF files. The detection rate of PDF malware by current antivirus software is very low. A PDF file is easy to edit and manipulate because it is a text format, providing a low barrier to malware authors. Analyzing PDF files for malware is nonetheless difficult because of (a) the complexity of the formatting language, (b) the parsing idiosyncrasies in Adobe Reader, and (c) undocumented correction techniques employed in Adobe Reader. In May 2011, Esparza demonstrated that PDF malware could be hidden from 42 of 43 antivirus packages by combining multiple obfuscation techniques [4]. One reason current antivirus software fails is the ease of varying byte sequences in PDF malware, thereby rendering conventional signature-based virus detection useless. The compression and encryption functions produce sequences of bytes that are each functions of multiple input bytes. As a result, padding the malware payload with some whitespace before compression/encryption can change many of the bytes in the final payload. In this study we analyzed a corpus of 2591 benign and 87 malicious PDF files. While this corpus is admittedly small, it allowed us to test a system for collecting indicators of embedded PDF malware. We will call these indicators features throughout the rest of this report. The features are extracted using an instrumented PDF viewer, and are the inputs to a prediction model that scores the likelihood of a PDF file containing malware. The prediction model is constructed from a sample of labeled data by a machine learning algorithm (specifically, decision tree ensemble learning). Preliminary experiments show that the model is able to detect half of the PDF malware in the corpus with zero false alarms. We conclude the report with suggestions for extending this work to detect a greater variety of PDF malware.

99 GENERAL AND MISCELLANEOUS↗

Rapid Model Import Tool (RMIT)

Our project is about developing a tool to implement conversion of 3D Computer Aided Design (CAD) models produced with software such as Delmia, 3DS Max, or Maya, into a size and format compatible with the Unity 3D environment. RMIT will be used to aid KSC engineering personnel in the design, development, testing, operations, and training on spacecraft, launch vehicles, facilities, and ground support equipment. For our project, we are using Blender, a free/open-source 3D graphics software, with the goal of developing, testing, and deploying a 3D CAD model converter tool. I worked on using Blender to import 3D CAD models exported from CATIA software into Collada file format. The Collada file format has file extension DAE. Importing the Collada DAE file as is into Blender, generates dots and dashes. In 3DS Max, there is an existing OpenCollada plugin and with that plugin, 3DS Max can import the DAE file successfully. But, Blender does not seem to have an OpenCollada plugin, so I worked on writing a new OpenCollada plugin for Blender. Since 3DS Max was able to display the image, I looked into comparing differences between the original DAE file and the DAE file exported from 3DS Max using the OpenCollada plugin. As Collada documents describing digital assets are XML files with file extension DAE, Collada files contains XML tags, making them easily modifiable. After some research, it appears that Blender does not like primitive 2D tags like tristrips and trifans. Changing those tags to polygons slightly improved the image, but the pieces were exploded. I found after further research comparing differences between the original file and the file exported from 3DS Max that the values inside the translate tags in the original file are scaled down by a factor of 25.4 in the exported file from 3DS Max, representing the millimeters to inches conversion (1 inch = 25.4 millimeters). After scaling down values inside all of the translate tags by 25.4, the exploded pieces stuck back in, but the image needed further improvement. I have been able to create a new plugin in Blender that takes the original DAE file, replaces the primitive 2D tags tristrips and trifans with polygons, scales down the values inside the translate tags by a factor of 25.4, and saves the changes into a temporary DAE file. After the temporary DAE file is imported into Blender, the temp file is then deleted, keeping the original DAE file intact. Starting with a DAE file that is exported using the NASA Enterprise Visualization Application (NEVA), a Collada exporter, from CATIA gives better results. NEVA is a Design Visualization product that is used for exporting 3D models from CATIA. With NEVA, the up axis is defined in the top-level node if navigation gravity is enabled. With that file, just replacing the primitive tags tristrips and trifans with polygons in yields a much improved image in Blender. As we identify more differences between the original DAE file and the DAE file exported from 3DS Max, this plugin can be improved further. Our goal is to have a model that is formatted and sized for import into Unity, and we are trying out different 3D programs to see which will work best.

Ayyangar, Arjun↗

ParFlow-CLM model simulation files for Maina et al., Journal of Hydrology, 2022

This dataset contains the files to run a ParFlow-CLM integrated hydrologic model simulation for Maina et al., HESS, 2022. It also contains the associated daily pressure output of those simulations. Simulations are for a hillslope-similarity approach based on seasonal groundwater changes in the East River, Colorado. Comparisons are made to different clustering, or functional zonation approaches. We assess the ability of these clustering approaches to identify and categorize hillslopes with similar static characteristics, hydroclimate, land surface processes, and subsurface dynamics in a mountainous watershed. ParFlow binary files (.pfb) are model specific, and can be modified/viewed with conversion to .sa, .silo, or .vtk file formats. ParFlow input scripts are in the .tcl file format. Associated post-processing scripts are included as .f90 files.

54 ENVIRONMENTAL SCIENCES↗

Transferable Output ASCII Data (TOAD) gateway: Version 1.0 user's guide

The Transferable Output ASCII Data (TOAD) Gateway, release 1.0 is described. This is a software tool for converting tabular data from one format into another via the TOAD format. This initial release of the Gateway allows free data interchange among the following file formats: TOAD; Standard Interface File (SIF); Program to Optimize Simulated Trajectories (POST) input; Comma Separated Value (TSV); and a general free-form file format. As required, additional formats can be accommodated quickly and easily.

Bingel, Bradford D.↗