Search NASA⌕ Search

SEARCH · Search NASA

Results for “custom data sets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

Python & Qt, Powerful Tools for Technical Computing

The objective of this presentation is to give a brief overview of Python computer language and Qt for Python which provides an interface to Python for building graphical applications. The Qt language provides a method for rapid programming of Graphical User Interfaces (GUIs) that are highly scalable, robust and platform independent. Both Python and Qt provide a powerful set of tools for Dynamic Analysis which are based on Open-Source software. Many problems and calculations in Structural Dynamics such as Power Spectral Density, Shock Response Spectrum and Vibration Response Spectrum can be easily calculated using these tools. The advantage of using Open-Source software is the ability to create custom graphical user interfaces similar to Matlab without the expense of software licensing and the ability to customize the software to an organization's specific needs. Also, another advantage is the ability to know which algorithms are being used by the GUI, know the numerical limitations and scale to large size data sets. I will end the presentation by demonstrating a Structural Dynamics GUI I created that was designed primarily to interactively analyze Post Flight high speed data provided by the ground station telemetry networks.

Grillo, Vincent↗

COMBIgor: Data-Analysis Package for Combinatorial Materials Science

Combinatorial experiments involve synthesis of sample libraries with lateral composition gradients requiring spatially-resolved characterization of structure and properties. Here, due to maturation of combinatorial methods and their successful application in many fields, the modern combinatorial laboratory produces diverse and complex data sets requiring advanced analysis and visualization techniques. In order to utilize these large data sets to uncover new knowledge, the combinatorial scientist must engage in data science. For data science tasks, most laboratories adopt common-purpose data management and visualization software. However, processing and cross-correlating data from various measurement tools is no small task for such generic programs. Here we describe COMBIgor, a purpose-built open-source software package written in the commercial Igor Pro environment, designed to offer a systematic approach to loading, storing, processing, and visualizing combinatorial data sets. It includes (1) methods for loading and storing data sets from combinatorial libraries, (2) routines for streamlined data processing, and (3) data analysis and visualization features to construct figures. Most importantly, COMBIgor is designed to be easily customized by a laboratory, group, or individual in order to integrate additional instruments and data-processing algorithms. Utilizing the capabilities of COMBIgor can significantly reduce the burden of data management on the combinatorial scientist.

97 MATHEMATICS AND COMPUTING↗

High Level Synthesis of RDF Queries for Graph Analytics

In this paper we present a set of techniques that enable the synthesis of efficient custom accelerators for memory intensive, irregular applications. To address irregular applications challenges (large memory footprints, unpredictable fine- grained data accesses, and high synchronization intensity), and exploit their opportunities (thread level parallelism, memory level parallelism), we propose a novel accelerator design which take advantage of an adaptive and Distributed Controller (DC) architecture, and a Memory Interface (MI) that supports parallel memory subsystems. Among the multitude of algorithms that may benefit from our solution, we focus on the acceleration of graph analytics applications, and in particular, on the syn- thesis of SPARQL queries on Resource Description Framework (RDF) databases. We achieve this objective by incorporating the synthesis techniques into Bambu, an Open Source high- level synthesis tools, and interfacing the system with GEMS, the Graph database Engine for Multithreaded Systems. The front- end of GEMS generates optimized C implementations of the input queries, modeled as pattern matching routines, which are then automatically synthesized by Bambu. We validate our approach synthesizing several SPARQL queries from the Lehigh University Benchmark (LUBM).

HLS, FPGA, Convey, RDF, GEMS, Memory-intensive,↗

WADeG Cell Phone

The on cell phone software captures the images from the CMOS camera periodically, stores the pictures, and periodically transmits those images over the cellular network to the server. The cell phone software consists of several modules: CamTest.cpp, CamStarter.cpp, StreamIOHandler .cpp, and covertSmartDevice.cpp. The camera application on the SmartPhone is CamStarter, which is "the" user interface for the camera system. The CamStarter user interface allows a user to start/stop the camera application and transfer files to the server. The CamStarter application interfaces to the CamTest application through registry settings. Both the CamStarter and CamTest applications must be separately deployed on the smartphone to run the camera system application. When a user selects the Start button in CamStarter, CamTest is created as a process. The smartphone begins taking small pictures (CAPTURE mode), analyzing those pictures for certain conditions, and saving those pictures on the smartphone. This process will terminate when the user selects the Stop button. The camtest code spins off an asynchronous thread, StreamIOHandler, to check for pictures taken by the camera. The received image is then tested by StreamIOHandler to see if it meets certain conditions. If those conditions are met, the CamTest program is notified through the setting of a registry key value and the image is saved in a designated directory in a custom BMP file which includes a header and the image data. When the user selects the Transfer button in the CamStarter user interface, the covertsmartdevice code is created as a process. Covertsmartdevice gets all of the files in a designated directory, opens a socket connection to the server, sends each file, and then terminates.

Cogliati, JoshuaJ.↗

AWAKEN Site A1 - NREL Ceilometer (Vaisala CL51) / Derived Data

Site A1 netCDF L3 data files have level 3 (L3) data that have gone through the calculation service and contain all the data from the algorithms, including mixing layer height values and quality index data. L3 default files contain L3 data that use the default preset for a live plot. File naming schema: L3_DEFAULT_ _YYYYMMDDHHMM_ _ .nc Name Description L3 Identification of the data level DEFAULT Identification of the L3 file type CUSTOM OFFLINE STATION_NUMBER WMO station number, if defined YYYYMMDDHHMM UTC time ParameterKey Identification of the advanced algorithm settings. See the table below for an explanation. FREE_FORMAT File suffix, if defined

17 WIND ENERGY↗

WFIP3 - BARG site - NREL Ceilometer (Vaisala CL51) / Derived Data

Site A1 netCDF L3 data files have level 3 (L3) data that have gone through the calculation service and contain all the data from the algorithms, including mixing layer height values and quality index data. L3 default files contain L3 data that use the default preset for a live plot. File naming schema: L3_DEFAULT_ _YYYYMMDDHHMM_ _ .nc Name Description L3 Identification of the data level DEFAULT Identification of the L3 file type CUSTOM OFFLINE STATION_NUMBER WMO station number, if defined YYYYMMDDHHMM UTC time ParameterKey Identification of the advanced algorithm settings. See the table below for an explanation. FREE_FORMAT File suffix, if defined

17 WIND ENERGY↗

WFIP3 - BARG site - NREL Ceilometer (Vaisala CL51) / Derived Data

BARG ceilometer: netCDF L3 data files have level 3 (L3) data that have gone through the calculation service and contain all the data from the algorithms, including mixing layer height values, and quality index data. L3 default files contain L3 data that use the default preset for a live plot. File naming schema: L3_DEFAULT_ _YYYYMMDDHHMM_ _ .nc Name Description L3 Identification of the data level DEFAULT Identification of the L3 file type CUSTOM OFFLINE STATION_NUMBER WMO station number, if defined YYYYMMDDHHMM UTC time ParameterKey Identification of the advanced algorithm settings. See the table below for an explanation. FREE_FORMAT File suffix, if defined

17 WIND ENERGY↗

WFIP3 - NOAA SHIP site - NREL Ceilometer (Vaisala CL51) / Derived Data

NOAA SHIP ceilometer: netCDF L3 data files have level 3 (L3) data that have gone through the calculation service and contain all the data from the algorithms, including mixing layer height values, and quality index data. L3 default files contain L3 data that use the default preset for a live plot. File naming schema: L3_DEFAULT_ _YYYYMMDDHHMM_ _ .nc Name Description: L3 Identification of the data level DEFAULT Identification of the L3 file type CUSTOM OFFLINE STATION_NUMBER WMO station number, if defined YYYYMMDDHHMM UTC time ParameterKey Identification of the advanced algorithm settings. See the table below for an explanation. FREE_FORMAT File suffix, if defined

17 WIND ENERGY↗

WFIP3 - CACO site - NREL Ceilometer (Vaisala CL51) / Derived Data

CACO ceilometer: netCDF L3 data files have level 3 (L3) data that have gone through the calculation service and contain all the data from the algorithms, including mixing layer height values, and quality index data. L3 default files contain L3 data that use the default preset for a live plot. File naming schema: L3_DEFAULT_ _YYYYMMDDHHMM_ _ .nc Name Description: L3 Identification of the data level DEFAULT Identification of the L3 file type CUSTOM OFFLINE STATION_NUMBER WMO station number, if defined YYYYMMDDHHMM UTC time ParameterKey Identification of the advanced algorithm settings. See the table below for an explanation. FREE_FORMAT File suffix, if defined

17 WIND ENERGY↗

Electrification Futures Study Flexible Load Profiles

This data set includes hourly profiles for flexible load developed for the Electrification Futures Study (EFS). The load profiles represent projected end-use electricity demand that is assumed to be flexible (i.e., can be shifted throughout a day) for various scenarios of flexibility (Base, Enhanced), electrification (Reference, Medium, High), and technology advancement (Slow, Moderate, Rapid), and were developed as inputs into the ReEDS model. The quantity of flexible load is estimated using assumptions on the level of flexibility and customer participation within each subsector modeled in the EFS. Detailed assumptions and modeling implementation will be documented in ongoing EFS analyses. Flexible load profiles are provided for a subset of years (2018, 2020, 2024, 2030, 2040, 2050) and are aggregated to the state and sector level. Total electricity load profiles can be found in a related EFS data set (https://dx.doi.org/10.7799/1593122). NOTE: Due to the file size, Mac users may experience issues decompressing the zip files using the Mac Archive Utility. In those cases, decompressing using the command line is recommended. - Mai, Trieu, Paige Jadun, Jeffrey Logan, Colin McMillan, Matteo Muratori, Daniel Steinberg, Laura Vimmerstedt, Ryan Jones, Benjamin Haley, and Brent Nelson. 2018. Electrification Futures Study: Scenarios of Electric Technology Adoption and Power Consumption for the United States. National Renewable Energy Laboratory. NREL/TP-6A20-71500. https://doi.org/10.2172/1459351. - Murphy, Caitlin, Trieu Mai, Yinong Sun, Paige Jadun, Matteo Muratori, Brent Nelson, Ryan Jones. Forthcoming. Electrification Futures Study: Scenarios of Power System Evolution and Infrastructure Development for the United States. National Renewable Energy Laboratory. - Sun, Yinong, Paige Jadun, Brent Nelson, Matteo Muratori, Caitlin Murphy, Jeffrey Logan, and Trieu Mai. Forthcoming. Electrification Futures Study: Methodological Approaches for Assessing Long-Term Power System Impacts of End-Use Electrification. National Renewable Energy Laboratory.

buildings↗

Prediction Models for Dynamic Demand Response

As Smart Grids move closer to dynamic curtailment programs, Demand Response (DR) events will become necessary not only on fixed time intervals and weekdays predetermined by static policies, but also during changing decision periods and weekends to react to real-time demand signals. Unique challenges arise in this context vis-a-vis demand prediction and curtailment estimation and the transformation of such tasks into an automated, efficient dynamic demand response (D 2 R) process. While existing work has concentrated on increasing the accuracy of prediction models for DR, there is a lack of studies for prediction models for D 2 R, which we address in this paper. Our first contribution is the formal definition of D 2 R, and the description of its challenges and requirements. Our second contribution is a feasibility analysis of very-short-term prediction of electricity consumption for D 2 R over a diverse, large-scale dataset that includes both small residential customers and large buildings. Our third, and major contribution is a set of insights into the predictability of electricity consumption in the context of D 2 R. Specifically, we focus on prediction models that can operate at a very small data granularity (here 15-min intervals), for both weekdays and weekends - all conditions that characterize scenarios for D 2 R. We find that short-term time series and simple averaging models used by Independent Service Operators and utilities achieve superior prediction accuracy. We also observe that workdays are more predictable than weekends and holiday. Also, smaller customers have large variation in consumption and are less predictable than larger buildings. Key implications of our findings are that better models are required for small customers and for non-workdays, both of which are critical for D 2 R. Also, prediction models require just few days’ worth of data indicating that small amounts of historical training data can be used to make reliable predictions, simplifying the complexity of big data challenge associated with D 2 R.

Aman, Saima↗

Automated RF Phase Adjustment for Beam Stabilization in the Fermilab Linac

The Fermilab Linac experiences longitudinal beam phase drift, leading to increased particle loss, conventionally corrected through labor-intensive manual RF adjustments. This project explores machine learning-based automation for drift correction, employing a prototype-based classification approach. Our model utilizes a 34-dimensional feature set (RF settings and BPM readings) and leverages a 7x27 response matrix for system modeling. To overcome limited real-world data, we generate synthetic data, enhancing model training and generalizability. Custom loss functions, including a surrogate energy-consistent loss and a temporal smoothness constraint, ensure physically plausible drift predictions. The goal is a robust system for autonomous phase adjustments, ensuring stable beam acceleration and reduced manual intervention.

Chichili, R. R. [Illinois U., Chicago]↗

Automated RF Phase Adjustment for Beam Stabilization in the Fermilab Linac

The Fermilab Linac experiences longitudinal beam phase drift, leading to increased particle loss, conventionally cor- rected through labor-intensive manual RF adjustments. This project explores machine learning-based automation for drift correction, employing a prototype-based classification approach. Our model utilizes a 34-dimensional feature set (RF settings and BPM readings) and leverages a 7x27 response matrix for system modeling. To overcome limited real-world data, we generate synthetic data, enhancing model training and generalizability. Custom loss functions, including a sur- rogate energy-consistent loss and a temporal smoothness constraint, ensure physically plausible drift predictions. The goal is a robust system for autonomous phase adjustments, ensuring stable beam acceleration and reduced manual intervention.

Chichili, R. R. [U. Illinois, Chicago]↗

Incorporating the APS Catalog of the POSS I and Image Archive in ADS

The primary purpose of this contract was to develop the software to both create and access an on-line database of images from digital scans of the Palomar Sky Survey. This required modifying our DBMS (called Star Base) to create an image database from the actual raw pixel data from the scans. The digitized images are processed into a set of coordinate-reference index and pixel files that are stored in run-length files, thus achieving an efficient lossless compression. For efficiency and ease of referencing, each digitized POSS I plate is then divided into 900 subplates. Our custom DBMS maps each query into the corresponding POSS plate(s) and subplate(s). All images from the appropriate subplates are retrieved from disk with byte-offsets taken from the index files. These are assembled on-the-fly into a GIF image file for browser display, and a FITS format image file for retrieval. The FITS images have a pixel size of 0.33 arcseconds. The FITS header contains astrometric and photometric information. This method keeps the disk requirements manageable while allowing for future improvements. When complete, the APS Image Database will contain over 130 Gb of data. A set of web pages query forms are available on-line, as well as an on-line tutorial and documentation. The database is distributed to the Internet by a high-speed SGI server and a high-bandwidth disk system. URL is http://aps.umn.edu/IDB/. The image database software is written in perl and C and has been compiled on SGI computers with MIX5.3. A copy of the written documentation is included and the software is on the accompanying exabyte tape.

Humphreys, Roberta M.↗

Collaborative Data Publication Utilizing the Open Data Repository's (ODR) Data Publisher

Introduction: For small communities in diverse fields such as astrobiology, publishing and sharing data can be a difficult challenge. While large, homogenous fields often have repositories and existing data standards, small groups of independent researchers have few options for publishing standards and data that can be utilized within their community. In conjunction with teams at NASA Ames and the University of Arizona, the Open Data Repository's (ODR) Data Publisher has been conducting ongoing pilots to assess the needs of diverse research groups and to develop software to allow them to publish and share their data collaboratively. Objectives: The ODR's Data Publisher aims to provide an easy-to-use and implement software tool that will allow researchers to create and publish database templates and related data. The end product will facilitate both human-readable interfaces (web-based with embedded images, files, and charts) and machine-readable interfaces utilizing semantic standards. Characteristics: The Data Publisher software runs on the standard LAMP (Linux, Apache, MySQL, PHP) stack to provide the widest server base available. The software is based on Symfony (www.symfony.com) which provides a robust framework for creating extensible, object-oriented software in PHP. The software interface consists of a template designer where individual or master database templates can be created. A master database template can be shared by many researchers to provide a common metadata standard that will set a compatibility standard for all derivative databases. Individual researchers can then extend their instance of the template with custom fields, file storage, or visualizations that may be unique to their studies. This allows groups to create compatible databases for data discovery and sharing purposes while still providing the flexibility needed to meet the needs of scientists in rapidly evolving areas of research. Research: As part of this effort, a number of ongoing pilot and test projects are currently in progress. The Astrobiology Habitable Environments Database Working Group is developing a shared database standard using the ODR's Data Publisher and has a number of example databases where astrobiology data are shared. Soon these databases will be integrated via the template-based standard. Work with this group helps determine what data researchers in these diverse fields need to share and archive. Additionally, this pilot helps determine what standards are viable for sharing these types of data from internally developed standards to existing open standards such as the Dublin Core (http://dublincore.org) and Darwin Core (http://rs.twdg.org) metadata standards. Further studies are ongoing with the University of Arizona Department of Geosciences where a number of mineralogy databases are being constructed within the ODR Data Publisher system. Conclusions: Through the ongoing pilots and discussions with individual researchers and small research teams, a definition of the tools desired by these groups is coming into focus. As the software development moves forward, the goal is to meet the publication and collaboration needs of these scientists in an unobtrusive and functional way.

easy to use and implement software tool↗

A Ground Systems Architecture Transition for A Distributed Operations System

The Marshall Space Flight Center (MSFC) Ground Systems Department (GSD) recently undertook an architecture change in the product line that serves the ISS program. As a result, the architecture tradeoffs between data system product lines that serve remote users versus those that serve control center flight control teams were explored extensively. This paper describes the resulting architecture that will be used in the ISS payloads program, and the resulting functional breakdown of the products that support that architecture. It also describes the lessons learned from the path that was followed, as a migration of products cause the need to reevaluate the allocation of functions across the architecture. The result is a set of innovative ground system solutions that is scalable so it can support facilities of wide-ranging sizes, from a small site up to large control centers. Effective use of system automation, custom components, design optimization for data management, data storage, data transmissions, and advanced local and wide area networking architectures, plus the effective use of Commercial-Off-The-Shelf (COTS) products, provides flexible Remote Ground System options that can be tailored to the needs of each user. This paper offers a description of the efficiency and effectiveness of the Ground Systems architectural options that have been implemented, and includes successful implementation examples and lessons learned.

Sellers, Donna↗

A Ground Systems Architecture Transition for a Distributed Operations System

The Marshall Space Flight Center (MSFC) Ground Systems Department (GSD) recently undertook an architecture change in the product line that serves the ISS program. As a result, the architecture tradeoffs between data system product lines that serve remote users versus those that serve control center flight control teams were explored extensively. This paper describes the resulting architecture that will be used in the International Space Station (ISS) payloads program, and the resulting functional breakdown of the products that support this architecture. It also describes the lessons learned from the path that was followed, as a migration of products cause the need to reevaluate the allocation of functions across the architecture. The result is a set of innovative ground system solutions that is scalable so it can support facilities of wide-ranging sizes, from a small site up to large control centers. Effective use of system automation, custom components, design optimization for data management, data storage, data transmissions, and advanced local and wide area networking architectures, plus the effective use of Commercial-Off-The-Shelf (COTS) products, provides flexible Remote Ground System options that can be tailored to the needs of each user. This paper offers a description of the efficiency and effectiveness of the Ground Systems architectural options that have been implemented, and includes successful implementation examples and lessons learned.

Sellers, Donna↗

Distributed non-negative matrix factorization with determination of the number of latent features

The holistic analysis and understanding of the latent (that is, not directly observable) variables and patterns buried in large datasets is crucial for data-driven science, decision making and emergency response. Such exploratory analyses require devising unsupervised learning methods for data mining and extraction of the latent features, and non-negative matrix factorization (NMF) is one of the prominent such methods. NMF is based on compute-intense non-convex constrained minimization, which, for large datasets requires fast and distributed algorithms. However, current parallel implementations of NMF fail to estimate the number of latent features. In practice, identifying these features is both difficult and significant for pattern recognition and latent feature analysis, especially for large dense matrices. Here, we introduce a distributed NMF algorithm coupled with distributed custom clustering followed by a stability analysis on dense data, which we call DnMFk, to determine the number of latent variables. The results on synthetic data and the classical Swimmer data set demonstrate the accuracy of model determination while scaling nearly linearly across multiple processors for large data. Further, we employ DnMFk to determine the number of hidden features from a terabyte matrix.

97 MATHEMATICS AND COMPUTING↗