Search NASA⌕ Search

Engineering topics

Guntupally, Kavya

Publications and source records attributed to Guntupally, Kavya.

FAIR Interfaces for Geospatial Scientific Data Searches

Several factors must be considered in designing a highly accurate, reliable, scalable, and user-friendly geospatial data search interfaces. This paper examines four critical questions that ought to be considered during design phase: (1) Is the search interface or API that provides the search capability useable by both humans and machines? (2) Are the results consistent and reliable? (3) Is the output response format free to use, community-defined, and non-propriety? (4) Does the API clearly state the usage clauses? This paper discusses how certain data repositories at the US Department of Energy's Oak Ridge National Laboratory apply FAIR data principles to enable geospatial searches and address the above-mentioned questions.

Devarakonda, Ranjeet↗

Enabling modern data discovery for atmospheric measurements

The Atmospheric Radiation Measurement (ARM) user facility is a US Department of Energy Office of Science user facility that is managed and operated through a collaborative effort led by nine US Department of Energy national laboratories. The ARM Data Center, located at Oak Ridge National Laboratory, is responsible for the timely collection, processing, and delivery of data products to the scientific community. The ARM Data Center holds more than 11,000 data products, including metadata collected from field campaigns, instruments, value-added products, and principal investigator–contributed data. These data sets are checked for successful transfer (for most data, this transfer is carried out automatically via the network; however, some of the largest data sets and some of the most remote sites require manual shipping of hard disks) and both the data and metadata are processed to a standard format, which is an ARM-standardized structure, via the Network Common Data Form. The Network Common Data Form is a self-describing binary format with many compatible software tools. Once processed, the data are cataloged, stored in the ARM Data Archive, and made discoverable through association with an array of metadata-characterizing information, such as location and measurement classification. These metadata enable powerful search capabilities through the ARM Data Center Data Discovery interface. This paper discusses the workflow of how the new discovery system has been redesigned from user requirements and how the data are distributed to the scientific community.

54 ENVIRONMENTAL SCIENCES↗

Automated Indexing of Structured Scientific Metadata Using Apache Solr

Scientific datasets are continuously growing with the amount of raw data being collected worldwide. This amount of data poses the biggest challenge to web search engines on how to retrieve them efficiently. This paper discusses how major scientific data centers are using popular open-source search platforms such as Solr [1] to retrieve structured data stored in data sources such as relational database management systems using its import handler mechanisms [2]. Additionally, we will also focus on how we can configure Solr to serve advanced full-text, faceted search capabilities, along with its key features, which simplify representing and delivering better performance to the scientific search interfaces.

Guntupally, Kavya↗

Big Federal Data Centers Implementing FAIR Data Principles: ARM Data Center Example

Atmospheric Radiation Measurement (ARM) is a multi-laboratory/multi-institutional, US Department of Energy Office of Science National User Facility. ARM’s data is currently hosted at the ARM Data Center (ADC) in Oak Ridge, Tennessee. The ADC holds more than 12,000 data products, with a total holding of more than 1.8 PB of data that dates back to 1992. This includes data from instruments, value-added products, model outputs, field campaigns, and principle investigator contributed data. In this paper, we discuss how big federal scientific data centers, such as ARM, that use modern and scalable architecture apply findable, accessible, interoperable, and reusable (FAIR) data principles to improve overall efficiency. These principles mainly emphasize machine-to-machine interactions that are directly applicable to ARM because of its data volume.

Devarakonda, Ranjeet↗

Provenance–aware workflow for data quality management and improvement for large continuous scientific data streams

Data quality assessment, management and improvement is integral part of any big data intensive scientific research to ensure accurate, reliable and reproducible science discoveries. The task of maintaining the quality of data, however, is non-trivial and pose a challenge for a program like Department of Energy’s Atmospheric Radiation Measurement (ARM) that collects data from hundreds of instruments across the world,and distributes thousands of streaming data products that are continuously growing in near-real-time in an archive 1.7 Petabyte in size and growing. We present a computational data processing workflow to collect the data quality issue via an easy and intuitive web-based portal that allows reporting of any quality issues for any site, facility or instruments at a granularity down to individual variable in the data files. Portal allows instrument specialists and scientists to provide corrective actions in form of symbolic equation. A parallel processing framework applies the data improvement to large volume of data in efficient parallel environment, while optimizing data transfer and file I/O operations. Corrected files are systematically versioned and archived. A provenance tracking module tracks and records any change made to the data during its entire life cycle which are communicated transparently to the scientific users. Developed in Python using open source technologies, software architecture enables efficient and fast management and improvement of data in an operational data center environment.

Kumar, Jitendra↗

Spring Boot based REST API to Improve Data Quality Report Generation for Big Scientific Data: ARM Data Center Example

Web application technologies are growing rapidly with continuous innovation and improvements. This paper focuses on the popular Spring Boot [1] java-based framework for building web and enterprise applications and how it provides the flexibility for service-oriented architecture (SOA). One challenge with any Spring-based applications is its level of complexity with configurations. Spring Boot makes it easy to create and deploy stand-alone, production-grade Spring applications with very little Spring configuration. Example, if we consider Spring Model-View-Controller (MVC) framework [2], we need to configure dispatcher servlet, web jars, a view resolver, and component scan among other things. To solve this, Spring Boot provides several Auto Configuration options to setup the application with any needed dependencies. Another challenge is to identify the framework dependencies and associated library versions required to develop a web application. Spring Boot offers simpler dependency management by using a comprehensive, but flexible, framework and the associated libraries in one single dependency, which provides all the Spring related technology that you need for starter projects as compared to CRUD web applications. This framework provides a range of additional features that are common across many projects such as embedded server, security, metrics, health checks, and externalized configuration. Web applications are generally packaged as war and deployed to a web server, but Spring Boot application can be packaged either as war or jar file, which allows to run the application without the need to install and/or configure on the application server. In this paper, we discuss how Atmospheric Radiation Measurement (ARM) Data Center (ADC) at Oak Ridge National Laboratory, is using Spring Boot to create a SOA based REST [4] service API, that bridges the gap between frontend user interfaces and backend database. Using this REST service API, ARM scientists are now able to submit reports via a user form or a command line interface, which captures the same data quality or other important information about ARM data.

Guntupally, Kavya↗