A Pragmatic Approach for Securing Systems to Process Big Data
This presentation will present a pragmatic security approach that could be used to systems that process large files of data.
Computer security↗
SEARCH · Search NASA
Results for “process data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Processing LiDAR Data to Predict Natural Hazards
ELF-Base and ELF-Hazards (wherein 'ELF' signifies 'Extract LiDAR Features' and 'LiDAR' signifies 'light detection and ranging') are developmental software modules for processing remote-sensing LiDAR data to identify past natural hazards (principally, landslides) and predict future ones. ELF-Base processes raw LiDAR data, including LiDAR intensity data that are often ignored in other software, to create digital terrain models (DTMs) and digital feature models (DFMs) with sub-meter accuracy. ELF-Hazards fuses raw LiDAR data, data from multispectral and hyperspectral optical images, and DTMs and DFMs generated by ELF-Base to generate hazard risk maps. Advanced algorithms in these software modules include line-enhancement and edge-detection algorithms, surface-characterization algorithms, and algorithms that implement innovative data-fusion techniques. The line-extraction and edge-detection algorithms enable users to locate such features as faults and landslide headwall scarps. Also implemented in this software are improved methodologies for identification and mapping of past landslide events by use of (1) accurate, ELF-derived surface characterizations and (2) three LiDAR/optical-data-fusion techniques: post-classification data fusion, maximum-likelihood estimation modeling, and hierarchical within-class discrimination. This software is expected to enable faster, more accurate forecasting of natural hazards than has previously been possible.
Fairweather, Ian↗
Real-time charged track reconstruction for CLAS12
Abstract This paper presents the results of charged particle track reconstruction in CLAS12 using artificial intelligence. In our approach, we use machine learning algorithms to reconstruct tracks, including their momentum and direction, with high accuracy from raw hits of the CLAS12 drift chambers. The reconstruction is performed in real-time, with the rate of data acquisition, and allows for the identification of event topologies in real-time. This approach revolutionizes the Nuclear Physics experiments' data processing, allowing us to identify and categorize the experimental data on the fly, and will lead to a significant reduction in experiment data processing. It can also be used in streaming readout applications leading to more efficient data acquisition and post-processing.
Instruments & Instrumentation↗
Compiler-generated memory mapping hints
According to one embodiment, a method of creating compiler-generated memory mapping hints in a computer system includes analyzing code, by a compiler of the computer system, to identify data access patterns in the code. System configuration information defining data processing system characteristics of a target system for the code is accessed. The data processing system characteristics include a plurality of processing resources and memory domain characteristics relative to the processing resources. A preferred allocation of data in memory domains of the target system is determined based on mapping the code to one or more selected processing resources and mapping the data to one or more of the memory domains based on the memory domain characteristics relative to the one or more selected processing resources. The preferred allocation is stored as compiler-generated memory mapping hints in a format accessible by a physical memory mapping resource of the target system.
97 MATHEMATICS AND COMPUTING↗
Quality control in the processing of telemetry data
Quality control in processing of telemetry data
DATA PROCESSING↗
Interactive image processing for meteorological applications at NASA/Goddard Space Flight Center
The meteorological data processing program at the space center is centered on the analysis of geostationary satellite data. Its objectives are to detect and predict severe storms, to improve short range and long range global numerical weather forecasting models, and to develop improved satellite systems. The Image Display and Manipulation System (IDAMS) for earth resources satellite data processing is described, as is the special purpose software package, METPAK, for performing meteorology operations on IDAMS. The Atmospheric and Oceanographic Information Processing System (AOIPS), a dual terminal multiprocessor is also described.
Billingsley, J. B.↗
DSN command system Mark III-78
The Deep Space Network command Mark III-78 data processing system includes a capability for a store-and-forward handling method. The functions of (1) storing the command files at a Deep Space station; (2) attaching the files to a queue; and (3) radiating the commands to the spacecraft are straightforward. However, the total data processing capability is a result of assuming worst case, failure-recovery, or nonnominal operating conditions. Optional data processing functions include: file erase, clearing the queue, suspend radiation, command abort, resume command radiation, and close window time override.
Stinnett, W. G.↗
The Suomi National Polar-Orbiting Partnership (SNPP): Continuing NASA Research and Applications
The Suomi National Polar-orbiting Partnership (SNPP) satellite was successfully launched into a polar orbit on October 28, 2011 carrying 5 remote sensing instruments designed to provide data to improve weather forecasts and to increase understanding of long-term climate change. SNPP provides operational continuity of satellite-based observations for NOAA's Polar-orbiting Operational Environmental Satellites (POES) and continues the long-term record of climate quality observations established by NASA's Earth Observing System (EOS) satellites. In the 2003 to 2011 pre-launch timeframe, NASA's SNPP Science Team assessed the adequacy of the operational Raw Data Records (RDRs), Sensor Data Records (SDRs), and Environmental Data Records (EDRs) from the SNPP instruments for use in NASA Earth Science research, examined the operational algorithms used to produce those data records, and proposed a path forward for the production of climate quality products from SNPP. In order to perform these tasks, a distributed data system, the NASA Science Data Segment (SDS), ingested RDRs, SDRs, and EDRs from the NOAA Archive and Distribution and Interface Data Processing Segments, ADS and IDPS, respectively. The SDS also obtained operational algorithms for evaluation purposes from the NOAA Government Resource for Algorithm Verification, Independent Testing and Evaluation (GRAVITE). Within the NASA SDS, five Product Evaluation and Test Elements (PEATEs) received, ingested, and stored data and performed NASA's data processing, evaluation, and analysis activities. The distributed nature of this data distribution system was established by physically housing each PEATE within one of five Climate Analysis Research Systems (CARS) located at either at a NASA or a university institution. The CARS were organized around 5 key EDRs directly in support of the following NASA Earth Science focus areas: atmospheric sounding, ocean, land, ozone, and atmospheric composition products. The PEATES provided the system level interface with members of the NASA SNPP Science Team and other science investigators within each CARS. A sixth Earth Radiation Budget CARS was established at NASA Langley Research Center (NASA LaRC) to support instrument performance, data evaluation, and analysis for the SNPP Clouds and the Earth's Radiant Budget Energy System (CERES) instrument. Following the 2011 launch of SNPP, spacecraft commissioning, and instrument activation, the NASA SNPP Science Team evaluated the operational RDRs, SDRs, and EDRs produced by the NOAA ADS and IDPS. A key part in that evaluation was the NASA Science Team's independent processing of operational RDRs and SDRs to EDRs using the latest NASA science algorithms. The NASA science evaluation was completed in the December 2012 to April 2014 timeframe with the release of a series of NASA Science Team Discipline Reports. In summary, these reports indicated that the RDRs produced by the SNPP instruments were of sufficiently high quality to be used to create data products suitable for NASA Earth System science and applications. However, the quality of the SDRs and EDRs were found to vary greatly when considering suitability for NASA science. The need for improvements in operational algorithms, adoption of different algorithmic approaches, greater monitoring of on-orbit instrument calibration, greater attention to data product validation, and data reprocessing were prominent findings in the reports. In response to these findings, NASA, in late 2013, directed the NASA SNPP Science Team to use SNPP instrument data to develop data products of sufficiently high quality to enable the continuation of EOS time series data records and to develop innovative, practical applications of SNPP data. This direction necessitated a transition of the SDS data system from its pre-launch assessment mode to one of full data processing and production. To do this, the PEATES, which served as NASA's data product testing environment during the prelaunch and early on-orbit periods, were transitioned to Science Investigator-led Processing Systems (SIPS). The distributed data architecture was maintained in this new system by locating the SIPS at the same institutions at which the CARS and PEATES were located. The SIPS acquire raw SNPP instrument Level 0 (i.e. RDR) data over the full SNPP mission from the NOAA ADS and IDPS through the NASA SDS Data Distribution and Depository Element (SD3E). The SIPS process those data into NASA Level 1, Level 2, and global, gridded Level 3 standard products using peer-reviewed algorithms provided by members of the NASA Science Team. The SIPS work with the NASA SNPP Science Team in obtaining enhanced, refined, or alternate real-time algorithms to support the capabilities of the Direct Readout Laboratory (DRL). All data products, algorithm source codes, coefficients, and auxiliary data used in product generation are archived in an assigned NASA Distributed Active Archive Center (DAAC).
remote sensing↗
Trade-off analysis of modes of data handling for earth resources (ERS), volume 1
Data handling requirements are reviewed for earth observation missions along with likely technology advances. Parametric techniques for synthesizing potential systems are developed. Major tasks include: (1) review of the sensors under development and extensions of or improvements in these sensors; (2) development of mission models for missions spanning land, ocean, and atmosphere observations; (3) summary of data handling requirements including the frequency of coverage, timeliness of dissemination, and geographic relationships between points of collection and points of dissemination; (4) review of data routing to establish ways of getting data from the collection point to the user; (5) on-board data processing; (6) communications link; and (7) ground data processing. A detailed synthesis of three specific missions is included.
Source record↗
Cloud Algorithm Design and Performance for the 2002 Geoscience Laser Altimeter System Mission
A satellite borne lidar instrument, the Geoscience Laser Altimeter System (GLAS), is to be launched in late 2002 and will provide continuous profiling of atmospheric clouds and aerosol on a global basis. Data processing algorithms have been developed to provide operational data products in near real time. Basic data products for cloud observations are the height of the top and bottom of single to multiple cloud layers and the lidar calibrated observed backscatter cross section up to the level of signal attenuation. In addition the optical depth and vertical profile of visible extinction cross section of many transmissive cloud layers and most haze layers are to be derived. The optical thickness is derivable in some cases from the attenuation of the molecular scattering below cloud base. In other cases an assumption of the scattering phase function is required. In both cases a estimated correction for multiple scattering is required. The data processing algorithms have been tested in part from aircraft measurements used to simulated satellite data. The GLAS lidar observations will be made from an orbit that will allow inter comparison with all other existing satellite cloud measurements.
Spinhirne, J. D.↗
Twist Model Development and Results From the Active Aeroelastic Wing F/A-18 Aircraft
Understanding the wing twist of the active aeroelastic wing F/A-18 aircraft is a fundamental research objective for the program and offers numerous benefits. In order to clearly understand the wing flexibility characteristics, a model was created to predict real-time wing twist. A reliable twist model allows the prediction of twist for flight simulation, provides insight into aircraft performance uncertainties, and assists with computational fluid dynamic and aeroelastic issues. The left wing of the aircraft was heavily instrumented during the first phase of the active aeroelastic wing program allowing deflection data collection. Traditional data processing steps were taken to reduce flight data, and twist predictions were made using linear regression techniques. The model predictions determined a consistent linear relationship between the measured twist and aircraft parameters, such as surface positions and aircraft state variables. Error in the original model was reduced in some cases by using a dynamic pressure-based assumption and by using neural networks. These techniques produced excellent predictions for flight between the standard test points and accounted for nonlinearities in the data. This report discusses data processing techniques and twist prediction validation, and provides illustrative and quantitative results.
Lizotte, Andrew↗
Twist Model Development and Results from the Active Aeroelastic Wing F/A-18 Aircraft
Understanding the wing twist of the active aeroelastic wing (AAW) F/A-18 aircraft is a fundamental research objective for the program and offers numerous benefits. In order to clearly understand the wing flexibility characteristics, a model was created to predict real-time wing twist. A reliable twist model allows the prediction of twist for flight simulation, provides insight into aircraft performance uncertainties, and assists with computational fluid dynamic and aeroelastic issues. The left wing of the aircraft was heavily instrumented during the first phase of the active aeroelastic wing program allowing deflection data collection. Traditional data processing steps were taken to reduce flight data, and twist predictions were made using linear regression techniques. The model predictions determined a consistent linear relationship between the measured twist and aircraft parameters, such as surface positions and aircraft state variables. Error in the original model was reduced in some cases by using a dynamic pressure-based assumption. This technique produced excellent predictions for flight between the standard test points and accounted for nonlinearities in the data. This report discusses data processing techniques and twist prediction validation, and provides illustrative and quantitative results.
Lizotte, Andrew M.↗
Landscape-scale Characterization of Arctic Tundra Vegetation Composition, Structure, and Function with a Multi-sensor Unoccupied Aerial System: Supporting Data.
Airborne remote sensing data collected using the Brookhaven National Laboratory's (BNL) heavy-lift unoccupied aerial system (UAS) octocopter platform, the Osprey, operated by the Terrestrial Ecosystem Science and Technology (TEST) group (https://www.bnl.gov/testgroup). This package includes data from three flights flown over the NGEE-Arctic Council, Kougarok and Teller sites in July, 2018. The Osprey is a multi-sensor UAS platform that simultaneously measures very high spatial resolution optical red/green/blue (RGB) and thermal infrared (TIR) surface 'skin' temperature imagery, as well as surface reflectance at 1 nm intervals in the visible to near-infrared spectral range 350 - 1000 nm measured at regular intervals along each flight path. Derived image products include ortho-mosaiced RGB and TIR images, an RGB-based digital surface model (DSM) using the structure from motion (SfM) technique, digital terrain model (DTM), and a canopy height model (CHM). In addition, a VNIR surface reflectance file is provided for the trigger locations collected during each flight campaign. Ancillary aircraft data, flight mission parameters, and general flight conditions provided by the onboard flight and data collection computers are also included. Unprocessed and processed data products are included in this package (processing levels 0-3). Data and metadata are provided as text (*.txt, *.json, *.kml, *hdr, *.enp), tabular (*.dat, *.csv, *.waypoint, ENVI format (no extension)), point cloud (*.laz) and image (*.jpg, *.tif, *png) formats. This metadata document contains flight campaign, instrument and file metadata, along with a description of data processing levels, data products and file naming scheme.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy?s Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).
54 ENVIRONMENTAL SCIENCES↗
Scanning structural mapping at the Life Science X-ray Scattering Beamline
This work describes the instrumentation and software for microbeam scattering and structural mapping at the Life Science X-ray Scattering (LiX) beamline at NSLS-II. Using a two-stage focusing scheme, an adjustable beam size between a few micrometres and a fraction of a millimetre is produced at the sample position. Scattering data at small and wide angles are collected simultaneously on multiple Pilatus detectors. A recent addition of an in-vacuum Pilatus 900k detector, with the detector modules arranged in a C-shaped configuration, has improved the azimuthal angle coverage in the wide-angle data. As an option, fluorescence data can be collected simultaneously. Fly scans have been implemented to minimize the time interval between scattering patterns and to avoid unnecessary radiation damage to the sample. For weakly scattering samples, an in-vacuum sample environment has been developed here to minimize background scattering. Data processing for these measurements is highly sample-specific. To establish a generalized data process workflow, first the data are reduced to reciprocal coordinates at the time of data collection. The users can then quantify features of their choosing from these intermediate data and construct structural maps. As examples, results from in-vacuum mapping of onion epidermal cell walls and 2D tomographic sectioning of an intact poplar stem are presented.
36 MATERIALS SCIENCE↗
The Role of Remotely Sensed and Relayed Data in the Delaware River Basin
The planned integration of the existing water quality monitoring and data processing systems in the Delaware River Basin with a data relay experiment proposed for the ERTS-1 is discussed. The experiment is designed to use ERTS-1 as a data relay link for a maximum of 20 hydrologic stations in the basin, including stream gaging, reservoir level, ground water level, and water quality monitoring stations. This experiment has the potential for reducing the time lag between data collection and dissemination to less than 12 hours. The experiment will also provide impetus to develop an operational system of real time data processing and dissemination to handle the large quantity of data that will be obtained from the stations in the basin. The results of this experiment will demonstrate the relative merits of satellite relay of data versus conventional means of data telemetry and will provide a basis for the development of operational satellite relay of hydrologic data.
Paulson, R. W.↗
Workflow-based automatic processing for Internet of Floating Things crowdsourced data
Data from sensors incorporated into mobile devices, such as networked navigational sensors, can be used to capture detailed environmental information. We describe here a workflow and framework for using sensors on boats to construct unique new datasets of underwater topography (bathymetry). Starting with a large number of measurements of position, depth, etc., obtained from such an Internet of Floating Things, we illustrate how, with a specialized protocol, data can be communicated to cloud resources, even when using delayed, intermittent, or disconnected networks. We then propose a method for automatic sensor calibration based on a novel reputation approach. Sampled depth data are interpolated efficiently on a cloud computing platform in order to provide a continuously updated bathymetric database. Our prototype implementation uses the FACE-IT Galaxy workflow engine to manage network communication and exploits the computational power of GPGPUs in a virtualized cloud environment, working with a CUDA-parallel algorithm, for efficient data processing. We report on an initial evaluation involving data from a sailing vessel in Italian coastal waters.
97 MATHEMATICS AND COMPUTING↗
Comparison Analysis of the Eastern Range False Cape 915-MHz Doppler Radar Wind Profiler
The United States Space Force (USSF) is responsible for space vehicle launches at its Eastern Range (ER), which includes the Cape Canaveral Space Force Station (CCSFS). Multiple systems are used to measure the atmosphere at the ER, including suites of Doppler Radar Wind Profilers (DRWPs) operating at 915MHz that measure winds within the lowest few kilometers of the atmosphere. Observations of boundary layer winds can be used for multiple applications, including serving as input to toxic dispersion models and characterizing winds for low-level aborts. In fall 2020 the USSF requested NASA’s Marshall Space Flight Center (MSFC) Natural Environments Branch (NE) to evaluate wind output from the False Cape(FC) DRWP system (MSFC NE 2021). In that analysis, it was found that data availability was low in the lowest few hundred meters of a profile, shown in Figure 1. To increase the available data, the USSF modified the quality control (QC) algorithm for the FCDRWP, which collected data from October 2020 to January 2021. In Fall 2021, the USSF requested NASA’s MSFC NE to evaluate the modified QC algorithm by comparing the data processed by the modified method (referred to as beam data) to data processed with the original QC method (referred to as original data). This report describes the system and the analyses that MSFC NE conducted to compare the availability and quality of data from the two QC methods
Maggire Fielder↗
TEAMER: Pitching Foil Crossflow Turbine Efficiency Data
This dataset documents the efficiency testing of a pitching foil crossflow turbine, conducted at the University of New Hampshire's (UNH) Chase Ocean Engineering Laboratory tow tank facility. The tests explored various pitch phases and amplitudes, ranging from 0 to 18 degrees, across different flow speeds and turbine RPMs. Specifically, testing was performed at pitch amplitudes of 0,3,6,9,12,15 and 18 degrees. The results indicate that turbine efficiency improves with increased pitch amplitude, reaching optimal performance at 12 degrees. Modifications were made to the test frame by UNH to enhance the stiffness of the support structure. The testing specifically mapped the efficiency of a single quadrant of the Ocean Renewable Power Company (ORPC) Autonomous Turbine Generator Unit (ATGU), covering the full range of allowable pitching amplitudes and phases. The dataset provides both raw and processed data, including drag and torque measurements, and contains Python scripts used for data processing and visualization. This research was supported by funding from TEAMER RFTS 1 (Request for Technical Support).
16 TIDAL AND WAVE POWER↗