SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Many high-level vision systems use rule-based approaches to solving problems such as autonomous navigation and image understanding. The rules are usually elaborated by experts. However, this procedure may be rather tedious. In this paper, we propose a method to generate such rules automatically from training data. The proposed method is also capable of filtering out irrelevant features and criteria from the rules.
The development of machine vision based pilot aids to help reduce night approach and landing accidents is explored. The techniques developed are motivated by the desire to use the available information sources for navigation such as the airport lighting layout, attitude sensors and Global Positioning System to derive more precise aircraft position and orientation information. The fact that airport lighting geometry is known and that images of airport lighting can be acquired by the camera, has lead to the synthesis of machine vision based algorithms for runway relative aircraft position and orientation estimation. The main contribution of this research is the synthesis of seven navigation algorithms based on two broad families of solutions. The first family of solution methods consists of techniques that reconstruct the airport lighting layout from the camera image and then estimate the aircraft position components by comparing the reconstructed lighting layout geometry with the known model of the airport lighting layout geometry. The second family of methods comprises techniques that synthesize the image of the airport lighting layout using a camera model and estimate the aircraft position and orientation by comparing this image with the actual image of the airport lighting acquired by the camera. Algorithms 1 through 4 belong to the first family of solutions while Algorithms 5 through 7 belong to the second family of solutions. Algorithms 1 and 2 are parameter optimization methods, Algorithms 3 and 4 are feature correspondence methods and Algorithms 5 through 7 are Kalman filter centered algorithms. Results of computer simulation are presented to demonstrate the performance of all the seven algorithms developed.
This paper presents a novel approach to terrain-relative navigation with a visual camera and Inertial Measurement Unit (IMU). The proposed algorithm uses an Extended Kalman Filter (EKF) to combine an IMU propagated state estimate with batch correction estimates computed over a sliding window of measurements. The batch correction algorithm follows the Maximum Likelihood Estimation (MLE) approach used in other Bundle Adjustment systems. Unlike other systems, the proposed system parameterizes the state over the entire window in terms of the state at a single epoch. By ignoring IMU error over the window duration, we obtain a state epoch MLE that jointly estimates the epoch state and terrain parameters with drastically reduced computationally cost. This paper presents the general architecture which can be adapted for various state parameterizations and measurement inputs. For space applications with high-accuracy IMUs, the reduction in computational cost comes with only a modest increase in estimation errors. The increase in error is quantified via a linear covariance analysis presented in this paper. Furthermore, we present simulation results which show the applicability of this algorithm to planetary landing problems.
This report is a deliverable for the “Final Report” task of DOE NEET Project 19-16790, "Context-Aware Safety Information Display for Nuclear Field Workers." This project's overall goal is to test the hypothesis that integrating computer vision and process reasoning methods will enable proactive visualization of the safe operation and maintenance processes of Nuclear Power Plants (NPP) for field workers. Augmented Reality (AR) glasses adopting such proactive safety information visualization techniques can significantly increase personnel safety and reduce the NPP’s operating costs. The current practice of monitoring NPPs requires workers to switch between digital models, data, and physical workspaces in identifying relevant but potentially occluded objects and in assessing the risks of operation and maintenance processes. On the other hand, frequently changed field conditions require field workers to report to supervisors for real-time guidance. Such guidance is essential to ensure that changing conditions will not invalidate or endanger the work order and other ongoing processes that may jeopardize NPP operations. Additionally, incorrect recognition of equipment objects can result in communication errors and safety problems. AR techniques can assist engineers in viewing the physical workspaces with objects labeled with detailed operation procedures and safety reminders during field operations. The project team developed an “Intelligent Context-Aware Safety Information Display” (ICAD) for supporting Nuclear Power Plant (NPP) field workers in achieving safe and efficient execution of a series of operational tasks in uncertain and changing workspaces of an NPP. Before designing the ICAD-AR prototype, the project team synthesized NPP operational knowledge models through literature review studies, surveys, interviews with domain experts, and knowledge modeling. The project team conducted an extensive study of the operational procedures of various NPPs, and digital technologies that can support the safe and efficient execution of those procedures in different NPP operational contexts. This literature review helped the project team conduct surveys and interviews with nuclear engineers and field workers to identify three categories of information. The NPP knowledge modeling efforts reveal that the three categories of information identified have different levels of importance in a typical procedure of carrying out a series of tasks to achieve a specific NPP operation goal (e.g., shutdown, mode changes). These three categories of information include 1) Workspace dynamics – the changing spatial arrangements of workspaces, tools, protection equipment, and supporting materials, 2) Workflow prognostics – the dynamic dependencies between different parts of an NPP that functionally support and influence each other in terms of safety and efficiency, and 3) Hazards – objects and spaces that contain hazardous materials or physical conditions that can pose risks to workers or mechanical systems. The project team has profiled the importance levels of these categories of information into a knowledge model. This knowledge model specifies what types of information are more critical for a given task in a given workspace so that computers can automatically identify critical objects and sensors in a scene for delivering context-ware safety information to field workers through AR devices. Significant research development of this project results in technical research outcomes and a prototyping system that illustrates the technical feasibility of establishing an ICAD-AR system supporting the proactive safety information display for nuclear field workers. This final report summarizes the project team’s technological achievements in the past three years. Overall, the project team completed the development and integration of five techniques into a prototype ICAD Augmented Reality (ICAD-AR) system and demonstrated the developed system’s real-time execution in a mechanical room. The project team completed the analysis of using this prototype in other types of workspaces based on 3D image data and digital design models collected from two additional workspaces (a water treatment plant and a flow loop training facility). The integrated techniques include 1) Natural Language Processing (NLP) algorithms supporting the generation and updates of nuclear fieldwork process models based on text analysis of work packages and operation manuals; 2) sensor log analysis for predicting control actions in given sensor reading contexts; 3) computer vision algorithms for automatic localization and navigation of workers; 4) object detection algorithms for identifying task-related objects and correlated sensors for safety checking; 5) AR technique as a platform for supporting the integration. The testing results of these five techniques have shown that 1) the sensor log analysis model can predict the next control action with an accuracy of 0.883; 2) the trained natural language processing model can extract more than 80% of the critical information from paper-based procedures (PBPs); 3) the navigation algorithm with the integration of Visual Inertial Odometry (VIO) and Non-Recursive Bayesian Filter methods make operator’s trajectory estimation resilient to drift error; 4) the computer vision algorithm can detect task-specific and safety-critical objects with an average accuracy of 95.3%. The project team used work procedures collected from a flow loop training facility and two datasets collected from two mechanical rooms simulating the workspaces of NPPs to demonstrate the technical capabilities of the developed ICAD-AR prototype. The demonstration validated the technical feasibility of establishing the ICAD-AR system for nuclear field workers and identified the challenges in 1) automatic text analysis of work packages; 2) use of limited samples of sensor logs for predicting the proper timings of control actions; 3) reliably tracking workers and their task progress in mechanical rooms with many similar objects.
bd Systems (a subsidiary of SAIC) has developed the Simulation Package for Autonomous Rendezvous Test and ANalysis (SPARTAN), a high-fidelity on-orbit simulation featuring multiple six-degree-of-freedom (6DOF) vehicles. SPARTAN has been developed in a modular fashion in Matlab/Simulink to test next-generation automated rendezvous and docking guidance, navigation,and control algorithms for NASA's new Vision for Space Exploration. SPARTAN includes autonomous state-based mission manager algorithms responsible for sequencing the vehicle through various flight phases based on on-board sensor inputs and closed-loop guidance algorithms, including Lambert transfers, Clohessy-Wiltshire maneuvers, and glideslope approaches The guidance commands are implemented using an integrated translation and attitude control system to provide 6DOF control of each vehicle in the simulation. SPARTAN also includes high-fidelity representations of a variety of absolute and relative navigation sensors that maybe used for NASA missions, including radio frequency, lidar, and video-based rendezvous sensors. Proprietary navigation sensor fusion algorithms have been developed that allow the integration of these sensor measurements through an extended Kalman filter framework to create a single optimal estimate of the relative state of the vehicles. SPARTAN provides capability for Monte Carlo dispersion analysis, allowing for rigorous evaluation of the performance of the complete proposed AR&D system, including software, sensors, and mechanisms. SPARTAN also supports hardware-in-the-loop testing through conversion of the algorithms to C code using Real-Time Workshop in order to be hosted in a mission computer engineering development unit running an embedded real-time operating system. SPARTAN also contains both runtime TCP/IP socket interface and post-processing compatibility with bdStudio, a visualization tool developed by bd Systems, allowing for intuitive evaluation of simulation results. A description of the SPARTAN architecture and capabilities is provided, along with details on the models and algorithms utilized and results from representative missions.
The ability to detect and locate runways/taxiways and obstacles in images captured using on-board sensors is an essential first step in the automation of low-altitude flight, landing, takeoff, and taxiing phase of aircraft navigation. Automation of these functions under different weather and lighting situations, can be facilitated by using sensors of different modalities. An aircraft-based Synthetic Vision System (SVS), with sensors of different modalities mounted on-board, complements the current ground-based systems in functions such as detection and prevention of potential runway collisions, airport surface navigation, and landing and takeoff in all weather conditions. In this report, we address the problem of detection of objects in monocular image sequences obtained from two types of sensors, a Passive Millimeter Wave (PMMW) sensor and a video camera mounted on-board a landing aircraft. Since the sensors differ in their spatial resolution, and the quality of the images obtained using these sensors is not the same, different approaches are used for detecting obstacles depending on the sensor type. These approaches are described separately in two parts of this report. The goal of the first part of the report is to develop a method for detecting runways/taxiways and objects on the runway in a sequence of images obtained from a moving PMMW sensor. Since the sensor resolution is low and the image quality is very poor, we propose a model-based approach for detecting runways/taxiways. We use the approximate runway model and the position information of the camera provided by the Global Positioning System (GPS) to define regions of interest in the image plane to search for the image features corresponding to the runway markers. Once the runway region is identified, we use histogram-based thresholding to detect obstacles on the runway and regions outside the runway. This algorithm is tested using image sequences simulated from a single real PMMW image.
Future lander missions will travel to ambitious, scientifically interesting locations near rough and dangerous terrain. They will need to operate with limited prior information about the terrain, and under varying lighting conditions. Landing safely and precisely in the face of these challenges is difficult for existing vision-based landing systems, which require detailed orbital reconnaissance, a priori hazard maps, and impose time-of-day restrictions on landing to ensure similar lighting conditions in orbital and descent imagery. Advanced 3D imaging LiDAR systems currently under development, and originally intended for single-scan hazard detection, have the potential to be operated continuously from altitudes of up to 5 km. Used together with existing inertial measurement units (IMUs), these sensors open a path-to-flight for a full navigation and mapping system, which could replace or augment a traditional landing sensor suite. A landing system based around these sensors can perform accurate altimetry, map-relative localization (MRL), LiDAR-inertial odometry, and map refinement in an illumination-insensitive manner, over unknown or partially known terrain. This paper outlines preliminary work on a LiDAR-inertial landing system that: estimates the spacecraft trajectory during entry, descent, and landing (EDL); and maps the topography of the terrain below, for future use in hazard detection and avoidance. An incremental, factor graph based, smoothing approach is used to solve for the maximum a posteriori trajectory of spacecraft states. Integrated IMU measurements and features tracked in adjacent range and intensity images are used to estimate motion (LiDAR-inertial odometry). LiDAR scans are binned into motion-corrected digital elevation models (DEMs), which are matched to an existing orbital topographic map to provide absolute position information (MRL). The estimated trajectory is then used to project the LiDAR scans into the map frame, creating a variable-resolution quadtree topographic map suitable for hazard detection and avoidance. Existing topographic maps from throughout the solar system (i.e., Earth, the Moon, Mars, Ceres, Vesta, Europa, Enceladus, and Eros) are upsampled for use in EDL simulations. The Mars 2020 Lander Vision System Simulator (LVSS) is extended to simulate LiDAR-inertial data for realistic EDL trajectories. Results of the algorithm operating on the simulated data are presented. Estimated spacecraft trajectory and refined map are compared to ground truth to assess estimation accuracy.
In January, 2004, the Mars Exploration Rover (MER) mission landed two rovers, Spirit and Opportunity, on the surface of Mars. Several autonomous navigation capabilities were employed in space for the first time in this mission. ]n the Entry, Descent, and Landing (EDL) phase, both landers used a vision system called the, Descent Image Motion Estimation System (DIMES) to estimate horizontal velocity during the last 2000 meters (m) of descent, by tracking features on the ground with a downlooking camera, in order to control retro-rocket firing to reduce horizontal velocity before impact. During surface operations, the rovers navigate autonomously using stereo vision for local terrain mapping and a local, reactive planning algorithm called Grid-based Estimation of Surface Traversability Applied to Local Terrain (GESTALT) for obstacle avoidance. ]n areas of high slip, stereo vision-based visual odometry has been used to estimate rover motion, As of mid-June, Spirit had traversed 3405 m, of which 1253 m were done autonomously; Opportunity had traversed 1264 m, of which 224 m were autonomous. These results have contributed substantially to the success of the mission and paved the way for increased levels of autonomy in future missions.
A hardware-in-the-loop ground system was developed for simulating a robotic servicer spacecraft tracking a target satellite at short range. A relative navigation sensor package "Argon" is mounted on the end-effector of a Fanuc 430 manipulator, which functions as the base platform of the robotic spacecraft servicer. Machine vision algorithms estimate the pose of the target spacecraft, mounted on a Rotopod R-2000 platform, relay the solution to a simulation of the servicer spacecraft running in "Freespace", which performs guidance, navigation and control functions, integrates dynamics, and issues motion commands to a Fanuc platform controller so that it tracks the simulated servicer spacecraft. Results will be reviewed for several satellite motion scenarios at different ranges. Key words: robotics, satellite, servicing, guidance, navigation, tracking, control, docking.
A machine-vision system capable of detecting obstacles large enough to damage or trap a robotic vehicle is undergoing development. The system includes (1) a pattern generator that projects concentric circles of laser light forward onto the terrain, (2) a stereoscopic pair of cameras that are aimed forward to acquire images of the circles, (3) a frame grabber and digitizer for acquiring image data from the cameras, and (4) a single-board computer that processes the data. The system is being developed as a prototype of machine- vision systems to enable robotic vehicles ( rovers ) on remote planets to avoid craters, large rocks, and other terrain features that could capture or damage the vehicles. Potential terrestrial applications of systems like this one could include terrain mapping, collision avoidance, navigation of robotic vehicles, mining, and robotic rescue. This system is based partly on the same principles as those of a prior stereoscopic machine-vision system in which the cameras acquire images of a single stripe of laser light that is swept forward across the terrain. However, this system is designed to afford improvements over some of the undesirable features of the prior system, including the need for a pan-and-tilt mechanism to aim the laser to generate the swept stripe, ambiguities in interpretation of the single-stripe image, the time needed to sweep the stripe across the terrain and process the data from many images acquired during that time, and difficulty of calibration because of the narrowness of the stripe. In this system, the pattern generator does not contain any moving parts and need not be mounted on a pan-and-tilt mechanism: the pattern of concentric circles is projected steadily in the forward direction. The system calibrates itself by use of data acquired during projection of the concentric-circle pattern onto a known target representing flat ground. The calibration- target image data are stored in the computer memory for use as a template in processing terrain images. During operation on terrain, the images acquired by the left and right cameras are analyzed. The analysis includes (1) computation of the horizontal and vertical dimensions and the aspect ratios of rectangles that bound the circle images and (2) comparison of these aspect ratios with those of the template. Coordinates of distortions of the circles are used to identify and locate objects. If the analysis leads to identification of an object of significant size, then stereoscopicvision algorithms are used to estimate the distance to the object. The time taken in performing this analysis on a single pair of images acquired by the left and right cameras in this system is a fraction of the time taken in processing the many pairs of images acquired in a sweep of the laser stripe across the field of view in the prior system. The results of the analysis include data on sizes and shapes of, and distances and directions to, objects. Coordinates of objects are updated as the vehicle moves so that intelligent decisions regarding speed and direction can be made. The results of the analysis are utilized in a computational decision-making process that generates obstacle-avoidance data and feeds those data to the control system of the robotic vehicle.
A geologist characterizing a field site typically wanders from one interesting geologic feature to another. Performing geology remotely with a mobile robot, we have observed the same behavior: geologists see a visually-interesting feature and wish to approach it for closer inspection. To date, navigating a mobile robot to a visually-interesting feature has been accomplished by driving to a location close to the feature. This introduces two problems: assigning a location to the feature and navigating the rover to that location. In practice, solutions to both of these problems are susceptable to positional error. Fundamentally, we can see where we want the robot to go, but it is difficult to precisely quantify where either the target or the robot are located. The development of vision-based control of robot manipulators suggests an alternative approach for mobile robot explorers. We have developed a vision-based control system that enables the Marsokhod rover to drive to within sampling distance of visually-designated rock or natural feature. We will describe this system and our initial results using it during a field experiment in the Painted Desert of Arizona.
Unmanned micro air vehicles (MAVs) will play an important role in future reconnaissance and search and rescue applications. In order to conduct persistent surveillance and to conserve energy, MAVs need the ability to land, and they need the ability to enter (ingress) buildings and other structures to conduct reconnaissance. To be safe and practical under a wide range of environmental conditions, landing and ingress maneuvers must be autonomous, using real-time, onboard sensor feedback. To address these key behaviors, we present a novel method for vision-based autonomous MAV landing and ingress using a single camera for two urban scenarios: landing on an elevated surface, representative of a rooftop, and ingress through a rectangular opening, representative of a door or window. Real-world scenarios will not include special navigation markers, so we rely on tracking arbitrary scene features; however, we do currently exploit planarity of the scene. Our vision system uses a planar homography decomposition to detect navigation targets and to produce approach waypoints as inputs to the vehicle control algorithm. Scene perception, planning, and control run onboard in real-time; at present we obtain aircraft position knowledge from an external motion capture system, but we expect to replace this in the near future with a fully self-contained, onboard, vision-aided state estimation algorithm. We demonstrate autonomous vision-based landing and ingress target detection with two different quadrotor MAV platforms. To our knowledge, this is the first demonstration of onboard, vision-based autonomous landing and ingress algorithms that do not use special purpose scene markers to identify the destination.
Relative navigation remains the most challenging part of spacecraft rendezvous and docking. In recent years, flash LIDARs, have been increasingly selected as the go-to sensors for proximity operations and docking. Flash LIDARS are generally lighter and require less power that scanning Lidars. Flash LIDARs do not have moving parts, and they are capable of tracking multiple targets as well as generating a 3D map of a given target. However, there are some significant drawbacks of Flash Lidars that must be resolved if their use is to be of long-term significance. Overcoming the challenges of Flash LIDARs for navigation-namely, low technology readiness level, lack of historical performance data, target identification, existence of false positives, and performance of vision processing algorithms as intermediaries between the raw sensor data and the Kalman filter-requires a world-class testing facility, such as the Lockheed Martin Space Operations Simulation Center (SOSC). Ground-based testing is a critical step for maturing the next-generation flash LIDAR-based spacecraft relative navigation. This paper will focus on the tests of an integrated relative navigation system conducted at the SOSC in January 2014. The intent of the tests was to characterize and then improve the performance of relative navigation, while addressing many of the flash LIDAR challenges mentioned above. A section on navigation performance and future recommendation completes the discussion.
An algorithm denoted Gamma-SLAM performs further processing, in real time, of preprocessed digitized images acquired by a stereoscopic pair of electronic cameras aboard an off-road robotic ground vehicle to build accurate maps of the terrain and determine the location of the vehicle with respect to the maps. Part of the name of the algorithm reflects the fact that the process of building the maps and determining the location with respect to them is denoted simultaneous localization and mapping (SLAM). Most prior real-time SLAM algorithms have been limited in applicability to (1) systems equipped with scanning laser range finders as the primary sensors in (2) indoor environments (or relatively simply structured outdoor environments). The few prior vision-based SLAM algorithms have been feature-based and not suitable for real-time applications and, hence, not suitable for autonomous navigation on irregularly structured terrain. The Gamma-SLAM algorithm incorporates two key innovations: Visual odometry (in contradistinction to wheel odometry) is used to estimate the motion of the vehicle. An elevation variance map (in contradistinction to an occupancy or an elevation map) is used to represent the terrain. The Gamma-SLAM algorithm makes use of a Rao-Blackwellized particle filter (RBPF) from Bayesian estimation theory for maintaining a distribution over poses and maps. The core idea of the RBPF approach is that the SLAM problem can be factored into two parts: (1) finding the distribution over robot trajectories, and (2) finding the map conditioned on any given trajectory. The factorization involves the use of a particle filter in which each particle encodes both a possible trajectory and a map conditioned on that trajectory. The base estimate of the trajectory is derived from visual odometry, and the map conditioned on that trajectory is a Cartesian grid of elevation variances. In comparison with traditional occupancy or elevation grid maps, the grid elevation variance maps are much better for representing the structure of vegetated or rocky terrain.
Currently, airport surface communications are fragmented across multiple types of systems. These communication systems for airport operations at most airports today are based dedicated and separate architectures that cannot support system-wide interoperability and information sharing. The requirements placed upon the Communications, Navigation, and Surveillance (CNS) systems in airports are rapidly growing and integration is urgently needed if the future vision of the National Airspace System (NAS) and the Next Generation Air Transportation System (NGATS) 2025 concept are to be realized. To address this and other problems such as airport surface congestion, the Space Based Technologies Project s Surface ICNS Network Architecture team at NASA Glenn Research Center has assessed airport surface communications requirements, analyzed existing and future surface applications, and defined a set of architecture functions that will help design a scalable, reliable and flexible surface network architecture to meet the current and future needs of airport operations. This paper describes the systems approach or methodology to networking that was employed to assess airport surface communications requirements, analyze applications, and to define the surface network architecture functions as the building blocks or components of the network. The systems approach used for defining these functions is relatively new to networking. It is viewing the surface network, along with its environment (everything that the surface network interacts with or impacts), as a system. Associated with this system are sets of services that are offered by the network to the rest of the system. Therefore, the surface network is considered as part of the larger system (such as the NAS), with interactions and dependencies between the surface network and its users, applications, and devices. The surface network architecture includes components such as addressing/routing, network management, network performance and security.
Measuring vehicle locations relative to a driver's vehicle is a critical component in the analysis of driving data from both postanalysis (such as in naturalistic driving studies) or in autonomous vehicle navigation. In this work we describe a method to estimate vehicle positions from a forward-looking video camera using intrinsic camera calibration, estimates of extrinsic parameters, and a convolutional neural network trained to detect and locate vehicles in video data. We compare the measurements we achieve with this method with ground truth and with radar data available from a naturalistic driving study. We identify regions where video is preferred, where radar is preferred, and explore trade-offs between the two methods in regions where the preference is more ambiguous. We describe applications of these measurements for transportation analysis.
A helicopter flight test experiment was conducted at the NASA Ames Research Center to obtain a database consisting of video imagery and accurate measurements of camera motion, camera calibration parameters, and true range information. The database was developed to allow verification of monocular passive range estimation algorithms for use in the autonomous navigation of rotorcraft during low altitude flight. The helicopter flight experiment is briefly described. Four data sets representative of the different helicopter maneuvers and the visual scenery encountered during the flight test are presented. These data sets will be made available to researchers in the computer vision community.
Integration of skills into an autonomous robot that performs a complex task is described. Time constraints prevented complete integration of all the described skills. The biggest problem was tuning the sensor-based region-finding algorithm to the environment involved. Since localization depended on matching regions found with the a priori map, the robot became lost very quickly. If the low level sensing of the world is not working, then high level reasoning or map making will be unsuccessful.