The 'problem' with automation - Inappropriate feedback and interaction, not 'over-automation'

Abstract

Automation in high-risk industry is often blamed for causing harm and increasing the chance of human error when failures occur. It is proposed that the problem is not the presence of automation, but rather its inappropriate design. The problem is that the operations are performed appropriately under normal conditions, but there is inadequate feedback and interaction with the humans who must control the overall conduct of the task. The problem is that the automation is at an intermediate level of intelligence, powerful enough to take over control which used to be done by people, but not powerful enough to handle all abnormalities. Moreover, its level of intelligence is insufficient to provide the continual, appropriate feedback that occurs naturally among human operators. To solve this problem, the automation should either be made less intelligent or more so, but the current level is quite inappropriate. The overall message is that it is possible to reduce error through appropriate design considerations.