“Who watches the watchmen” a question posed by Alan Moore has much broader implications into the realm of process control. The core of this political sentiment is accountability and it has broad applications to sensors and how we interpret their data.
Equipment fails and it is not a question of if but rather when. A good example is the 2019 Boeing airplane crash where investigations found that cause was a faulty angle of attack sensor. A very good research paper goes deep into this and I will be borrowing on this idea https://www.sciencedirect.com/science/article/abs/pii/S0952197622000744. The root of the argument is how do you trust sensor data. If you have 1 sensor and it is providing you faulty data you do not know, with 2 sensors (one good & one faulty) you can see the sensor data is different but you won’t know which one is right and with finally 3 sensors (two good & one faulty) you can have the sensors ‘vote’ on the correct sensor value and then tell which is faulty. The obvious error is if you have 2 faulty sensors out of 3 the wrong value will be chosen. This can be extended infinitely but the probability of 2 faulty sensors that are regularly checked is unlikely and so 3 sensors is the agreed norm for critical voting processes. The previously noted research paper goes into how 2 real sensors and a virtual sensor could be used to the same effect and has some really interesting implications to save on sensor cost.
Extending this to process control we can have the exact same failure modes. One example is a control valve used to control fluid flow rates. If said control valve fails (“Gasp, by God how could a control valve ever fail or stick” /s) how would we know. The most trivial solution is to have a “watcher” for all control points. If we have a control valve to control fluid flow rate we need a flow meter to measure the effect of that control valve. This way if the control unit is faulty we can measure that and then investigate. The concern is that if the flow meter is faulty we can have false negatives or fail to catch true negatives. For true critical to safety (CTS) or critical to quality (CTQ) parameters is may be beneficial to design processes with redundant sensors or innovative intertwined solutions.
One real life example is a steam heater used to heat a process stream. We had a control valve fail (after root cause analysis with maintenance we found that it was damaged and was very prone to sticking) and the sensor measuring the flow rate (well not really measuring flow rate but rather pressure differential and then estimating flow rate) was not able to pick up the failure. This would be a catastrophic failure leading to over-temp material but we had temperature sensors in the material being heated by the steam and so were able to catch this condition. The main point here is processes overlap. Surface level views may indicate the faulty valve and sensor are isolated but their effects affect downstream processes so it is possible to detect their failures through downstream sensors.
What this also implies is that failures in a large complex processes can be caught by direct process sensors as well as downstream sensors much further down the line. Another example to show the point is in a boiler where if the heater (gas, electric, etc) fails and the temperature sensors itself fails to catch it, downsteam processes that use the boiler stream would be able to catch this failure due to low pressure or low temperature.
Interconnecting a process like this and creating a system to catch failures like this can be very difficult work and poses its own risks but the core tenant of “Who watches the watchmen” is something all process engineers should be aware of as both equipment and sensors fail and may have significant impact to process.
