Why run inference at the edge
Cloud analytics are valuable, but some industrial uses cannot depend on them. A vision inspection decision has to be made as the part passes the camera. A vibration anomaly detector should keep working when the wide-area link is down. Raw images and waveforms are expensive to send and sometimes should not leave the site. Running the trained model on a local device addresses latency, availability, bandwidth and data locality together.
ONNX and ONNX Runtime in brief
ONNX (Open Neural Network Exchange) is an open format for representing trained models. A model trained in a common framework is exported to ONNX, and ONNX Runtime executes it on the target device, on CPU or, through execution providers, on accelerators. It supports both x86 and ARM hardware, so the same exported model can run on an industrial PC or a compact gateway. Training stays where compute is plentiful; only inference moves to the edge.
Measure on the target, not on your laptop
Model size and speed claims mean little until measured on the actual device with realistic input. Record the latency distribution (median and worst case, not just the average), memory use and CPU load while the rest of the edge workload is running. A model that is fast alone may miss its deadline next to the data-acquisition and buffering containers.
Quantization: a trade-off, not a free lunch
Quantization stores weights, and sometimes activations, with fewer bits, for example INT8 instead of 32-bit floating point. It reduces model size and often speeds inference, particularly on small CPUs. It can also reduce accuracy. Validate the quantized model on your own data, compare it with the original and keep the original as a fallback.
Put safeguards around the model
- Validate inputs. Check ranges, missing values and sensor quality before they reach the model. A model fed garbage returns confident garbage.
- Use confidence thresholds with a defined fallback, such as flagging the item for human review when confidence is low.
- Never let model output drive a machine action without independent safeguards that do not depend on the model being right.
- Log inputs, outputs and confidence for later review.
Keep results moving
Buffer results locally and forward them to the historian or broker when the link allows, with timestamps from the source (see MQTT topic design for a payload layout). Plan for the replay burst after an outage with the data budget calculator.
Manage the model lifecycle
- Version every model and record which version produced which result.
- Deliver models as signed packages through the same pipeline as other edge software (CI/CD for edge devices).
- Roll out in stages, and run a new version in shadow mode beside the old one before it takes over.
- Keep a rollback path that does not need the network.
- Monitor input distribution and accuracy over time; processes drift, and a model trained on last year’s conditions will degrade quietly.
Prepare the training data carefully too: see data preprocessing for industrial ML. For where edge hardware fits, read edge computing for industrial IT teams.