A data acquisition system being “redundant” does not mean it will not lose data in an outage. Many plants learn this when they see a gap in the production report after a network outage. Three concepts answer three different questions, and each needs its own design.
Redundancy: keep the service running. Buffering: do not lose data during an outage. Recovery: bring system and data back to normal after a failure.
Three concepts, three questions
| Redundancy | Buffering | Recovery | |
|---|---|---|---|
| Question | If a component fails, does the service continue? | Is data lost while the target is unreachable? | How soon, and with what data, do we get back up after a failure? |
| Mechanism | A second component takes over (failover) | Write to a local queue, send in order when the link returns (store-and-forward) | Restore from backup, replay, reconciliation |
| Example | Automatic switch-over between two Kepware Server instances (RedundancyMaster) | A disk-based queue in the collector; resend to the broker or database | Project backup, database backup, gap filling |
| Limit | A short data gap may occur during switch-over | Data is lost if the buffer fills | A backup is current only up to the moment it was taken |
How does buffering work?
The collector (edge) timestamps data at the source. If it can reach the target, it forwards directly. If not, it writes records to a local queue (preferably disk-based so a restart does not affect it) and, when the link returns, resends the records in order. At the target, an id and timestamp prevent the same record being written twice.
How big should the buffer be?
Buffer size is the product of two things: the longest expected outage and the data rate. An example calculation: 2,000 tags, 1 update per second, about 40 bytes per record → ~80 KB per second, ~288 MB per hour. To survive an eight-hour outage you need to reserve ~2.3 GB plus a safety margin. Sending on change (deadband) reduces volume. How long the outage can last is a business decision independent of network and system architecture: “What is the largest data gap that is acceptable?”
Design steps
- Define acceptable loss. Which data must be lossless (batch result, energy meter) and which can be approximate (trend)?
- Provide buffering at the source for critical data. Some collectors (for example some MQTT and database writers) offer a store-and-forward option; verify the configuration in their documentation.
- Prevent duplicates. Use a unique key of id + timestamp at the target (moving PLC data into SQL).
- Think about redundancy separately. A second instance and automatic switch-over for the critical collection server; but close the gap that may occur during switch-over with buffering.
- Rehearse recovery. Test restore time and result from backup periodically; a written plan is not enough.
- Monitor. Raise alerts for buffer fill, queue length and resend rate.
Common mistakes
- The assumption “our server is redundant, so there will be no data loss”.
- Keeping the buffer only in memory; records are lost on restart.
- Creating duplicate records on resend.
- Not defining what happens when the buffer is full (does it drop the oldest or the newest?).
- Assigning the timestamp at resend time, losing the event time (data quality and timestamps).
These principles also apply to OPC Router flows, Kepware Server and MQTT-based architectures (UNS). An outage scenario must always be tested in the integration acceptance test (acceptance tests).
Frequently asked questions
If I have two redundant servers, do I still need buffering?
Yes. Buffering closes the gap that can occur during switch-over or when both servers are unreachable; the two complement each other.
What happens when the buffer fills?
It depends on policy: the oldest records may be dropped or new records rejected. Which is acceptable is a business decision and should be written down up front.
Are recovery and backup the same?
No. Backup stores data; recovery is the process of bringing the system back to working order from that backup, and it must be rehearsed.
