Practical guides · Practical guide

GxP historian data: compression and interpolation

A trend can look normal while a brief limit excursion is absent from the chosen display. Trace historian data from sensor through processing to evaluation to establish what the stored and displayed values can support.

Table of contents

At a glance

Are interpolated or compressed historian values original measurements?

Not automatically. Distinguish acquired, stored, calculated and displayed values. Interpolation estimates intermediate values according to a rule. Compression may reduce or aggregate information. Fitness for a GxP use depends on capture, configuration, timing, quality flags and the decision being made. Calling data raw does not establish an unchanged, complete measurement series.

Describe the entire data pathway

A process parameter can originate at the sensor, be processed in the controller, transmitted via an interface, stored in the historian, and recalculated in the report. Each step may influence the validity of the conclusion. Describe the measurement point, unit, data acquisition rate, transmission, storage, and utilized analysis. The system boundary aligns with the intended use. A review of the trend alone fails to detect changes occurring prior to the trend.

Determine which information is required for the decision. Is it a long-term trend, a maximum value, an out-of-specification time point, or a batch-specific process evaluation? The same presentation may be appropriate for one purpose and overly coarse for another. A clear definition of "raw value" is necessary: the directly recorded measurement, a pre-processed control value, or an event stored in the archive. Ensure that the terms used in the project are clearly and consistently defined.

Sources: [1] [2]

Data capture, exception handling, and compression differ

Manufacturers use different mechanisms to limit data volumes. The cited PI documentation distinguishes Exception from Compression at different points in the data flow. Their actual effects depend on the components and settings. The manufacturer's recommended starting values are not GMP acceptance limits. Determine what loss of detail the intended use of the data can tolerate.

In addition, distinguish between reduced data retention and lossless file compression. A ZIP archive and a time-series selection of stored events serve distinct purposes. Verify that short changes, steady phases, and rapid transitions remain adequately represented. A blanket disablement of all compression may generate substantial data volumes, but it is not itself a automated quality assurance. Crucially, the collection of required information must be justified and validated.

Sources: [3]

Interpolation generates an inferred conclusion

Interpolation determines values between existing data points based on a defined rule. Linear interpolation assumes a continuous trend between two values; a stepwise representation treats different states accordingly. The appropriate method depends on the signal type. A binary valve signal must not be evaluated as a continuous parameter, such as temperature, without proper technical justification. The display setting may alter interpretation, even though the recorded events remain unchanged.

A regularly spaced output is therefore not automatically a list of actually measured values. Indicate source and calculation method. Assess how missing data, invalid quality attributes, and large time intervals are handled. A visually continuous line may mask a data gap. Define when interpolation is acceptable and when evaluation must indicate an incomplete or unassessable time period.

Example: a brief temperature peak

In the example, the process temperature is typically 20 °C. For a short period, the temperature rises to 28 °C, while the allowable upper limit in the considered process is 25 °C. A smoothed minute-averaged value may remain below 25 °C. This does not imply that no deviation occurred. The correct assessment depends on whether the requirement specifies a maximum value, a mean value, or an allowable duration.

Build a well-known signal profile featuring a peak, plateau, drop, and return. Compare recorded values, archived events, and any relevant outputs. The independent setpoint contains the professionally relevant characteristics. Do not merely assess whether a curve appears plausible. The following matrix serves as a standalone work aid; the numbers and limits apply only to this example and do not represent general specifications for temperature monitoring.

Signal features, potential contamination and testing questions
Signal featurePossible contaminationAppropriate test question
Sharp peak Notification hides maximum Can the reviewer still detect the excursion?
Long Plateau A few archived records Is the duration correctly reconstructible?
Data gap Line connects the missing area Is missing information visible?
Binary state Linear Interpolated Values Does the analysis match the signal type?
Poor quality Value used without identifier Is an invalid state taken into account?
Shift time change Duplicate local time Is the sequence unambiguously defined?

Sources: [1]

Properly calculate mean values and time proportions technically accurate

Distinguish between event-weighted and time-weighted average values. Two stored values may represent entirely different time periods. In the example calculation, 10 corresponds to nine minutes and 20 to one minute. A simple average of the two event values yields 15; a time-weighted average over ten minutes results in 11. The appropriate calculation depends on the signal model and the specific question being addressed.

Define interval boundaries, inclusion of boundary points, and handling of invalid time periods. A duration exceeding the threshold limit may differ if linear interpolation is applied between two events or if the previous state is carried forward. Retain the selected rule and validate it using independently calculated examples. Correct operation of a reporting tool does not confirm that its default calculation aligns with process requirements.

Timestamps, units, and quality attributes are retained

Determine what the timestamp represents: measurement time, receipt time, archiving time, or later calculation time. Delayed transmission can cause these time points to diverge. For process assessment, the actual event reference is often decisive. Check time zone, resolution, synchronization, and the handling of delayed values. Assigning a correct value to the wrong time point can distort the temporal association to a batch.

You will also receive unit, signal status, and required metadata. "Bad", missing, or outside a valid range must not be silently treated as a normal measurement value for further processing. Define the handling of substitute values and manual corrections. Review historically altered scaling or units. If a report references old values with current units, the formally readable analysis may be technically incorrect. The required historical context is part of the traceability concept.

Sources: [1] [2]

Monitor changes to tags and reports

Configuration settings, data capture rate, compression parameters, and calculations can impact documentation quality. Therefore, relevant adjustments should be treated as evaluated changes. Document the baseline, rationale, impact, and appropriate validation evidence. A tighter threshold may render previously sufficient temporal resolution inadequate. The evaluation must therefore not be limited to technical parameter modifications.

Review the rights assigned for configuration, retrospective data modifications, and reporting functionality. The reviewer must understand the relevant changes and be able to access the required history. For new dashboards, evaluate the underlying queries, filters, and aggregation methods. A similar trend may arise from a different data source or retrieval mechanism. Retain the validated calculation logic in alignment with the current version to ensure future results remain reconstructible.

Sources: [1]

Archiving and Suitability Decision

An export must contain the values required for the intended purpose and the necessary context. A rasterized CSV may contain interpolated values and shall not be unverifiedly designated as a complete export of stored events. Define retrieval mode, time period, filters, unit, and quality attributes. Additionally, verify whether calculations and historical configurations are required for future interpretation.

The suitability decision integrates data flow, information reduction, analysis, and process requirements. Specific boundaries are defined. If a brief, relevant exceedance remains undetectable within the intended data capture, a detailed closing report provides no further benefit; the architecture or process rule must be adjusted. A robust historian audit trail demonstrates which statements are credible and defensible. It does not guarantee that any subsequent requested analysis can be derived from the existing data.

Sources: [1] [2]

Project experience and collaboration

Compare the displayed trend with the stored event

A smooth trend line can conceal a short spike. With IT and the process owners, we trace known events from capture and storage to the display or report. We distinguish timing, quality flags and the rules for aggregation or interpolation. A focused comparison with a suitable reference may suffice for one report. If the same historian supports several quality decisions, we define the data each decision needs. This establishes which display and retention arrangements serve each purpose.

Frequently asked questions about Interpreting historian data

Is every historian value a raw measurement value?

No. A value may already have been processed, condensed, or calculated. Describe its origin and data pathway. Crucially, determine which information remains relevant for the intended use. The designation of "raw data" on a surface does not automatically imply an unaltered, complete measurement series.

Is data compression prohibited in the GxP environment?

A general statement would be incorrect. Distinguish between lossless data storage and the selection or compression of measurement values. Assess and evaluate the impact on the required process information. Technical manufacturer recommendations are not universal GMP limits. The setting must align with the specific technical requirements.

Sources: [3] [1]

Can a mean value limit exceedance exclude a batch?

Not without appropriate specifications. An average can mask short-term peaks. When a process requires a maximum value or an allowable duration of exceedance, such statements shall be evaluated separately. Define calculation methodology, time period, and handling of data gaps prior to deriving any conclusion from an analysis.

What distinguishes event and time-averaged mean values?

An event-based measurement weights stored values equally. A time-based average takes into account their temporal representation according to the defined signal model. With irregular event intervals, results may deviate significantly. Verify the required calculation using known examples and document interpolation methods and interval boundaries.

Is a screenshot of the trend sufficient as evidence?

It can document the representation, but does not verify the source, complete data, access mode, and calculation logic. Supplement the controlled configuration state and appropriate reference comparisons. Particularly in compression and interpolation, a visually plausible curve may obscure materially relevant information.

What must a Historian-Export include?

This depends on the subsequent statement intended. Commonly relevant factors include values with clear temporal reference, unit of measure, quality attributes, signal identity, and required configuration context. Verify whether stored events or derived raster values are exported. The file format alone does not determine completeness or suitability.

Sources: [2]

Sources and review date

As of October 14, 2026. Regulatory classification and manufacturer responsibilities are separately documented. Examples, test matrices, and templates are standalone reference tools; they are customized for actual usage purposes.

  1. European Commission: EU GMP Annex 11, Computerised Systems (2011)Current version as per EudraLex inventory as of 14.10.2026. Application, risks, operation, and data lifecycle; specific testing proposals are derived independently.
  2. PIC/S: PI 041-1, Good Practices for Data Management and Integrity (2021)Inspection Guidance on Data Integrity; in particular electronic data, transmission, access, and review.
  3. AVEVA: Exception, Compression and their Impacts on PI System PerformanceOfficial manufacturer statement regarding Exception and Compression as data filters. Parameters and specific components must still be verified on a version- and application-specific basis; reviewed on 14.10.2026.

Sources checked: .

Daniel Herrmann, Daniel Herrmann Consulting

Daniel Herrmann Consulting

How we can support you

Daniel Herrmann Consulting combines professional assessment with hands-on work on the specific system. During the first meeting, we clarify your intended use, existing documentation, and the next most appropriate action step.

  • Computer system validation →

    Bring your GxP systems into validated operation, from laboratory software, MES and eQMS to ERP. We define the scope, plan risk-based tests using GAMP 5 and hand the evidence to QA for release.

Send a project enquiry Customer testimonials →