Video / Data Synchronization

The STANAG Player SDK provides automatic synchronization of video, audio, KLV metadata, and private data frames based on their presentation timestamps (PTS).
Once matching frames are found across all active PIDs, the SDK triggers a callback with the synchronized data, forwarding a list of synced video/data frames to the application layer.

You can control synchronization tolerance using the MaxDelay parameter, which defines the maximum allowed time difference (in ms) between frames to be considered "in sync."

Note: Timestamp (PTS) information is only available for KLV packets when operating in SYNC_KLV mode.

Sync Scenarios

Frame Capture Mode

This mode is primarily used when processing uncompressed video frames is required.
In Frame Capture Mode, the SyncFrameEvent provides a list of fully synchronized frames and data packets. No further synchronization is needed on your side.

Note:
The list is delivered immediately once it's available, so you can begin processing as soon as possible.
However, directly displaying the frames "as-is" may result in choppy playback.

To enable smooth playback:

  • Use the timeToRender value included in each packet (when PTS is available) to delay rendering appropriately.
  • Implement a simple FIFO queue to buffer the list of frames.
  • Wait until the correct presentation time before rendering each frame.

If a packet does not have valid timing info, its timeToRender will be set to -1. In such cases, you can fall back to the video frame's timing information in the same list for scheduling playback.

struct StreamFrameInfoWr
{
    StreamType      streamType;     // specifies the packet PID type
    int             streamId;       // pid number of the stream
    byte[]          data;           // data buffer
    unsigned long   dataSize;       // frame's data buffer size
    Int64           timeStamp;      // time of IMediaSample
    long            timeToRender;   // time to render in msec
};

Video Rendering Mode

In Video Rendering Mode, the SDK ensures smooth playback by managing the frame presentation interval internally.

In this mode:

  • The SyncFrameEvent still provides a list of synchronized frames and data packets.
  • However, these frames may not exactly match the video frame currently displayed in the window at that moment.

How it works

  • Internally, the SyncSampleGrabber filter outputs video (YUV) frames.
  • These frames are passed through a scaler and rendered using VMR9 (Video Mixing Renderer 9).
  • VMR9 schedules frame display based on the sample's PTS (Presentation Time Stamp).
  • When the source is a file, the grabber filter’s output FIFO typically contains several frames in advance.
  • This buffering provides extra time for data processing, as you receive frames slightly ahead of display time.

Metadata Synchronization for Overlay

If you're using metadata (e.g., KLV) to overlay on the video (e.g., telemetry display), and want it to match the currently visible frame:

  • Delay the metadata using a FIFO queue, matching the frame display time.
  • This ensures the data is aligned with the actual video shown to the user.

Note:
This extra buffering and synchronization step is not needed if you are using Frame Accuracy Mode.
For more details, see Frame Accuracy Mode.