Use source wallclock (NTP) in RTSP/RTCP streams to setpts for inter-media synchronization
I'm working on a project to multiplex several RTSP streams, from cameras, into a simple "grid" output. The issue is that the cameras are intended to form an ultra-wide (panoramic) view and while total latency is never as low as one would like, relatively latency is a real problem.
I'm not expecting frame-accurate synchronization as one might get from genlocked analog or SDI sources, rather superficially synchronized - to within a few hundred msec, as would be sufficient for the visual perception of a live panorama. As source start times and network connection times are not deterministic, manually tuning with incremental absolute offsets wouldn't work (though could help). Currently the best I've been able to get is about 3 seconds differential latency, which is a fun way to do an automated one-person "wave" but not really the desired effect. All devices are synced to NTP, which is more than sufficiently in sync for the application if I could figure out how to use it.
below is the command I'm using to connect, composite, and display that yields the fastest frame rates and least drop frames. Curiously, changing setpts from N/(8.33*TB) to the recommended for live stream setpts='(RTCTIME - RTCSTART) / (TB * 1000000)' for each stream results in really choppy playback (but no improvement in inter-stream sync).
What I was looking for was some way to set offsets to either:
* Least Latency - treat each inbound source as "live" and "best effort" rather than trying to sync to the start of each stream on the receiver (which is fairly arbitrary) or the start of the stream on the source (which is completely arbitrary). I suspect this is doable, but I haven't found an obvious way to do it yet - any advice much appreciated.
* Absolute sync to NTP packets. This seems like it should be the "right" solution. I've found some links that suggest I'm not the only one looking for a multi-source NTP sync solution, but I haven't found anyone with a clear solution.
I've tried the a few setpoints constructions to no obvious improvement in absolute synchronization including:
On Trac #4586 inspired me to try -use_wallclock_as_timestamps, but that seems to be receiver wallclock, not source. The only other option in options_table.h that seems relevant is "start_time_realitme" - but this is encode.
It looks like there was a patch proposed in 2016 as a step toward multi-source synchronization:
libavformat/rtpdec.c seems to parse the NTP RTCP timestamps, but I don't see any method of using them to setpts (or otherwise sync sources), though it looks like this was a topic of discussion back in 2008:
Which is to say, I looked through all the hints I could find, and while there are many tantalizing clues, nothing indicates an obvious solution to me. Is there perhaps some obvious parameter I'm just not understanding or other solution escaping my search engine skills?
Re: Use source wallclock (NTP) in RTSP/RTCP streams to setpts for inter-media synchronization
-------- Original Message --------
Subject: Re: [FFmpeg-user] Use source wallclock (NTP) in RTSP/RTCP streams to setpts for inter-media synchronization
From: Carl Eugen Hoyos <[hidden email]>
To: FFmpeg user questions <[hidden email]>
Date: 2021-01-18 00:24+0300
> As long as you are using the setpts filter, you disable all
> synchronisation and you
> cannot get the output you want.
Makes sense, thanks for the hint. A link to a screen cap of the command without setpts attached below. The time code is burned in at the sources. I let it run for a few minutes and the streams did not sync.