How to dynamically drop frames as required to keep transcoding real time?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

How to dynamically drop frames as required to keep transcoding real time?

Adrian Cable
Hi,
I have a question that I thought should have an ‘easy’ answer – I have spent a lot of time searching and I cannot find it, so thought I would ask here.

My application sounds simple. I have an input RTSP stream, which I need to transcode and output as an RTP stream. Command line is very simple, something like:

./ffmpeg -i rtsp://… -vf scale=1024:-1 -c:v libx264 -f rtp rtp://…

Now, ffmpeg is running on a CPU-constrained platform, and it may be that, in order for the transcoding to ‘keep up’ with the input, the output framerate needs to be less than the input framerate, which is totally fine. The problem/question is: how do I get ffmpeg to drop frames as required to keep the pipeline running real time?

I can’t simply ‘guess’ a sustainable output framerate and set using -r XXX or -vf fps=fps=XXX, because the properties of the input can change at any time, so any XXX won’t be constant. Besides, if the ‘guess’ is too high for the CPU power I have available, the encoder won’t keep up with the input frames and so will ‘run away’ (i.e. the time difference between each input frame and its corresponding output frame will get bigger and bigger), which is the problem I am trying to solve. On the other hand, if the ‘guess’ is too low, I will end up with real-time output but I will also be potentially transcoding to a lower output frame rate than I have the CPU power to do, which I don’t want either.

What I need is some kind of dynamic framerate filter that doesn’t take a number at all (or maybe takes simply a maximum), and drops frames if and only if the encoder can’t keep up … so the pipeline produces as high a frame rate output as possible, while still being real time.

I cannot figure out how to do this. Surely this wish can’t be super uncommon as you would imagine it would be quite important in all sorts of streaming applications where unlimited transcoding power is not guaranteed to be available. Can anyone help?

-Adrian
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How to dynamically drop frames as required to keep transcoding real time?

Mark Filipak (ffmpeg)
On 2021-04-05 21:02, Adrian Cable wrote:

> Hi,
> I have a question that I thought should have an ‘easy’ answer – I have spent a lot of time searching and I cannot find it, so thought I would ask here.
>
> My application sounds simple. I have an input RTSP stream, which I need to transcode and output as an RTP stream. Command line is very simple, something like:
>
> ./ffmpeg -i rtsp://… -vf scale=1024:-1 -c:v libx264 -f rtp rtp://…
>
> Now, ffmpeg is running on a CPU-constrained platform, and it may be that, in order for the transcoding to ‘keep up’ with the input, the output framerate needs to be less than the input framerate, which is totally fine. The problem/question is: how do I get ffmpeg to drop frames as required to keep the pipeline running real time?
>
> I can’t simply ‘guess’ a sustainable output framerate and set using -r XXX or -vf fps=fps=XXX, because the properties of the input can change at any time, so any XXX won’t be constant.

XXX *will* be constant and independent of the input frame rate.
"Convert the video to specified constant frame rate by duplicating or dropping frames as necessary."

I hope this helps,
Mark.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How to dynamically drop frames as required to keep transcoding real time?

Adrian Cable
Mark – sorry if I wasn’t clear in my original email. The problem I am describing is real, not theoretical. I have enormous respect for the time of everyone on this list, and would not have emailed if my problem could be solved with “-vsync cfr” or anything as straightforward and as well documented.

XXX *will not* be constant and *will not* be independent of the input frame rate. This is unfortunately the problem I am trying to solve.

XXX will not be constant because the CPU load on the device (due to other running processes) varies over time. If other processes are loading the CPU, there will be less CPU available for encoding, so XXX will be lower. If the CPU load on the device (due to other running processes) is low, XXX will be higher. And, because CPU load (due to other running processes) may change during the transcode, XXX will vary during the transcode.

XXX will not be independent of the input frame rate, because decoding the input frames consumes CPU and memory bandwidth, which leaves less available for the re-encoding part. If the input frame rate is 60 fps, for example, XXX will be lower than if the input frame rate were 5 fps.

-Adrian


From: ffmpeg-user <[hidden email]> on behalf of Mark Filipak (ffmpeg) <[hidden email]>
Date: Wednesday, April 7, 2021 at 6:43 AM
To: [hidden email] <[hidden email]>
Subject: Re: [FFmpeg-user] How to dynamically drop frames as required to keep transcoding real time?
On 2021-04-05 21:02, Adrian Cable wrote:

> Hi,
> I have a question that I thought should have an ‘easy’ answer – I have spent a lot of time searching and I cannot find it, so thought I would ask here.
>
> My application sounds simple. I have an input RTSP stream, which I need to transcode and output as an RTP stream. Command line is very simple, something like:
>
> ./ffmpeg -i rtsp://… -vf scale=1024:-1 -c:v libx264 -f rtp rtp://…
>
> Now, ffmpeg is running on a CPU-constrained platform, and it may be that, in order for the transcoding to ‘keep up’ with the input, the output framerate needs to be less than the input framerate, which is totally fine. The problem/question is: how do I get ffmpeg to drop frames as required to keep the pipeline running real time?
>
> I can’t simply ‘guess’ a sustainable output framerate and set using -r XXX or -vf fps=fps=XXX, because the properties of the input can change at any time, so any XXX won’t be constant.

XXX *will* be constant and independent of the input frame rate.
"Convert the video to specified constant frame rate by duplicating or dropping frames as necessary."

I hope this helps,
Mark.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How to dynamically drop frames as required to keep transcoding real time?

Carl Eugen Hoyos-2
In reply to this post by Adrian Cable
Am Di., 6. Apr. 2021 um 03:02 Uhr schrieb Adrian Cable <[hidden email]>:
>
> The problem/question is: how do I get ffmpeg to drop frames as required to
> keep the pipeline running real time?

This is not possible with ffmpeg, the application.
Did you look at the x264 presets?

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How to dynamically drop frames as required to keep transcoding real time?

Mark Filipak (ffmpeg)
In reply to this post by Adrian Cable
On 2021-04-07 10:10, Adrian Cable wrote:
> Mark – sorry if I wasn’t clear in my original email. The problem I am describing is real, not theoretical. I have enormous respect for the time of everyone on this list, and would not have emailed if my problem could be solved with “-vsync cfr” or anything as straightforward and as well documented.
>
> XXX *will not* be constant and *will not* be independent of the input frame rate. This is unfortunately the problem I am trying to solve.

Sorry, I thought XXX referred to the output -- -vf fps=fps=XXX is *output* frame rate. To the best
of my knowledge, the input frame rate of VFR video can't be specified.

> XXX will not be constant because the CPU load on the device (due to other running processes) varies over time. If other processes are loading the CPU, there will be less CPU available for encoding, so XXX will be lower. ...

If the input FR drops below the CFR output, the 'fps' filter will repeat input frames in order to
maintain CFR output.

>... If the CPU load on the device (due to other running processes) is low, XXX will be higher. ...

If input FR rises above the CFR output, the 'fps' filter will drop input frames in order to maintain
CFR output.

>... And, because CPU load (due to other running processes) may change during the transcode, XXX will vary during the transcode.

It seems to me that what you want is to somehow maintain picture rate regardless of frame rate. I'm
sorry if I misunderstand, but I don't think I can help you. Sorry. I suggest you look into variable
resolution encoding in order to maintain constant picture rate (but at lower resolution if/when
input FR exceeds output FR). I don't know how to do that.

Good hunting,
Mark.

> XXX will not be independent of the input frame rate, because decoding the input frames consumes CPU and memory bandwidth, which leaves less available for the re-encoding part. If the input frame rate is 60 fps, for example, XXX will be lower than if the input frame rate were 5 fps.
>
> -Adrian
>
>
> From: ffmpeg-user <[hidden email]> on behalf of Mark Filipak (ffmpeg) <[hidden email]>
> Date: Wednesday, April 7, 2021 at 6:43 AM
> To: [hidden email] <[hidden email]>
> Subject: Re: [FFmpeg-user] How to dynamically drop frames as required to keep transcoding real time?
> On 2021-04-05 21:02, Adrian Cable wrote:
>> Hi,
>> I have a question that I thought should have an ‘easy’ answer – I have spent a lot of time searching and I cannot find it, so thought I would ask here.
>>
>> My application sounds simple. I have an input RTSP stream, which I need to transcode and output as an RTP stream. Command line is very simple, something like:
>>
>> ./ffmpeg -i rtsp://… -vf scale=1024:-1 -c:v libx264 -f rtp rtp://…
>>
>> Now, ffmpeg is running on a CPU-constrained platform, and it may be that, in order for the transcoding to ‘keep up’ with the input, the output framerate needs to be less than the input framerate, which is totally fine. The problem/question is: how do I get ffmpeg to drop frames as required to keep the pipeline running real time?
>>
>> I can’t simply ‘guess’ a sustainable output framerate and set using -r XXX or -vf fps=fps=XXX, because the properties of the input can change at any time, so any XXX won’t be constant.
>
> XXX *will* be constant and independent of the input frame rate.
> "Convert the video to specified constant frame rate by duplicating or dropping frames as necessary."
>
> I hope this helps,
> Mark.
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
>


--
Mars? We can't live there! Mars is fatal: no magnetosphere to deflect cosmic rays. A tropical
sunshade orbiting Earth would work. It's the only way. We need to start construction now. Forget Mars.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How to dynamically drop frames as required to keep transcoding real time?

Adrian Cable
Mark,

> Sorry, I thought XXX referred to the output -- -vf fps=fps=XXX is *output* frame rate. To the best
> of my knowledge, the input frame rate of VFR video can't be specified.

I’m referring to XXX as the maximum *output* frame rate that the pipeline can support at a given time, while keeping up with the input.

> If input FR rises above the CFR output, the 'fps' filter will drop input frames in order to maintain
> CFR output.

Sure! But this isn’t what I’m looking for. If the actual output FR drops below the FR set (by -r XXX or -vf fps=fps=XXX etc.), the encoder will not drop frames. Instead the delay between input and output will just increase forever. I am looking for a way to make the encoder drop frames if the actual output FR drops below the FR set. For example: if I set -r 15 but (due to CPU limits) the encoder can only manage to produce 6 fps, right now it won’t drop frames, and as a result won’t keep up with the input. I want to make it drop frames if needed so it keeps up with the input.

-Adrian


From: ffmpeg-user <[hidden email]> on behalf of Mark Filipak (ffmpeg) <[hidden email]>
Date: Wednesday, April 7, 2021 at 2:49 PM
To: [hidden email] <[hidden email]>
Subject: Re: [FFmpeg-user] How to dynamically drop frames as required to keep transcoding real time?
On 2021-04-07 10:10, Adrian Cable wrote:
> Mark – sorry if I wasn’t clear in my original email. The problem I am describing is real, not theoretical. I have enormous respect for the time of everyone on this list, and would not have emailed if my problem could be solved with “-vsync cfr” or anything as straightforward and as well documented.
>
> XXX *will not* be constant and *will not* be independent of the input frame rate. This is unfortunately the problem I am trying to solve.

Sorry, I thought XXX referred to the output -- -vf fps=fps=XXX is *output* frame rate. To the best
of my knowledge, the input frame rate of VFR video can't be specified.

> XXX will not be constant because the CPU load on the device (due to other running processes) varies over time. If other processes are loading the CPU, there will be less CPU available for encoding, so XXX will be lower. ...

If the input FR drops below the CFR output, the 'fps' filter will repeat input frames in order to
maintain CFR output.

>... If the CPU load on the device (due to other running processes) is low, XXX will be higher. ...

If input FR rises above the CFR output, the 'fps' filter will drop input frames in order to maintain
CFR output.

>... And, because CPU load (due to other running processes) may change during the transcode, XXX will vary during the transcode.

It seems to me that what you want is to somehow maintain picture rate regardless of frame rate. I'm
sorry if I misunderstand, but I don't think I can help you. Sorry. I suggest you look into variable
resolution encoding in order to maintain constant picture rate (but at lower resolution if/when
input FR exceeds output FR). I don't know how to do that.

Good hunting,
Mark.

> XXX will not be independent of the input frame rate, because decoding the input frames consumes CPU and memory bandwidth, which leaves less available for the re-encoding part. If the input frame rate is 60 fps, for example, XXX will be lower than if the input frame rate were 5 fps.
>
> -Adrian
>
>
> From: ffmpeg-user <[hidden email]> on behalf of Mark Filipak (ffmpeg) <[hidden email]>
> Date: Wednesday, April 7, 2021 at 6:43 AM
> To: [hidden email] <[hidden email]>
> Subject: Re: [FFmpeg-user] How to dynamically drop frames as required to keep transcoding real time?
> On 2021-04-05 21:02, Adrian Cable wrote:
>> Hi,
>> I have a question that I thought should have an ‘easy’ answer – I have spent a lot of time searching and I cannot find it, so thought I would ask here.
>>
>> My application sounds simple. I have an input RTSP stream, which I need to transcode and output as an RTP stream. Command line is very simple, something like:
>>
>> ./ffmpeg -i rtsp://… -vf scale=1024:-1 -c:v libx264 -f rtp rtp://…
>>
>> Now, ffmpeg is running on a CPU-constrained platform, and it may be that, in order for the transcoding to ‘keep up’ with the input, the output framerate needs to be less than the input framerate, which is totally fine. The problem/question is: how do I get ffmpeg to drop frames as required to keep the pipeline running real time?
>>
>> I can’t simply ‘guess’ a sustainable output framerate and set using -r XXX or -vf fps=fps=XXX, because the properties of the input can change at any time, so any XXX won’t be constant.
>
> XXX *will* be constant and independent of the input frame rate.
> "Convert the video to specified constant frame rate by duplicating or dropping frames as necessary."
>
> I hope this helps,
> Mark.
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
>


--
Mars? We can't live there! Mars is fatal: no magnetosphere to deflect cosmic rays. A tropical
sunshade orbiting Earth would work. It's the only way. We need to start construction now. Forget Mars.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How to dynamically drop frames as required to keep transcoding real time?

Mark Filipak (ffmpeg)
On 2021-04-07 21:51, Adrian Cable wrote:
> Mark,
>
>> Sorry, I thought XXX referred to the output -- -vf fps=fps=XXX is *output* frame rate. To the best
>> of my knowledge, the input frame rate of VFR video can't be specified.
>
> I’m referring to XXX as the maximum *output* frame rate that the pipeline can support at a given time, while keeping up with the input.

I think what you want is an encoder with a quality throttle that can be used to dial down the
quality (and hence, reduce encoding time) when it can't keep up with the input at the current
compression factor. I'm pretty sure that such an encoder control exists because what you want to do
is not at all uncommon. That is, if I understand what you want to do.

I don't know how to do it because I don't stream, but I've seen the issue addressed here previously.
Perhaps someone more knowledgeable will read this and help.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How to dynamically drop frames as required to keep transcoding real time?

Adrian Cable
Mark – yes – you’re understanding exactly right! I was thinking specifically of the encoder ‘throttle’ being a frame rate control, but other forms of quality control that could be self-adjusted by the encoder to keep up with the input would also work for me.

I did look through the history here but couldn’t find anything quite applicable. I do appreciate you taking the time to understand my issue! Hopefully someone who has direct experience of how to achieve this will read this as a result and be able to help.

-Adrian


From: ffmpeg-user <[hidden email]> on behalf of Mark Filipak (ffmpeg) <[hidden email]>
Date: Wednesday, April 7, 2021 at 7:19 PM
To: [hidden email] <[hidden email]>
Subject: Re: [FFmpeg-user] How to dynamically drop frames as required to keep transcoding real time?
On 2021-04-07 21:51, Adrian Cable wrote:
> Mark,
>
>> Sorry, I thought XXX referred to the output -- -vf fps=fps=XXX is *output* frame rate. To the best
>> of my knowledge, the input frame rate of VFR video can't be specified.
>
> I’m referring to XXX as the maximum *output* frame rate that the pipeline can support at a given time, while keeping up with the input.

I think what you want is an encoder with a quality throttle that can be used to dial down the
quality (and hence, reduce encoding time) when it can't keep up with the input at the current
compression factor. I'm pretty sure that such an encoder control exists because what you want to do
is not at all uncommon. That is, if I understand what you want to do.

I don't know how to do it because I don't stream, but I've seen the issue addressed here previously.
Perhaps someone more knowledgeable will read this and help.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How to dynamically drop frames as required to keep transcoding real time?

Mark Filipak (ffmpeg)
On 2021-04-07 22:33, Adrian Cable wrote:
> Mark – yes – you’re understanding exactly right! I was thinking specifically of the encoder ‘throttle’ being a frame rate control, but other forms of quality control that could be self-adjusted by the encoder to keep up with the input would also work for me.
>
> I did look through the history here but couldn’t find anything quite applicable. I do appreciate you taking the time to understand my issue! Hopefully someone who has direct experience of how to achieve this will read this as a result and be able to help.

Then you should restart this topic with a subject line that doesn't say "drop frames".

This is dropping frames, and it's exactly what the 'fps' filter does.
[A]  [B]  [C]  [D] [E] [F] [G] [H] [I] [J] ...input
[A]  [B]  [C]  [D]  [E]  [F]  [G]  [I]  [H] ...output CFR ([H] dropped)

This is repeating frames, and it's also exactly what the 'fps' filter does.
[A]  [B]  [C]  [D]   [E]   [F]  [G]  [H]  [I] ...input
[A]  [B]  [C]  [D]  [D]  [E]  [F]  [G]  [H] ...output CFR ([D] repeated)

CFR: constant frame rate.

Of course, the pictures will judder when frames are either dropped or repeated.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How to dynamically drop frames as required to keep transcoding real time?

Nicolas George
In reply to this post by Adrian Cable
Adrian Cable (12021-04-08):
> Mark – yes – you’re understanding exactly right! I was thinking
> specifically of the encoder ‘throttle’ being a frame rate control, but
> other forms of quality control that could be self-adjusted by the
> encoder to keep up with the input would also work for me.
>
> I did look through the history here but couldn’t find anything quite
> applicable. I do appreciate you taking the time to understand my
> issue! Hopefully someone who has direct experience of how to achieve
> this will read this as a result and be able to help.

You realize that you are wasting your time by ignoring Carl Eugen's
authoritative answer, right?

If you want real help with your problem, start by stopping your
top-posting.

Regards,

--
  Nicolas George

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How to dynamically drop frames as required to keep transcoding real time?

Mark Filipak (ffmpeg)
On 2021-04-08 04:56, Nicolas George wrote:

> Adrian Cable (12021-04-08):
>> Mark – yes – you’re understanding exactly right! I was thinking
>> specifically of the encoder ‘throttle’ being a frame rate control, but
>> other forms of quality control that could be self-adjusted by the
>> encoder to keep up with the input would also work for me.
>>
>> I did look through the history here but couldn’t find anything quite
>> applicable. I do appreciate you taking the time to understand my
>> issue! Hopefully someone who has direct experience of how to achieve
>> this will read this as a result and be able to help.
>
> You realize that you are wasting your time by ignoring Carl Eugen's
> authoritative answer, right?
>
> If you want real help with your problem, start by stopping your
> top-posting.
>
> Regards,

For some unknown reason, We don't always get posts. I didn't get Carl Eugen's authoritative answer.
I responded to Adrian's question because no one else seemed to respond. I'm posting this simply to
inform you.

Regards,
Mark.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".