ffmpeg bottleneck, thread-safe ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

ffmpeg bottleneck, thread-safe ?

julien.gibory
Hello,

I share with you a performance trouble I get while transcoding some high resolution inputs.
I'm currently using a "huge" VM to transcode an hevc 8K movie to a lower bitrate to perform some tests.

The VM :
76 vCPUs (Skylake Gold 6161 v5 bi-pro 2.2Ghz) | 304 GB | cc3.19xlarge.4 | OBS CentOS 7.7

Transcoding :

*        Command :

o   /home/local/ffmpeg_build/bin/ffmpeg -i /data/testjulien/iss_ep01telco_DT_video360_h265_7680x7680_10b_250Mbps_stereo_v001.mp4 -maxrate 25M -bufsize 200M  -b:v 25000k -c:v libx265 -pix_fmt yuvj420p -c:a copy  -f hls -hls_time 4 -hls_flags independent_segments -hls_list_size 5 -strftime 1 -hls_segment_filename /data/www/httpsvmmalo/test_capa/file_%m-%d_%H-%M-%S.m4s /data/www/httpsvmmalo/test_capa/stream.m3u8

*        Version :

o   ffmpeg version 4.3.2

*        input file :

o   Stream #0:0(und): Video: hevc (Main 10) (hev1 / 0x31766568), yuv420p10le(tv, bt709/bt709/unknown), 7680x7680 [SAR 1:1 DAR 1:1],                 202195 kb/s, 30 fps, 30 tbr, 30k tbn, 30 tbc (default)

*        Output file :

o   Stream #0:0(und): Video: hevc (libx265),                                          yuvj420p(pc),                                                  7680x7680 [SAR 1:1 DAR 1:1], q=-1--1, 25000 kb/s,   30 fps, 90k tbn,              30 tbc (default)

During the transcoding job, I see a very low speed, probably processing the huge 8K source file.

frame=   32 fps=3.1 q=0.0  size=N/A time=00:00:01.53 bitrate=N/A speed=0.148x
frame=  333 fps=2.6 q=1.9  size=N/A time=00:00:11.58 bitrate=N/A speed=0.0911x
frame= 1133 fps=2.6 q=17.9 size=N/A time=00:00:38.25 bitrate=N/A speed=0.0869x
frame= 2023 fps=2.7 q=26.0 size=N/A time=00:01:07.90 bitrate=N/A speed=0.0909x
frame= 2807 fps=2.8 q=13.2 size=N/A time=00:01:34.03 bitrate=N/A speed=0.0923x

I see my machine is not totally used at a system performances point of view (see htop below) !

Questions :

*        Do you know what is limiting the transcoding speed of ffmpeg ?

o   Are some tasks monothreaded in the ffmpeg code ?

o   Are there some missing improvements in my ffmpeg command ?

o   Are FUTEX communications (shown by strace) limiting the transcoding speed?

[cid:image003.jpg@01D72198.E24865A0]





Best regards,

[Logo Orange]<http://www.orange.com/>

Julien Gibory
Orange<http://annuaire.sso.infra.ftgroup/entities/ou=Orange,ou=entities>/IMT<http://annuaire.sso.infra.ftgroup/entities/ou=IMT,ou=Orange,ou=entities>/OLS<http://annuaire.sso.infra.ftgroup/entities/ou=OLS,ou=IMT,ou=Orange,ou=entities>/HOME<http://annuaire.sso.infra.ftgroup/entities/ou=HOME,ou=OLS,ou=IMT,ou=Orange,ou=entities>/TVSKC<http://annuaire.sso.infra.ftgroup/entities/ou=TVSKC,ou=HOME,ou=OLS,ou=IMT,ou=Orange,ou=entities>/TAE<http://annuaire.sso.infra.ftgroup/entities/ou=TAE,ou=TVSKC,ou=HOME,ou=OLS,ou=IMT,ou=Orange,ou=entities>
Fixe : +33 2 99 39 94 97 <https://monsi.sso.francetelecom.fr/index.asp?target=http%3A%2F%2Fclicvoice.sso.francetelecom.fr%2FClicvoiceV2%2FToolBar.do%3Faction%3Ddefault%26rootservice%3DSIGNATURE%26to%3D+33%202%2099%2039%2094%2097%20>
Mobile : +33 6 72 21 39 78 <https://monsi.sso.francetelecom.fr/index.asp?target=http%3A%2F%2Fclicvoice.sso.francetelecom.fr%2FClicvoiceV2%2FToolBar.do%3Faction%3Ddefault%26rootservice%3DSIGNATURE%26to%3D+33%206%2072%2021%2039%2078%20>
[hidden email]<mailto:[hidden email]>


_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

image001.png (226 bytes) Download Attachment
image003.jpg (391K) Download Attachment
strace_1.txt (4K) Download Attachment
ffmpeg.txt Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ffmpeg bottleneck, thread-safe ?

Carl Eugen Hoyos-2
Am Do., 25. März 2021 um 17:04 Uhr schrieb <[hidden email]>:

> 76 vCPUs (Skylake Gold 6161 v5 bi-pro 2.2Ghz) | 304 GB | cc3.19xlarge.4 | OBS CentOS 7.7

> o   /home/local/ffmpeg_build/bin/ffmpeg -i /data/testjulien/iss_ep01telco_DT_video360_h265_7680x7680_10b_250Mbps_stereo_v001.mp4 -maxrate 25M -bufsize 200M  -b:v 25000k -c:v libx265 -pix_fmt yuvj420p -c:a copy  -f hls -hls_time 4 -hls_flags independent_segments -hls_list_size 5 -strftime 1 -hls_segment_filename /data/www/httpsvmmalo/test_capa/file_%m-%d_%H-%M-%S.m4s /data/www/httpsvmmalo/test_capa/stream.m3u8

When asking questions on this mailing list, you are expected to
provide the command
line you tested together with the complete, uncut console output,
attachments are
rarely useful-

> o   ffmpeg version 4.3.2

Please test current FFmpeg git head.

In general, video decoding can be parallelized (although restrictions
exist), encoding
parallelization has limitations, it is therefore likely that you found
a bottleneck of x265.
But to be able to clearly decide if this is true, you can first test
decoding speed (only),
then test encoding from testsrc2 (which should be faster than decoding
an 8k video),
and don't use hls muxing to rule out any bottleneck there.

> Ce message et ses pieces jointes peuvent contenir des informations confidentielles

Please remove this nonsense from emails sent to a public mailing list.

I wonder what "thread-safe" means in your subject...

Carl Eugen

PS:
Before you ask the x265 developers why they are not maxing out your CPUs:
There was a patch once that allowed the x265 encoder to use more cpu cycles
but since it reduced encoding speed it was decided not to merge it...
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".