Encoding x264 hangs in a high turn over pipeline

[Luke]

Hello,
We are testing ffmpeg in our compression pipeline as our commercial
software does not provide the functionality we need.

We run a multithreaded compression (ffmpeg multithreading isn't used,
our own solution is) using the following command:
ffmpeg -i #input# -f mp4 -vcodec libx264 -s 1024x576 -b:v 2000k -g 33
-acodec libmp3lame -b:a 128k -ar 44100 -vf "movie=#logo# [wm]; [in][wm]
overlay=10:main_h-overlay_h-10 [out]" -y #output#

Approx. 75% threads complete without any problems, the remaining ones
get stuck - cpu usage on each thread drops to 0, memory usage gets stuck
at around 300MB.

Initially we used aac and libvo_aacenc coders for audio, but the hangs
seemed more frequent.

Tested on a few recent ffmpeg builds, including the one compiled on 20/07.

Any help would be greatly appreciated.
Many thanks,
Luke
--

*Luke Niejadlik*
Audiomotion Studios Ltd.
T: [phone]
F: [phone]
www.audiomotion.com <http://www.audiomotion.com/>

Develop Awards - Best Services Winner 2011, 2009 & 2004

*/JOIN THE MOVEMENT!/* - www.facebook.com/audiomotion.studios
<http://www.facebook.com/audiomotion.studios> - twitter.com/audiomotion
<http://twitter.com/audiomotion> - http://www.join-the-movement.co.uk/

_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[Nicolas George-2]

L'octidi 8 thermidor, an CCXX, Luke a écrit :

> We are testing ffmpeg in our compression pipeline as our commercial
> software does not provide the functionality we need.
>
> We run a multithreaded compression (ffmpeg multithreading isn't
> used, our own solution is) using the following command:
> ffmpeg -i #input# -f mp4 -vcodec libx264 -s 1024x576 -b:v 2000k -g
> 33 -acodec libmp3lame -b:a 128k -ar 44100 -vf "movie=#logo# [wm];
> [in][wm] overlay=10:main_h-overlay_h-10 [out]" -y #output#
>
> Approx. 75% threads complete without any problems, the remaining
> ones get stuck - cpu usage on each thread drops to 0, memory usage
> gets stuck at around 300MB.

Can you reproduce the problem with another video codec, such as the internal
MPEG4 encoder, or maybe even with -f null?

Can you try to find out where ffmpeg is stuck? A way to do that would be to
run the debug version (ffmpeg_g), and when it is stuck, attach it with gdb
(gdb ffmpeg_g $(pidof ffmpeg_g)), get a backtrace and then detach it: a few
iterations of that would show where ffmpeg spends most of its time.

> Initially we used aac and libvo_aacenc coders for audio, but the
> hangs seemed more frequent.

You will probably get more quality out of lame anyway.

Regards,

--
  Nicolas George

_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[rogerdpack2]

> We run a multithreaded compression (ffmpeg multithreading isn't used, our
> own solution is) using the following command:
> ffmpeg -i #input# -f mp4 -vcodec libx264 -s 1024x576 -b:v 2000k -g 33
> -acodec libmp3lame -b:a 128k -ar 44100 -vf "movie=#logo# [wm]; [in][wm]
> overlay=10:main_h-overlay_h-10 [out]" -y #output#

What do you mean that your own solution is using multithreading?

> Approx. 75% threads complete without any problems, the remaining ones get
> stuck - cpu usage on each thread drops to 0, memory usage gets stuck at
> around 300MB.

What OS?
Full uncut complete console and output please?
Also gdb backtraces of stuck threads?
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[Luke]

Hello,
Thanks for your responses.
> Can you reproduce the problem with another video codec, such as the internal
> MPEG4 encoder, or maybe even with -f null?
I tested this using internal mp4, as well as libvpx. In both cases the
pipeline completed without a glitch on a sample of 1400 videos.
> Can you try to find out where ffmpeg is stuck? A way to do that would be to
> run the debug version (ffmpeg_g), and when it is stuck, attach it with gdb
> (gdb ffmpeg_g $(pidof ffmpeg_g)), get a backtrace and then detach it: a few
> iterations of that would show where ffmpeg spends most of its time.
Unfortunately the system is running Windows 7 and we have no option of
running linux (users not linux friendly).
> What do you mean that your own solution is using multithreading?
As you will notice from the attached logs the pipeline is based on Ruby.
The whole objective of it is to ingest files from the capture servers,
rename them, create branding, compress & watermark, and distribute
compressed output to client machines as well as HTML5 web app. Which
should explain testing libvo_aac and VP8. The Ruby app creates runs
multiple threads based on the size of the queue.
> What OS?
Windows 7
> Full uncut complete console and output please?
Logs from FFMPEG attached. One of a comression which completed fine, and
3 which got stuck. Getting stuck seems a bit random, as force closing
and reopening the thread usually allows the compression to complete
fine. If not on the 1st try, then 2nd. Usually 1st re-run is enough.
We're trying another approach - building timeout into the multithreading
process. However it would be quicker and more efficient if we were able
to run FFMPEG without a glitch.
> Also gdb backtraces of stuck threads?
Correct me if I'm wrong, but I won't be able to get it in Windows, will I?

Many thanks for your help,

*Luke Niejadlik*
Audiomotion Studios Ltd.
T: [phone]
F: [phone]
www.audiomotion.com <http://www.audiomotion.com/>

Develop Awards - Best Services Winner 2011, 2009 & 2004

*/JOIN THE MOVEMENT!/* - www.facebook.com/audiomotion.studios
<http://www.facebook.com/audiomotion.studios> - twitter.com/audiomotion
<http://twitter.com/audiomotion> - http://www.join-the-movement.co.uk/




_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[rogerdpack2]

> I tested this using internal mp4, as well as libvpx. In both cases the
> pipeline completed without a glitch on a sample of 1400 videos.

This sounds vaguely familiar...f it fails with libx264 but works well
with the others, you may be running up against a threading issue we're
trying to resolve in the zeranoe builds.
Could you try with this build and see if it works?
http://x32.elijst.nl/FFmpeg-20120622.7z

>> Also gdb backtraces of stuck threads?
>
> Correct me if I'm wrong, but I won't be able to get it in Windows, will I?

Believe it or not the mingw people have ported gdb to windows, and it
works quite well.
http://sourceforge.net/projects/mingw-w64/files/External%20binary%20packages%20(Win64%20hosted)/gdb/
It can even attach to remote processes.
You'll want to be running a debug version of ffmpeg, though even a non
debug version of it might show some interesting material.

basically start gdb, then
(gdb) attach <pidd>
(gdb) thread apply all bt
GL!
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[Luke]

> This sounds vaguely familiar...f it fails with libx264 but works well
> with the others, you may be running up against a threading issue we're
> trying to resolve in the zeranoe builds.
> Could you try with this build and see if it works?
> http://x32.elijst.nl/FFmpeg-20120622.7z
I tried this, but the overlay filter did not work in it. However going
along theses lines I worked my way down the previous builds and found
one that works without a glitch. Which probably confirms what you
suspected anyway.

> Believe it or not the mingw people have ported gdb to windows, and it
> works quite well.
> http://sourceforge.net/projects/mingw-w64/files/External%20binary%20packages%20(Win64%20hosted)/gdb/
> It can even attach to remote processes.
> You'll want to be running a debug version of ffmpeg, though even a non
> debug version of it might show some interesting material.
>
> basically start gdb, then
> (gdb) attach<pidd>
> (gdb) thread apply all bt
> GL!

I didn't have a need to do this in the end. I tested the build in most
likely scenarios off line. I also replaced the build on the live
compression system and ran it all day yesterday - older build worked
like a charm. However if you'd find the outcome of thread traces useful
for your purposes I'm happy to do some testing for you.

Once again, thank you for your help. Fewer late nights for me now!
All the best,
Luke

_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[rogerdpack2]

>> Could you try with this build and see if it works?
>> http://x32.elijst.nl/FFmpeg-20120622.7z
>
> I tried this, but the overlay filter did not work in it. However going along
> theses lines I worked my way down the previous builds and found one that
> works without a glitch. Which probably confirms what you suspected anyway.

Which builds "last worked" and which ones "first started showing" the error?
-r
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[Maxim]

Hi Guys,

I've a similar problem with frozen ffmpeg instances, 2 of 8 instances is
frozen, cpu usage on each instance drops to 0. I've tried to remove
-threads parameter or set -threads 1, result is the same.

I've uploaded video file and 8 cmds to fileshare:
http://depositfiles.com/files/q51mub4qq
http://turbobit.net/7ff93nfen4pw.html

You can use start.bat to start 8 instances at once.

I've attached a log files of frozen ffmpegs:
log_6.txt
log_7.txt

ffmpeg version N-42939-g236ecc3 Copyright (c) 2000-2012 the FFmpeg 
developers
built on Jul 26 2012 20:50:57 with gcc 4.7.1 (GCC)

Windows 7 x64
8Gb memory
Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz

--
Kind Regards, Maxim

_______________________________________________
ffmpeg-user mailing list
[e-mail]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[Luke]

For me using one of the older builds sorted the problem out - try this
(assuming you're using Windows):
http://ffmpeg.zeranoe.com/builds/win64/static/ffmpeg-20120409-git-6bfb304-win64-static.7z

Cheers,

*Luke Niejadlik*
Audiomotion Studios Ltd.
T: [phone]
F: [phone]
www.audiomotion.com <http://www.audiomotion.com/>

Develop Awards - Best Services Winner 2011, 2009 & 2004

*/JOIN THE MOVEMENT!/* - www.facebook.com/audiomotion.studios
<http://www.facebook.com/audiomotion.studios> - twitter.com/audiomotion
<http://twitter.com/audiomotion> - http://www.join-the-movement.co.uk/

On 03/08/2012 10:16, Maxim Kopeyka wrote:

> Hi Guys,
>
> I've a similar problem with frozen ffmpeg instances, 2 of 8 instances
> is frozen, cpu usage on each instance drops to 0. I've tried to remove
> -threads parameter or set -threads 1, result is the same.
>
> I've uploaded video file and 8 cmds to fileshare:
> http://turbobit.net/7ff93nfen4pw.html
>
> ffmpeg version N-42939-g236ecc3 Copyright (c) 2000-2012 the FFmpeg
> developers
> built on Jul 26 2012 20:50:57 with gcc 4.7.1 (GCC)
>
> Windows 7 x64
> 8Gb memory
> Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz
>

_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[rogerdpack2]

> For me using one of the older builds sorted the problem out - try this
> (assuming you're using Windows):
> http://ffmpeg.zeranoe.com/builds/win64/static/ffmpeg-20120409-git-6bfb304-win64-static.7z

Interestingly, I did try it out--that one also "hangs" for me.  Were
you using it with -threads 1?  I'll run some more tests and see if I
can figure out which old versions worked well...
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[Luke]

Hello,
No - I use almost exactly the same command as before. I also managed to go
back to using AAC. There are occasional hangs - but much less frequent.
Say, once or twice per week, however such fallout is not an issue to us.
Here's the current command:

ffmpeg.exe -i #in# -f mp4 -vcodec libx264 -pix_fmt yuv420p -s 1024x576 -b:v
2000k -g 33 -acodec libvo_aacenc -b:a 128k -ar 44100 -vf "movie=#logo#
[wm]; [in][wm] overlay=10:main_h-overlay_h-10 [out]" -y -loglevel panic
#out.mp4#

I'm planning to do some testing today and tomorrow, so hopefully I'll be
able to give you some more info.

Cheers,
Luke

On 4 August 2012 06:35, Roger Pack <[hidden email]> wrote:

> > For me using one of the older builds sorted the problem out - try this
> > (assuming you're using Windows):
> >
> http://ffmpeg.zeranoe.com/builds/win64/static/ffmpeg-20120409-git-6bfb304-win64-static.7z
>
> Interestingly, I did try it out--that one also "hangs" for me.  Were
> you using it with -threads 1?  I'll run some more tests and see if I
> can figure out which old versions worked well...
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>

--
*

Luke Niejadlik
Audiomotion Studios Ltd.
www.audiomotion.com

JOIN THE MOVEMENT! - www.facebook.com/audiomotion.studios -
twitter.com/audiomotion
*
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

[rogerdpack2]

> No - I use almost exactly the same command as before. I also managed to go
> back to using AAC. There are occasional hangs - but much less frequent.
> Say, once or twice per week, however such fallout is not an issue to us.
> Here's the current command:
>
> ffmpeg.exe -i #in# -f mp4 -vcodec libx264 -pix_fmt yuv420p -s 1024x576 -b:v
> 2000k -g 33 -acodec libvo_aacenc -b:a 128k -ar 44100 -vf "movie=#logo#
> [wm]; [in][wm] overlay=10:main_h-overlay_h-10 [out]" -y -loglevel panic
> #out.mp4#


Yeah for me this command:
"C:\\downloads\\ffmpeg-20120409-git-6bfb304-win64-static\\ffmpeg-20120409-git-6bfb304-win64-static\\bin\\ffmpeg.exe
-threads 6 -y -i sintel.mpg -pass 1 -t 75 -c:v libx264 -an nul.mp4"

results in these timings:

[6.6186, 6.6536, 6.7067, 6.7275, 6.7331, 6.7605, 6.8356, 6.9484,
7.0748, 7.1302, 12.1823, 13.9774, 16.0897, 17.1871, 18.0254]

so it's failing like 1/3 times, but YMMV.

However, if I specify "-threads 1" it seems to run all right.  I'm
checking into it a bit...
-r
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user