Out of virtual memory (swap, I assume)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Out of virtual memory (swap, I assume)

Mark Filipak (ffmpeg)
Windows 10-1803
Memory: 32GB
Virtual memory: 128GB on an SSD-RAID0.

No other applications running.

ffmpeg -i "THE LAST EMPEROR [1987] source part5.mkv" -map 0 -filter_complex
"minterpolate=fps=48000/1001:mi_mode=mci:mc_mode=obmc:scd=fdiff:scd_threshold=10:vsbmc=1:search_param=32,
telecine=pattern=3322,pp=linblenddeint, setpts=N*1001/60000/TB" -codec:v libx265 -x265-params
"crf=20:qcomp=0.60" -codec:a copy -codec:s copy "THE LAST EMPEROR [1987] part5.mkv"

As the transcode proceeds, I watch the virtual memory commit climb steadily.
As the transcode proceeds, I watch the memory allocation top out and then hover there.
I went to bed, but I assume that when the virtual memory commit tops out, ffmpeg reports the
following error:
"Error while filtering: Cannot allocate memory
"Failed to inject frame into filter network: Cannot allocate memory
"Error while processing the decoded data for stream #0:0
"Conversion failed!"

There's something about 'source part5.mkv' that causes the fault, but I'll be damned if I can find it.

I tried the part 5 transcode 3 times:
1, While concurrently transcoding parts 1-4 (which all succeeded), and
2, On its own, and finally
3, I again extracted part 5 via MKVToolNIX, and transcoded again.
That last time I made sure that 'source part5.mkv' played correctly (with correct running times in
MPV) and with monotonically increasing timestamps before transcoding for the 3rd time.

I took 4 pertinent files, renamed them, zipped them as part5.zip   ...6,138,036,350 bytes.
Inside part5.zip is:
[part5.zip]\source part5.mkv   ...6,150,347,006 bytes
[part5.zip]\source part5.mkv timestamps.txt   ...5,095,698 bytes
[part5.zip]\part5.mkv failure #3.log   ...3,372,428 bytes
[part5.zip]\part5.mkv timestamps.txt   ...3,114,801 bytes

*** Well, dropbox seems to be malfunctioning. Can you suggest an alternative? ***

Any/all ideas regarding how I can successfully transcode part 5 will be much appreciated even if
it/they fail.

Thanks and Good Health. Regards,
Mark.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Paul B Mahol
minterpolate filter may leak memory, i havent tried to run it under
valgrind because i have only 4GB of RAM, and slow CPU.

On Tue, Feb 2, 2021 at 7:22 PM Mark Filipak (ffmpeg) <[hidden email]>
wrote:

> Windows 10-1803
> Memory: 32GB
> Virtual memory: 128GB on an SSD-RAID0.
>
> No other applications running.
>
> ffmpeg -i "THE LAST EMPEROR [1987] source part5.mkv" -map 0
> -filter_complex
> "minterpolate=fps=48000/1001:mi_mode=mci:mc_mode=obmc:scd=fdiff:scd_threshold=10:vsbmc=1:search_param=32,
>
> telecine=pattern=3322,pp=linblenddeint, setpts=N*1001/60000/TB" -codec:v
> libx265 -x265-params
> "crf=20:qcomp=0.60" -codec:a copy -codec:s copy "THE LAST EMPEROR [1987]
> part5.mkv"
>
> As the transcode proceeds, I watch the virtual memory commit climb
> steadily.
> As the transcode proceeds, I watch the memory allocation top out and then
> hover there.
> I went to bed, but I assume that when the virtual memory commit tops out,
> ffmpeg reports the
> following error:
> "Error while filtering: Cannot allocate memory
> "Failed to inject frame into filter network: Cannot allocate memory
> "Error while processing the decoded data for stream #0:0
> "Conversion failed!"
>
> There's something about 'source part5.mkv' that causes the fault, but I'll
> be damned if I can find it.
>
> I tried the part 5 transcode 3 times:
> 1, While concurrently transcoding parts 1-4 (which all succeeded), and
> 2, On its own, and finally
> 3, I again extracted part 5 via MKVToolNIX, and transcoded again.
> That last time I made sure that 'source part5.mkv' played correctly (with
> correct running times in
> MPV) and with monotonically increasing timestamps before transcoding for
> the 3rd time.
>
> I took 4 pertinent files, renamed them, zipped them as part5.zip
>  ...6,138,036,350 bytes.
> Inside part5.zip is:
> [part5.zip]\source part5.mkv   ...6,150,347,006 bytes
> [part5.zip]\source part5.mkv timestamps.txt   ...5,095,698 bytes
> [part5.zip]\part5.mkv failure #3.log   ...3,372,428 bytes
> [part5.zip]\part5.mkv timestamps.txt   ...3,114,801 bytes
>
> *** Well, dropbox seems to be malfunctioning. Can you suggest an
> alternative? ***
>
> Any/all ideas regarding how I can successfully transcode part 5 will be
> much appreciated even if
> it/they fail.
>
> Thanks and Good Health. Regards,
> Mark.
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Mark Filipak (ffmpeg)
On 02/02/2021 01:38 PM, Paul B Mahol wrote:
> minterpolate filter may leak memory, i havent tried to run it under
> valgrind because i have only 4GB of RAM, and slow CPU.

Fair enough. Introduce me to valgrind and I'll see if I can help.

Disclosure: I don't do 'C'. I don't know how to compile in Windows -- The instructions I've found
on-line were ambiguous or required Internet access within Windows (which I do not allow). I've tried
cross-compiling in a Linux VM, but that failed. I gave up.

--
Someone's sneaking in and turning up the flame so that my food burns.
I'm sure of it.
And the older I get, the more sure of it I become.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Mark Filipak (ffmpeg)
In reply to this post by Paul B Mahol
On 02/02/2021 01:38 PM, Paul B Mahol wrote:
> minterpolate filter may leak memory, i havent tried to run it under
> valgrind because i have only 4GB of RAM, and slow CPU.

I have a bag of various memory modules. Do you want it?

Also, I have a spare Lenovo laptop that I think has quite a bit of memory (surely 16MB or more).
Would you like it?

Free. All free to you.

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Mark Filipak (ffmpeg)
In reply to this post by Paul B Mahol
Correction: "16GB" was "16MB".

On 02/02/2021 01:38 PM, Paul B Mahol wrote:
> minterpolate filter may leak memory, i havent tried to run it under
> valgrind because i have only 4GB of RAM, and slow CPU.

I have a bag of various memory modules. Do you want it?

Also, I have a spare Lenovo laptop that I think has quite a bit of memory (surely 16GB or more).
Would you like it?

Free. All free to you.

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Carl Eugen Hoyos-2
In reply to this post by Paul B Mahol
Am Di., 2. Feb. 2021 um 19:38 Uhr schrieb Paul B Mahol <[hidden email]>:
>
> minterpolate filter may leak memory, i havent tried to run it under
> valgrind because i have only 4GB of RAM, and slow CPU.

valgrind shows no leak, afair minterpolate only allocates its
buffers once at initialization.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Paul B Mahol
On Tue, Feb 2, 2021 at 11:20 PM Carl Eugen Hoyos <[hidden email]> wrote:

> Am Di., 2. Feb. 2021 um 19:38 Uhr schrieb Paul B Mahol <[hidden email]>:
> >
> > minterpolate filter may leak memory, i havent tried to run it under
> > valgrind because i have only 4GB of RAM, and slow CPU.
>
> valgrind shows no leak, afair minterpolate only allocates its
> buffers once at initialization.
>

It may still leak frames.


> Carl Eugen
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Mark Filipak (ffmpeg)
In reply to this post by Carl Eugen Hoyos-2
On 02/02/2021 05:20 PM, Carl Eugen Hoyos wrote:
> Am Di., 2. Feb. 2021 um 19:38 Uhr schrieb Paul B Mahol <[hidden email]>:
>>
>> minterpolate filter may leak memory, i havent tried to run it under
>> valgrind because i have only 4GB of RAM, and slow CPU.
>
> valgrind shows no leak, afair minterpolate only allocates its
> buffers once at initialization.
>
> Carl Eugen

If that's true, what's eating the swap? Transcoding parts 1, 2, 3, & 4 all succeeded when running
concurrently. Transcoding part 5 fails, even when ffmpeg is the only app running.

Dropbox apparently overcame its problem. My upload will finish in a few minutes. I'll post the link
to my package when it has completed.

--
I don't have a dog.
And furthermore, my dog doesn't bite.
And furthermore, you provoked him.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Mark Filipak (ffmpeg)
On 02/02/2021 05:52 PM, Mark Filipak (ffmpeg) wrote:
-snip-
> Dropbox apparently overcame its problem. My upload will finish in a few minutes. I'll post the link
> to my package when it has completed.

The upload finished, but dropbox won't let me transfer it, so no ticket, no link. Oh, dear.

--
I don't have a dog.
And furthermore, my dog doesn't bite.
And furthermore, you provoked him.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Carl Zwanzig
In reply to this post by Mark Filipak (ffmpeg)
On 2/2/2021 2:52 PM, Mark Filipak (ffmpeg) wrote:
> If that's true, what's eating the swap? Transcoding parts 1, 2, 3, & 4 all
> succeeded when running concurrently. Transcoding part 5 fails, even when
> ffmpeg is the only app running.

Out of curiosity, have you tried looking at the file with an mkv analyzer
like MediaInfo, Elecard's analyzer* or maybe another one? (Is there metadata
towards the end of the file containing junk info?)

How does it behave if you drop the minterpolate or one of the other filters,
output to a null file, or use a different output codec? (Anything to try
isolating the problem.)

*https://www.elecard.com/products/video-analysis/video-format-analyzer

Later,

z!
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Mark Filipak (ffmpeg)
On 02/03/2021 12:23 AM, Carl Zwanzig wrote:
> On 2/2/2021 2:52 PM, Mark Filipak (ffmpeg) wrote:
>> If that's true, what's eating the swap? Transcoding parts 1, 2, 3, & 4 all succeeded when running
>> concurrently. Transcoding part 5 fails, even when ffmpeg is the only app running.
>
> Out of curiosity, have you tried looking at the file with an mkv analyzer like MediaInfo, Elecard's
> analyzer* or maybe another one? (Is there metadata towards the end of the file containing junk info?)
>
> How does it behave if you drop the minterpolate or one of the other filters, output to a null file,
> or use a different output codec? (Anything to try isolating the problem.)

I'm doing as you suggest, but it takes so long I'll probably be up all night -- it's 2:30AM now.

Do you know of any tools that would help troubleshoot the problem? I'm running in Win10 using
Sysinternals Process Explorer, but it's not really a profiler.

> *https://www.elecard.com/products/video-analysis/video-format-analyzer

I would buy a good tool -- I actually prefer purchased over FOSS -- but their page is not very
informative or welcoming.

--
I don't have a dog.
And furthermore, my dog doesn't bite.
And furthermore, you provoked him.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume)

Mark Filipak (ffmpeg)
Here is the original command that fails:

ffmpeg -i "source part5.mkv" -map 0 -filter_complex
"minterpolate=fps=48000/1001:mi_mode=mci:mc_mode=obmc:scd=fdiff:scd_threshold=10:vsbmc=1:search_param=32,
telecine=pattern=3322,pp=linblenddeint, setpts=N*1001/60000/TB" -codec:v libx265 -x265-params
crf=20:qcomp=0.60 -codec:a copy -codec:s copy "part5.mkv"

Here is the test command for this run:

ffmpeg -i "source part5 (44561 frames).mkv" -map 0 -filter_complex
"telecine=pattern=3322,pp=linblenddeint, setpts=N*1001/30000/TB" -codec:v libx265 -x265-params
crf=20:qcomp=0.60 -codec:a copy -codec:s copy "part5 without minterpolate.mkv"

Test results:

Memory (of 32GB): 13.2GB to 25.4GB in 25 minutes, 31GB 15 minutes later, then hover between 31GB &
31.6GB thereafter -- ramp up then drop, ramp up then drop: I imagine that's normal malloc-free
cycles. But why would ffmpeg defer free'ing until it had filled memory?

Swap commit (of 128GB): 14.2GB to 27.3 in 25 minutes, 33GB 15 minutes later, 46GB 45 minites later
-- rising steadily... Aha! Then leveling off at 46.3GB. But why would swap commits begin immediately
instead of when memory has been filled?

A great many more "Starting new cluster due to timestamp", a continuous stream -- coming so fast &
furious that I can't read what the current frame number is before "Starting new cluster due to
timestamp" overwrites it.

Ah! The "Starting new cluster due to timestamp" stream has ended, and it ended at the same time that
swap commit leveled off.

Is that a clue?

The run just completed at 3:07AM. It started at 1:37AM, so took 1 hr. 40 min. That contrasts with
the original run (that included minterpolate) which ran all night before dying.

You know what I think? From the behavior of memory & swap commits, it looks to me that it's a
combination of "Starting new cluster due to timestamp" persisting for so many hours longer during
the original run.

Actually, there were 3 original runs because I tried it 3 times; all died.

=====

For what it's worth, here's my notes made while planning the source video cutting:

Divide transcoding the 222545 input frames between 5 concurrent tasks.
(222545-0)/5 = 44509 input frames per task

Define the bounds of the 5 segments.
0-44509, 44510-89018, 89019-133527, 133528-178036, 178037-222545

Adjust the segments so that each starts with the nearest key frame having an even frame number.
0-44503, 44504-88999, 89000-133447, 133448-177983, 177984-222545

Add 1 because MKVToolNIX expects frame numbers to start from 1, not zero.
1-44504, 44505-89000, 89001-133448, 133449-177984, 177985-222546

minterpolate silently drops the final 2 input frames, so add 2 frame overlaps to compensate.
1-44506, 44505-89002, 89001-133450, 133449-177986, 177985-222548

Via MKVToolNix, segment the input per the compensated bounds.
source-part1.mkv source-part2.mkv source-part3.mkv source-part4.mkv source-part5.mkv

Create 5 scripts and run them concurrently.
source part1.mkv.cmd
   ffmpeg -i source-part1.mkv -map 0 -filter_complex
"minterpolate=fps=48000/1001:mi_mode=mci:vsbmc=1, telecine=pattern=3322,pp=linblenddeint,
setpts=N*1001/60000/TB" -codec:v libx265 -x265-params crf=20:qcomp=0.60 -codec:a copy -codec:s copy
part1.mkv
   pause
source part2.mkv.cmd
   ffmpeg -i source-part2.mkv ... part2.mkv
   pause
source part3.mkv.cmd
   ffmpeg -i source-part3.mkv ... part3.mkv
   pause
source part4.mkv.cmd
   ffmpeg -i source-part4.mkv ... part4.mkv
   pause
source part5.mkv.cmd
   ffmpeg -i source-part5.mkv ... part5.mkv
   pause

Hope that the resulting motion vectors at segment joins will be close enough that they're not too
apparent.

--
I don't have a dog.
And furthermore, my dog doesn't bite.
And furthermore, you provoked him.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume) = telecine filter

Mark Filipak (ffmpeg)
In reply to this post by Carl Zwanzig
On 02/03/2021 12:23 AM, Carl Zwanzig wrote:
> On 2/2/2021 2:52 PM, Mark Filipak (ffmpeg) wrote:
>> If that's true, what's eating the swap? Transcoding parts 1, 2, 3, & 4 all succeeded when running
>> concurrently. Transcoding part 5 fails, even when ffmpeg is the only app running.
>
> Out of curiosity, have you tried looking at the file with an mkv analyzer like MediaInfo, Elecard's
> analyzer* or maybe another one? (Is there metadata towards the end of the file containing junk info?)
>
> How does it behave if you drop the minterpolate or one of the other filters, output to a null file,
> or use a different output codec? (Anything to try isolating the problem.)

If rising memory allocation and rising swap commit indicates a memory leak, then the telecine filter
has a memory leak.

=====
part5.cmd
-----
ffmpeg -i "THE LAST EMPEROR [1987] source part5.mkv" -map 0 -filter_complex
"minterpolate=fps=48000/1001:mi_mode=mci:mc_mode=obmc:scd=fdiff:scd_threshold=10:vsbmc=1:search_param=32,
telecine=pattern=3322,pp=linblenddeint, setpts=N*1001/60000/TB" -codec:v libx265 -x265-params
crf=20:qcomp=0.60 -codec:a copy -codec:s copy "THE LAST EMPEROR [1987] part5.mkv"
pause
exit

Results:
Though 'part1.cmd', 'part2.cmd', 'part3.cmd', and 'part4.cmd' completed when running concurrently,
'part5.cmd' died.
I then twice ran 'part5.cmd' in isolation and it died twice.
=====
part5 without minterpolate.cmd
-----
ffmpeg -i "THE LAST EMPEROR [1987] source part5 (44561 frames).mkv" -map 0 -filter_complex
"telecine=pattern=3322,pp=linblenddeint, setpts=N*1001/30000/TB" -codec:v libx265 -x265-params
crf=20:qcomp=0.60 -codec:a copy -codec:s copy "THE LAST EMPEROR [1987] part5 without minterpolate.mkv"
pause
exit

Results:

Expecting 55700 output frames.

1:07AM start.
Memory (of 32GB): 13.2GB
Swap commit (of 128GB): 14.2GB

1:32AM
Memory (of 32GB): 25.4GB
Swap commit (of 128GB): 27.3GB
A great number of "Starting new cluster due to timestamp" notices, a continuous stream -- coming so
fast & furious that I can't read what the current frame number is before "Starting new cluster due
to timestamp" overwrites it.

1:47AM
Memory (of 32GB): 31GB
Swap commit (of 128GB): 33GB

2:32AM
Memory (of 32GB): Hovering between 31GB & 31.6GB -- ramp up then drop, ramp up then drop: I imagine
that's normal malloc-free cycles. But why would ffmpeg defer free'ing until it had filled memory?
Swap commit (of 128GB): 46GB, rising steadily... Aha! Then leveling off at 46.3GB. But why would
swap commits begin immediately instead of when memory has been filled?

Ah! The "Starting new cluster due to timestamp" stream has ended, and it ended at the same time that
swap commit leveled off.

Is that a clue?

3:07AM done.
This contrasts with the original run (that included minterpolate) which ran all night, then died.

You know what I think? From the behavior of memory & swap commits, it looks to me that it's a
combination of "Starting new cluster due to timestamp" persisting for so many hours longer during
the original run.

Actually, there were 3 original runs because I tried it 3 times; all died.
=====
part5 with solely minterpolate.cmd
-----
ffmpeg -i "THE LAST EMPEROR [1987] part5.mkv" -map 0 -filter_complex
"minterpolate=fps=48000/1001:mi_mode=mci:mc_mode=obmc:scd=fdiff:scd_threshold=10:vsbmc=1:search_param=32,
setpts=N*1001/48000/TB" -codec:v libx265 -x265-params crf=20:qcomp=0.60 -codec:a copy -codec:s copy
"THE LAST EMPEROR [1987] part5 with solely minterpolate.mkv"
pause
exit

Results:

Expecting 89120 output frames.

2:14PM start.
Memory (of 32GB):= 10.3GB
Swap commit (of 128GB): 10.6GB

Thereafter, both memory and swap commit are flat.
Note: I let it run awhile after "Starting new cluster due to timestamp" began appearing.
There were much fewer "Starting new cluster due to timestamp" notices.

2:40PM
Memory = 10.6GB
Swap commit = 10.9GB

3:37 stopped.
=====
part5 with solely telecine.cmd
-----
ffmpeg -i "THE LAST EMPEROR [1987] source part5 (44561 frames).mkv" -map 0 -filter_complex
"telecine=pattern=3322, setpts=N*1001/30000/TB" -codec:v libx265 -x265-params crf=20:qcomp=0.60
-codec:a copy -codec:s copy "THE LAST EMPEROR [1987] part5 with solely telecine.mkv"
pause
exit

Results:

3:43PM start.
Memory (of 32GB): 9.9GB
Swap commit (of 128GB): 10.2GB

3:47PM
Memory (of 32GB): 11.4GB
Swap commit (of 128GB): 11.9GB

3:52PM begin getting "Starting new cluster due to timestamp" notices.
Memory (of 32GB): 13.2
Swap commit (of 128GB): 13.8GB

4:00
Memory (of 32GB): 17
Swap commit (of 128GB): 17.5GB

4:00 stopped.
=====

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume) = telecine+setpts combination

Mark Filipak (ffmpeg)
In reply to this post by Carl Zwanzig
On 02/03/2021 12:23 AM, Carl Zwanzig wrote:
> On 2/2/2021 2:52 PM, Mark Filipak (ffmpeg) wrote:
>> If that's true, what's eating the swap? Transcoding parts 1, 2, 3, & 4 all succeeded when running
>> concurrently. Transcoding part 5 fails, even when ffmpeg is the only app running.
>
> Out of curiosity, have you tried looking at the file with an mkv analyzer like MediaInfo, Elecard's
> analyzer* or maybe another one? (Is there metadata towards the end of the file containing junk info?)
>
> How does it behave if you drop the minterpolate or one of the other filters, output to a null file,
> or use a different output codec? (Anything to try isolating the problem.)

If rising memory allocation and rising swap commit indicates a memory leak, then the telecine filter
has a memory leak.

**WRONG**

OHMYGOD! The problem isn't telecine. The problem isn't setpts. The problem is telecine + setpts in
combination!

**CORRECTION**
If rising memory allocation and rising swap commit indicates a memory leak, then the combination of
telecine and setpts filters provokes a memory leak.

My trouble packet is 'telecine+setpts memory leak.zip'. I can't put it in dropbox because dropbox
has now added a 100 MB limit in an attempt to force me to pay.

'telecine+setpts memory leak.zip' is 328 MiB.

With no way to get 'telecine+setpts memory leak.zip' into the developers' hands, I guess all my
efforts will go to naught.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume) = telecine+setpts combination

Paul B Mahol
Well, perhaps your command queues too much frame and that is bad.

On Thu, Feb 4, 2021 at 10:34 PM Mark Filipak (ffmpeg) <[hidden email]>
wrote:

> On 02/03/2021 12:23 AM, Carl Zwanzig wrote:
> > On 2/2/2021 2:52 PM, Mark Filipak (ffmpeg) wrote:
> >> If that's true, what's eating the swap? Transcoding parts 1, 2, 3, & 4
> all succeeded when running
> >> concurrently. Transcoding part 5 fails, even when ffmpeg is the only
> app running.
> >
> > Out of curiosity, have you tried looking at the file with an mkv
> analyzer like MediaInfo, Elecard's
> > analyzer* or maybe another one? (Is there metadata towards the end of
> the file containing junk info?)
> >
> > How does it behave if you drop the minterpolate or one of the other
> filters, output to a null file,
> > or use a different output codec? (Anything to try isolating the problem.)
>
> If rising memory allocation and rising swap commit indicates a memory
> leak, then the telecine filter
> has a memory leak.
>
> **WRONG**
>
> OHMYGOD! The problem isn't telecine. The problem isn't setpts. The problem
> is telecine + setpts in
> combination!
>
> **CORRECTION**
> If rising memory allocation and rising swap commit indicates a memory
> leak, then the combination of
> telecine and setpts filters provokes a memory leak.
>
> My trouble packet is 'telecine+setpts memory leak.zip'. I can't put it in
> dropbox because dropbox
> has now added a 100 MB limit in an attempt to force me to pay.
>
> 'telecine+setpts memory leak.zip' is 328 MiB.
>
> With no way to get 'telecine+setpts memory leak.zip' into the developers'
> hands, I guess all my
> efforts will go to naught.
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume) = telecine+setpts combination

Mark Filipak (ffmpeg)
On 02/04/2021 04:50 PM, Paul B Mahol wrote:
> Well, perhaps your command queues too much frame and that is bad.

Perhaps?

ffmpeg -i source.mkv -map 0 -filter_complex "telecine, setpts=N*1001/30000/TB" -codec:v libx265
-x265-params crf=20:qcomp=0.60 -codec:a copy -codec:s copy "with both telecine and setpts.mkv"

What does "...perhaps your command queues too much frame" mean?


> On Thu, Feb 4, 2021 at 10:34 PM Mark Filipak (ffmpeg) <[hidden email]>
> wrote:
>
>> On 02/03/2021 12:23 AM, Carl Zwanzig wrote:
>>> On 2/2/2021 2:52 PM, Mark Filipak (ffmpeg) wrote:
>>>> If that's true, what's eating the swap? Transcoding parts 1, 2, 3, & 4
>> all succeeded when running
>>>> concurrently. Transcoding part 5 fails, even when ffmpeg is the only
>> app running.
>>>
>>> Out of curiosity, have you tried looking at the file with an mkv
>> analyzer like MediaInfo, Elecard's
>>> analyzer* or maybe another one? (Is there metadata towards the end of
>> the file containing junk info?)
>>>
>>> How does it behave if you drop the minterpolate or one of the other
>> filters, output to a null file,
>>> or use a different output codec? (Anything to try isolating the problem.)
>>
>> If rising memory allocation and rising swap commit indicates a memory
>> leak, then the telecine filter
>> has a memory leak.
>>
>> **WRONG**
>>
>> OHMYGOD! The problem isn't telecine. The problem isn't setpts. The problem
>> is telecine + setpts in
>> combination!
>>
>> **CORRECTION**
>> If rising memory allocation and rising swap commit indicates a memory
>> leak, then the combination of
>> telecine and setpts filters provokes a memory leak.
>>
>> My trouble packet is 'telecine+setpts memory leak.zip'. I can't put it in
>> dropbox because dropbox
>> has now added a 100 MB limit in an attempt to force me to pay.
>>
>> 'telecine+setpts memory leak.zip' is 328 MiB.
>>
>> With no way to get 'telecine+setpts memory leak.zip' into the developers'
>> hands, I guess all my
>> efforts will go to naught.
>> _______________________________________________
>> ffmpeg-user mailing list
>> [hidden email]
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>>
>> To unsubscribe, visit link above, or email
>> [hidden email] with subject "unsubscribe".
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
>


--
I don't have a dog.
And furthermore, my dog doesn't bite.
And furthermore, you provoked him.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Out of virtual memory (swap, I assume) = telecine+setpts combination

Paul B Mahol
On Thu, Feb 4, 2021 at 11:02 PM Mark Filipak (ffmpeg) <[hidden email]>
wrote:

> On 02/04/2021 04:50 PM, Paul B Mahol wrote:
> > Well, perhaps your command queues too much frame and that is bad.
>
> Perhaps?
>
>
Perhaps not. I just checked telecine+setpts with your arguments and it
leaks 0 bytes.


> ffmpeg -i source.mkv -map 0 -filter_complex "telecine,
> setpts=N*1001/30000/TB" -codec:v libx265
> -x265-params crf=20:qcomp=0.60 -codec:a copy -codec:s copy "with both
> telecine and setpts.mkv"
>
> What does "...perhaps your command queues too much frame" mean?
>
>
> > On Thu, Feb 4, 2021 at 10:34 PM Mark Filipak (ffmpeg) <
> [hidden email]>
> > wrote:
> >
> >> On 02/03/2021 12:23 AM, Carl Zwanzig wrote:
> >>> On 2/2/2021 2:52 PM, Mark Filipak (ffmpeg) wrote:
> >>>> If that's true, what's eating the swap? Transcoding parts 1, 2, 3, & 4
> >> all succeeded when running
> >>>> concurrently. Transcoding part 5 fails, even when ffmpeg is the only
> >> app running.
> >>>
> >>> Out of curiosity, have you tried looking at the file with an mkv
> >> analyzer like MediaInfo, Elecard's
> >>> analyzer* or maybe another one? (Is there metadata towards the end of
> >> the file containing junk info?)
> >>>
> >>> How does it behave if you drop the minterpolate or one of the other
> >> filters, output to a null file,
> >>> or use a different output codec? (Anything to try isolating the
> problem.)
> >>
> >> If rising memory allocation and rising swap commit indicates a memory
> >> leak, then the telecine filter
> >> has a memory leak.
> >>
> >> **WRONG**
> >>
> >> OHMYGOD! The problem isn't telecine. The problem isn't setpts. The
> problem
> >> is telecine + setpts in
> >> combination!
> >>
> >> **CORRECTION**
> >> If rising memory allocation and rising swap commit indicates a memory
> >> leak, then the combination of
> >> telecine and setpts filters provokes a memory leak.
> >>
> >> My trouble packet is 'telecine+setpts memory leak.zip'. I can't put it
> in
> >> dropbox because dropbox
> >> has now added a 100 MB limit in an attempt to force me to pay.
> >>
> >> 'telecine+setpts memory leak.zip' is 328 MiB.
> >>
> >> With no way to get 'telecine+setpts memory leak.zip' into the
> developers'
> >> hands, I guess all my
> >> efforts will go to naught.
> >> _______________________________________________
> >> ffmpeg-user mailing list
> >> [hidden email]
> >> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
> >>
> >> To unsubscribe, visit link above, or email
> >> [hidden email] with subject "unsubscribe".
> > _______________________________________________
> > ffmpeg-user mailing list
> > [hidden email]
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-user
> >
> > To unsubscribe, visit link above, or email
> > [hidden email] with subject "unsubscribe".
> >
>
>
> --
> I don't have a dog.
> And furthermore, my dog doesn't bite.
> And furthermore, you provoked him.
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".