How are ffmpeg internal frames structured?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

How are ffmpeg internal frames structured?

Mark Filipak (ffmpeg)
[decoder] --> [filters] --> [codecs] --> [encoder]

My question is:
How are docoder.output and encoder.input structured?

Yes, I know that filters can reformat the video, deinterlace, stack fields, etc.

I have been assuming that the structure is frames of pixels like this:
pixel[0,0] pixel[1,0] ... pixel[in_w-1,0] pixel[0,1] pixel[1,1] ... pixel[in_w-1,in_h-1]
In other words, raw video frames with lines abutted end-to-end (i.e. not deinterlaced).

Am I correct?

--
I don't have a dog.
And furthermore, my dog doesn't bite.
And furthermore, you provoked him.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How are ffmpeg internal frames structured?

Carl Eugen Hoyos-2
Am So., 7. Feb. 2021 um 01:34 Uhr schrieb Mark Filipak (ffmpeg)
<[hidden email]>:
>
> [decoder] --> [filters] --> [codecs] --> [encoder]

This looks wrong / misleading.
There is a transcoding pipeline:
demuxer -> decoder -> filter -> encoder -> muxer
And there are formats, codecs, filters, devices (and more) as
part of the FFmpeg project.

> My question is:
> How are decoder.output and encoder.input structured?
>
> Yes, I know that filters can reformat the video, deinterlace, stack fields, etc.

> I have been assuming that the structure is frames of pixels like this:
> pixel[0,0] pixel[1,0] ... pixel[in_w-1,0] pixel[0,1] pixel[1,1] ... pixel[in_w-1,in_h-1]

(The usual style is that rows come first, than columns.)

> In other words, raw video frames with lines abutted end-to-end

In abstract terms, this is of course correct, but note that planar and
packed pix_fmts exist, see libavutil/pixfmt.h, lines and frames are
often padded for performance reasons.

> (i.e. not deinterlaced).

I may misunderstand but I believe this part makes no sense
in above sentence.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How are ffmpeg internal frames structured?

Mark Filipak (ffmpeg)
On 02/06/2021 08:51 PM, Carl Eugen Hoyos wrote:

> Am So., 7. Feb. 2021 um 01:34 Uhr schrieb Mark Filipak (ffmpeg)
> <[hidden email]>:
>>
>> [decoder] --> [filters] --> [codecs] --> [encoder]
>
> This looks wrong / misleading.
> There is a transcoding pipeline:
> demuxer -> decoder -> filter -> encoder -> muxer
> And there are formats, codecs, filters, devices (and more) as
> part of the FFmpeg project.
>
>> My question is:
>> How are decoder.output and encoder.input structured?
>>
>> Yes, I know that filters can reformat the video, deinterlace, stack fields, etc.
>
>> I have been assuming that the structure is frames of pixels like this:
>> pixel[0,0] pixel[1,0] ... pixel[in_w-1,0]  pixel[0,1] pixel[1,1] ... pixel[in_w-1,in_h-1]
>
> (The usual style is that rows come first, than columns.)

Thanks (not x,y, eh?). Okay, then this:

pixel[0,0] pixel[0,1] ... pixel[0,in_w-1]  pixel[1,0] pixel[1,1] ... pixel[in_h-1,in_w-1]

>> In other words, raw video frames with lines abutted end-to-end
>
> In abstract terms, this is of course correct, but note that planar and
> packed pix_fmts exist, see libavutil/pixfmt.h, lines and frames are
> often padded for performance reasons.

Okay, thanks.

>> (i.e. not deinterlaced).
>
> I may misunderstand but I believe this part makes no sense
> in above sentence.
>
> Carl Eugen

"not deinterlaced" == not this:

pixel[0,0] pixel[0,1] ... pixel[0,in_w-1]  pixel[2,0] pixel[2,1] ... pixel[in_h-2,in_w-1] ...
pixel[1,0] pixel[1,1] ... pixel[1,in_w-1]  pixel[3,0] pixel[3,1] ... pixel[in_h-1,in_w-1]

and, I should add, not in macroblocks in slices.

The reason I ask is that it seems to me that some of the documentation assumes that readers already
know this.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How are ffmpeg internal frames structured?

Carl Eugen Hoyos-2
Am So., 7. Feb. 2021 um 03:27 Uhr schrieb Mark Filipak (ffmpeg)
<[hidden email]>:

> "not deinterlaced" == not this:
>
> pixel[0,0] pixel[0,1] ... pixel[0,in_w-1]  pixel[2,0] pixel[2,1] ... pixel[in_h-2,in_w-1] ...
> pixel[1,0] pixel[1,1] ... pixel[1,in_w-1]  pixel[3,0] pixel[3,1] ... pixel[in_h-1,in_w-1]

In the terminology of the FFmpeg project (which is the only one
relevant on this mailing list), above is not called "deinterlaced".

> and, I should add, not in macroblocks in slices.

I still believe that macroblocks cannot have a meaning for raw video,
same with slices.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How are ffmpeg internal frames structured?

Mark Filipak (ffmpeg)
On 02/06/2021 09:37 PM, Carl Eugen Hoyos wrote:

> Am So., 7. Feb. 2021 um 03:27 Uhr schrieb Mark Filipak (ffmpeg)
> <[hidden email]>:
>
>> "not deinterlaced" == not this:
>>
>> pixel[0,0] pixel[0,1] ... pixel[0,in_w-1]  pixel[2,0] pixel[2,1] ... pixel[in_h-2,in_w-1] ...
>> pixel[1,0] pixel[1,1] ... pixel[1,in_w-1]  pixel[3,0] pixel[3,1] ... pixel[in_h-1,in_w-1]
>
> In the terminology of the FFmpeg project (which is the only one
> relevant on this mailing list), above is not called "deinterlaced".

That's fine. I understand. If it's not called "deinterlaced", then what do I call it?

>> and, I should add, not in macroblocks in slices.
>
> I still believe that macroblocks cannot have a meaning for raw video,
> same with slices.

I agree. I'm simply being comprehensive.

--
I don't have a dog.
And furthermore, my dog doesn't bite.
And furthermore, you provoked him.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How are ffmpeg internal frames structured?

Carl Eugen Hoyos-2
Am So., 7. Feb. 2021 um 03:45 Uhr schrieb Mark Filipak (ffmpeg)
<[hidden email]>:

>
> On 02/06/2021 09:37 PM, Carl Eugen Hoyos wrote:
> > Am So., 7. Feb. 2021 um 03:27 Uhr schrieb Mark Filipak (ffmpeg)
> > <[hidden email]>:
> >
> >> "not deinterlaced" == not this:
> >>
> >> pixel[0,0] pixel[0,1] ... pixel[0,in_w-1]  pixel[2,0] pixel[2,1] ... pixel[in_h-2,in_w-1] ...
> >> pixel[1,0] pixel[1,1] ... pixel[1,in_w-1]  pixel[3,0] pixel[3,1] ... pixel[in_h-1,in_w-1]
> >
> > In the terminology of the FFmpeg project (which is the only one
> > relevant on this mailing list), above is not called "deinterlaced".
>
> That's fine. I understand. If it's not called "deinterlaced", then what do I call it?

De-interleaved as done by the il filter.

The reason FFmpeg decoders (if not buggy) always output
interleaved ("non de-interleaved") images is foremost (at least
imo) that you cannot connect a CRT to the output of an FFmpeg
decoder and that no driver - gpu - screen combination I have
seen so far could correctly display it.
I also suspect that no encoder would correctly deal with it.
There may be other reasons.
I believe it makes little sense to document that we don't do
something that would be useless, the documentation is already
very long documenting design decisions that are less obvious.

Carl Eugen

PS:
Using above definition, both FFmpeg's hevc and j2k decoders
are buggy.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: How are ffmpeg internal frames structured?

Mark Filipak (ffmpeg)
On 02/06/2021 09:58 PM, Carl Eugen Hoyos wrote:

> Am So., 7. Feb. 2021 um 03:45 Uhr schrieb Mark Filipak (ffmpeg)
> <[hidden email]>:
>>
>> On 02/06/2021 09:37 PM, Carl Eugen Hoyos wrote:
>>> Am So., 7. Feb. 2021 um 03:27 Uhr schrieb Mark Filipak (ffmpeg)
>>> <[hidden email]>:
>>>
>>>> "not deinterlaced" == not this:
>>>>
>>>> pixel[0,0] pixel[0,1] ... pixel[0,in_w-1]  pixel[2,0] pixel[2,1] ... pixel[in_h-2,in_w-1] ...
>>>> pixel[1,0] pixel[1,1] ... pixel[1,in_w-1]  pixel[3,0] pixel[3,1] ... pixel[in_h-1,in_w-1]
>>>
>>> In the terminology of the FFmpeg project (which is the only one
>>> relevant on this mailing list), above is not called "deinterlaced".
>>
>> That's fine. I understand. If it's not called "deinterlaced", then what do I call it?
>
> De-interleaved as done by the il filter.

Ah! I applaud that name. Brovo! I will not write "interlace" in the future except when I refer to
how a display device processes non-interleaved video lines (25i/30i) for the screen.

> The reason FFmpeg decoders (if not buggy) always output
> interleaved ("non de-interleaved") images is foremost (at least
> imo) that you cannot connect a CRT to the output of an FFmpeg
> decoder and that no driver - gpu - screen combination I have
> seen so far could correctly display it.

I agree.

> I also suspect that no encoder would correctly deal with it.
> There may be other reasons.
> I believe it makes little sense to document that we don't do
> something that would be useless, the documentation is already
> very long documenting design decisions that are less obvious.
>
> Carl Eugen
>
> PS:
> Using above definition, both FFmpeg's hevc and j2k decoders
> are buggy.

Thank you for being patient, Carl Eugen.

I apologize for any confusion. I'm not proposing that ffmpeg document what it doesn't do and I agree
with that. However, I've not found a description of what ffmpeg does... until you have confirmed it,
here. Thank you. Planar and packed pix_fmts is probably more info than what is needed to convey the
concept of ffmpeg's transcoding pipeline and of the general line format in the pipeline.

Long Life!
Mark.

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".