Seek succeeds but any frame read is all zeros

[gatagat]

First of all, thanks for releasing this library!

I encountered a bug in seek, that causes the subsequently read frames to be constant zero. It would be simple to just linearly loop through the frames, but there is no indication of the error. It happens only with some AVI files, however these files play/seek as expected in e.g. VLC.

I would like to have something like the following process_video function that would allow processing a selected range of the video frames. I see the error if I call it with use_seek=True.

def frame_generator(demuxer):
    for packet in demuxer:
        for frame in packet.decode():
            img = frame.to_image()
            img = np.asarray(img)
            yield img

def process_video(in_container, process_frame, start, stop, use_seek=False):
    demuxer = in_container.demux(in_container.streams.video[0])
    generator = frame_generator(demuxer)
    if start > 0:
        if use_seek is True:
            in_container.streams.video[0].seek(start, 'frame', any_frame=True)
        else:
            i = 0
            while i < start:
                generator.next()
                i += 1
    if stop is None:
        max_frames = np.inf
    else:
        max_frames = stop - start

    for i, img in enumerate(generator):
        process_frame(start + i, img)
        if i + 1 >= max_frames:
            break

Python 2.7.13
av: 0.4.1 (from pypi)

[jlaine]

Here is a simplified version of your code, which uses seek and works fine for me:

import av
import numpy as np


def process_video(container, process_frame, start, stop):
    stream = container.streams.video[0]

    if start > 0:
        stream.seek(start, 'frame', any_frame=True)

    index = start
    for packet in container.demux(stream):
        for frame in packet.decode():
            img = frame.to_image()
            img = np.asarray(img)
            process_frame(index, img)

            index += 1
            if stop is not None and index >= stop:
                return


def noop(index, img):
    print('-- frame %d --' % index)
    print(img)


container = av.open('test.mp4')
process_video(container, noop, 10, 20)

Could you please provide a sample video for which this does not work?

NOTE: it seems a bit strange to convert first to a PIL image then to a numpy array, why not use frame.to_ndarray directly?

[gatagat]

Here is a small test file that triggers the issue. The file is cut to just 3MB using dd. The output of ffmpeg -to was not causing the error anymore.

https://ufile.io/hl4bn

Use the following test to reproduce:

def noop(index, img):
    print('Frame %d: %d', (index, img.sum()))

process_video(av.open('test-trimmed.avi'), noop, 2, 4, use_seek=False)
process_video(av.open('test-trimmed.avi'), noop, 2, 4, use_seek=True)

The output without seeking:

Frame 2: 291076446
Frame 3: 305227299

The output with seeking:

Frame 2: 0
Frame 3: 0

Concerning the to_image, the video is in pal8 which to_ndarray cannot handle.

[gatagat]

I see the zero frames also with the current 6.0.0 release.

[jlaine]

Why are you purposefully damaging the AVI file? If you open the same file with VLC it complains that the index is broken, and seeking will not work correctly..

Could you provide an un-damaged file which reproduces the error please?

[mikeboers]

@gatagat Does this mean that you had a file which you weren't cutting with dd that was failing, and now it works fine (with newer PyAV)?

[gatagat]

The original file as well as the trimmed sample file both behave the same way, i.e., I get zero frames after a seek and correct frames without a seek.

I see the error in 6.0.0 as well.

[gatagat]

@jlaine Here is a different test file that is tiny enough to share in full https://ufile.io/1lacy

[mikeboers]

I have yet another version of the script in question:

import logging

import numpy as np

import av


logging.basicConfig()


def process_video(container, start, count, use_seek=False):

    global frame


    if start:
        if use_seek:
            container.seek(start, backward=True, stream=container.streams.video[0])
        else:
            for i, _ in enumerate(container.decode(video=0)):
                if i + 1 >= start:
                    break

    for packet in container.demux(video=0):
        print(packet)
        for frame in packet.decode():
            rgb_array = frame.to_ndarray(format='rgb24')
            pal_array = np.array(frame.planes[0])
            print('   ', frame)
            print('       ', rgb_array.sum(), pal_array.sum())

            if count is not None:
                count -= 1
                if count <= 0:
                    return


container = av.open('test2.avi')
process_video(container, 2, 1, use_seek=False)

container = av.open('test2.avi')
process_video(container, 2, 1, use_seek=True)

A few interesting things:

1. It doesn't matter if I seek in container time or stream time, or if it is bytes vs time mode, or backwards, or keyframes, or anything.
2. If I set count to None in the first pass, mine segfaults while getting the pal_array around frame 41.

[mikeboers]

If I am grabbing both RGB and PAL arrays, it segfaults at 41. If I'm just doing PAL, it segfaults at 85.

[mikeboers]

I can trigger it via np.array(plane), bytearray(plane), and plane.to_bytes().

[tikuma-lsuhsc]

This problem can now be solved with post-15.1 release as the solution is outlined in Issue #1987