Ensure that we don't read from invalid memory with B-frame qpel