Direct blocks should use motion vectors from the second reference frame