avdevice/xcbgrab: check if frame size fits in INT_MAX