avdevice/x11grab: fix error handling in pixel formats