rtsp: Lazily allocate the pollfd array