avfilter/vf_datascope: fix oscilloscope with gray>8 depth