[FFmpeg-user] Low latency H.264 streaming with ffmpeg

[Greger Burman]

I would like to hear from anyone who has experience with low latency video
streaming and ffmpeg.

In my application I take video from a PCIe capture device and stream live
over network. The end to end latency must be below 200ms otherwise it is
not usable. Let us assume that the network latency is near zero. This
question is about encoder+decoder+buffering latency.

Very low latency can be achieved when using the x264 encoder by using
settings like tune zerolatency. From my own testing with x264 (without
ffmpeg) I can confirm that this works. I have measured approx 90 ms (3
frames) of latency added by encoder+decoder combined. Awesome.

Sadly I have been unable to get anywhere near those numbers when using
ffmpeg and libx264. As player I use either ffplay or a dot-net player built
on the ffmpeg libraries, with about equal results. The lowest end to end
latency I have seen is 400-500ms. That includes capture latency also.

Yes, there are many unknowns here. To help pin point the problem I used
mjpeg streaming for comparison. Same everything except I change ffmpeg
vcodec to mjpeg and format to mjpeg. I change nothing with ffplay. With
mjpeg I now get approx 150ms end to end latency.

Observation:
x264 codec + mpegts format compared to mjpeg codec + mjpeg format adds
about 300 ms (10 frames) latency.

I have tried every ffplay-, ffmpeg- and libx264 setting that I believe
relevant. Most have not made any noticable difference in latency. I will
not list a lot of settings here. Here are the basic settings and commands
that I have used. I can post more details on request.

x264-preset:
vcodec=libx264
thread_type=slice
slices=1
# x264
profile=baseline
level=32
preset=superfast
tune=zerolatency
intra-refresh=1
crf=15
x264-params=vbv-maxrate=5000:vbv-bufsize=1:slice-max-size=1500:keyint=60

$ ffmpeg -r 30 -f dshow -i video="devicename" -pix_fmt yuv420p -an -vpre
x264-preset -f mpegts udp://127.0.0.1:8888

$ ffplay -fflags nobuffer -flags low_delay -framedrop -strict experimental
udp://127.0.0.1:8888

Everything is run on Windows 7 64-bit, with 32-bit executables.
ffmpeg version N-76539-g480bad7 Copyright (c) 2000-2015 the FFmpeg
developers
built with gcc 5.2.0 (GCC)
configuration: --enable-gpl --enable-version3 --disable-w32threads
--enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r
--enable-gnutls --enable-iconv --enable-libass --enable-libbluray
--enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype
--enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug
--enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb
--enable-libopenjpeg --enable-libopus --enable-librtmp
--enable-libschroedinger --enable-libsoxr --enable-libspeex
--enable-libtheora --enable-libtwolame --enable-libvidstab
--enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis
--enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264
--enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg
--enable-lzma --enable-decklink --enable-zlib
libavutil 55. 5.100 / 55. 5.100
libavcodec 57. 15.100 / 57. 15.100
libavformat 57. 14.100 / 57. 14.100
libavdevice 57. 0.100 / 57. 0.100
libavfilter 6. 15.100 / 6. 15.100
libswscale 4. 0.100 / 4. 0.100
libswresample 2. 0.101 / 2. 0.101
libpostproc 54. 0.100 / 54. 0.100

cheers

[Roger Pack]

Post by Greger Burman
I would like to hear from anyone who has experience with low latency video
streaming and ffmpeg.
In my application I take video from a PCIe capture device and stream live
over network. The end to end latency must be below 200ms otherwise it is
not usable. Let us assume that the network latency is near zero. This
question is about encoder+decoder+buffering latency.
Very low latency can be achieved when using the x264 encoder by using
settings like tune zerolatency. From my own testing with x264 (without
ffmpeg) I can confirm that this works. I have measured approx 90 ms (3
frames) of latency added by encoder+decoder combined. Awesome.
Sadly I have been unable to get anywhere near those numbers when using
ffmpeg and libx264. As player I use either ffplay or a dot-net player built
on the ffmpeg libraries, with about equal results. The lowest end to end
latency I have seen is 400-500ms. That includes capture latency also.

mplayer -benchmark has been the only client I've found with reasonable
latency, though I'm sure there are others/more.
I've gotten pretty low latency before with ultrafast+zerolatency

Basically my advice is "be wary of ffplay (and vlc)" I'm not sure if
its latency is low enough...

https://trac.ffmpeg.org/wiki/StreamingGuide

[Kenneth Fields]

yes, we’re having the same issue now.
but -benchmark also really degrades the stream.
Is there any options possible for -benchmark (moderate, strict)?

Ken

Post by Roger Pack
mplayer -benchmark has been the only client I've found with reasonable
latency, though I'm sure there are others/more.
I've gotten pretty low latency before with ultrafast+zerolatency
Basically my advice is "be wary of ffplay (and vlc)" I'm not sure if
its latency is low enough...

Post by Greger Burman
I would like to hear from anyone who has experience with low latency video
streaming and ffmpeg.
In my application I take video from a PCIe capture device and stream live
over network. The end to end latency must be below 200ms otherwise it is
not usable. Let us assume that the network latency is near zero. This
question is about encoder+decoder+buffering latency.
Very low latency can be achieved when using the x264 encoder by using
settings like tune zerolatency. From my own testing with x264 (without
ffmpeg) I can confirm that this works. I have measured approx 90 ms (3
frames) of latency added by encoder+decoder combined. Awesome.
Sadly I have been unable to get anywhere near those numbers when using
ffmpeg and libx264. As player I use either ffplay or a dot-net player built
on the ffmpeg libraries, with about equal results. The lowest end to end
latency I have seen is 400-500ms. That includes capture latency also.

https://trac.ffmpeg.org/wiki/StreamingGuide <https://trac.ffmpeg.org/wiki/StreamingGuide>
_______________________________________________
ffmpeg-user mailing list
http://ffmpeg.org/mailman/listinfo/ffmpeg-user <http://ffmpeg.org/mailman/listinfo/ffmpeg-user>

[Roger Pack]

Post by Kenneth Fields
yes, we’re having the same issue now.
but -benchmark also really degrades the stream.
Is there any options possible for -benchmark (moderate, strict)?

what type of degredation? are you using udp? are you including audio?

[Greger Burman]

Ok, so the ffmpeg based player is buffering frames somewhere even though
I'm telling it not to. This happens with h264 video but not with mjpeg.
That suggests to me that the latency is caused by either the h264 decoder
or the mpegts demuxer. I have also tried m4v instead of mpegts, with about
the same result.

Anyone who knows enough about the h264 decoding process to confirm/deny?
Is there a way to affect this behaviour without diving deep into the code?

cheers
//Greger

[Roger Pack]

Post by Greger Burman
Ok, so the ffmpeg based player is buffering frames somewhere even though
I'm telling it not to. This happens with h264 video but not with mjpeg.
That suggests to me that the latency is caused by either the h264 decoder
or the mpegts demuxer. I have also tried m4v instead of mpegts, with about
the same result.
Anyone who knows enough about the h264 decoding process to confirm/deny?
Is there a way to affect this behaviour without diving deep into the code?

I don't know much about the h264 decoder. If you put I-frames in more
often it might help. I've heard rumors of this flag
-fflags nobuffer
but really no little of any internal workings.

[rameshperumal]

Post by Greger Burman
x264 codec + mpegts format compared to mjpeg codec + mjpeg format adds
about 300 ms (10 frames) latency.

Hi,

I just started to transcode the 1-minute input video(mjpeg) from the USB
camera with the below command.

time ~/bin/ffmpeg -benchmark -threads 4 -t 00:01:00.000000 -f video4linux2
-input_format mjpeg -framerate 30 -i /dev/video0 -an -c:v libx264 -preset
ultrafast -tune zerolatency -thread_type 1 -r:v 30 -f null -

Output:
ffmpeg version 3.3 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 6.2.0 (Ubuntu 6.2.0-5ubuntu12) 20161005
configuration: --prefix=/usr --pkg-config-flags=--static
--extra-cflags=-I/usr/include --extra-ldflags=-L/usr/lib --bindir=/root/bin
--enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype
--enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264
--enable-libx265 --enable-nonfree --enable-vaapi --enable-libopus
--enable-libvpx --enable-hwaccel=h263_vaapi --enable-hwaccel=h264_vaapi
--enable-hwaccel=mpeg2_vaapi --enable-hwaccel=mpeg4_vaapi
--enable-hwaccel=vc1_vaapi --enable-hwaccel=vp8_vaapi
--enable-hwaccel=vp9_vaapi --enable-hwaccel=wmv3_vaapi
--enable-decoder='vp8,vp9'
libavutil 55. 58.100 / 55. 58.100
libavcodec 57. 89.100 / 57. 89.100
libavformat 57. 71.100 / 57. 71.100
libavdevice 57. 6.100 / 57. 6.100
libavfilter 6. 82.100 / 6. 82.100
libswscale 4. 6.100 / 4. 6.100
libswresample 2. 7.100 / 2. 7.100
libpostproc 54. 5.100 / 54. 5.100
Input #0, video4linux2,v4l2, from '/dev/video0':
Duration: N/A, start: 5864.558330, bitrate: N/A
Stream #0:0: Video: mjpeg, yuvj422p(pc, bt470bg/unknown/unknown),
640x480, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
Stream mapping:
Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
Press [q] to stop, [?] for help
No pixel format specified, yuvj422p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
[libx264 @ 0x555b636ea220] using cpu capabilities: MMX2 SSE2Fast SSSE3
SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x555b636ea220] profile High 4:2:2, level 3.0, 4:2:2 8-bit
Output #0, null, to 'pipe:':
Metadata:
encoder : Lavf57.71.100
Stream #0:0: Video: h264 (libx264), yuvj422p(pc), 640x480, q=-1--1, 30
fps, 30 tbn, 30 tbc
Metadata:
encoder : Lavc57.89.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame= 1800 fps= 30 q=-1.0 Lsize=N/A time=00:01:00.00 bitrate=N/A speed=
1x
video:8981kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB
muxing overhead: unknown
bench: utime=5.746s
bench: maxrss=57952kB
[libx264 @ 0x555b636ea220] frame I:8 Avg QP:19.12 size: 62426
[libx264 @ 0x555b636ea220] frame P:1792 Avg QP:22.09 size: 4853
[libx264 @ 0x555b636ea220] mb I I16..4: 100.0% 0.0% 0.0%
[libx264 @ 0x555b636ea220] mb P I16..4: 1.5% 0.0% 0.0% P16..4: 58.3%
0.0% 0.0% 0.0% 0.0% skip:40.2%
[libx264 @ 0x555b636ea220] coded y,uvDC,uvAC intra: 29.4% 66.3% 13.1% inter:
14.5% 42.2% 8.9%
[libx264 @ 0x555b636ea220] i16 v,h,dc,p: 32% 38% 18% 12%
[libx264 @ 0x555b636ea220] i8c dc,h,v,p: 36% 23% 31% 10%
[libx264 @ 0x555b636ea220] kb/s:1226.19

real 1m1.359s
user 0m5.772s
sys 0m0.115s


May I know the values (bench: utime=5.746s or user 0m5.772s or sys
0m0.115s) I need to consider to measure the end-to-end transcoding latency?
I am interested in measuring the combined encode and decode latency per
video frame. Please suggest me.



--
Sent from: http://www.ffmpeg-archive.org/
_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpeg.o