[FFmpeg-user] ffmpeg x265 on multi-cpus

Discussion:

fred fred

2017-04-10 09:58:59 UTC

Dear list,I am currently on this equipment :
$lscpuArchitecture: x86_64CPU op-mode(s): 32-bit, 64-bitByte Order: Little EndianCPU(s): 24On-line CPU(s) list: 0-23Thread(s) per core: 2Core(s) per socket: 6Socket(s): 2NUMA node(s): 2Vendor ID: GenuineIntelCPU family: 6Model: 63Model name: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHzStepping: 2CPU MHz: 1244.585BogoMIPS: 6803.58Virtualization: VT-xL1d cache: 32KL1i cache: 32KL2 cache: 256KL3 cache: 20480KNUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23
and $free -h total used free shared buff/cache availableMem: 62G 908M 54G 969M 7.2G 60GSwap: 67G 0B 67G

and tried to use ffmpeg last compilation, according to the ffmpeg installation page
$ffmpegffmpeg version N-85424-gadf9f04 Copyright (c) 2000-2017 the FFmpeg developers built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-11) configuration: --prefix=/ffmpeg_build --extra-cflags=-I/ffmpeg_build/include --extra-ldflags='-L/ffmpeg_build/lib -ldl' --bindir=/bin --pkg-config-flags=--static --enable-gpl --enable-nonfree --enable-libfdk_aac --enable-libfreetype --enable-libmp3lame --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 libavutil 55. 60.100 / 55. 60.100 libavcodec 57. 92.100 / 57. 92.100 libavformat 57. 72.100 / 57. 72.100 libavdevice 57. 7.100 / 57. 7.100 libavfilter 6. 84.101 / 6. 84.101 libswscale 4. 7.100 / 4. 7.100 libswresample 2. 8.100 / 2. 8.100 libpostproc 54. 6.100 / 54. 6.100Hyper fast Audio and Video encoder

I have tried all possible ways to use all my 16 cpus (x2 threads) to encode with libx265, but each time it fails...
-pools, --pools, -p, --numa-* ... and so on are not recognise at all
$ffmpeg -p 16 -i Ice.avi -c:v libx265 -preset veryslow -crf 28 -c:a aac -b:a 128k Ice_x265.mp4Unrecognized option 'p'.Error splitting the argument list: Option not found
Internet and my friend Google didn't provide me with any solution...
so how to proceed ???
Thank you !RegardsFred

_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-re

Erik Slagter

2017-04-10 10:11:30 UTC

Permalink

and $free -h total used free shared buff/cache availableMem: 62G 908M 54G 969M 7.2G 60GSwap: 67G 0B 67G
and tried to use ffmpeg last compilation, according to the ffmpeg installation page
$ffmpegffmpeg version N-85424-gadf9f04 Copyright (c) 2000-2017 the FFmpeg developers built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-11) configuration: --prefix=/ffmpeg_build --extra-cflags=-I/ffmpeg_build/include --extra-ldflags='-L/ffmpeg_build/lib -ldl' --bindir=/bin --pkg-config-flags=--static --enable-gpl --enable-nonfree --enable-libfdk_aac --enable-libfreetype --enable-libmp3lame --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 libavutil 55. 60.100 / 55. 60.100 libavcodec 57. 92.100 / 57. 92.100 libavformat 57. 72.100 / 57. 72.100 libavdevice 57. 7.100 / 57. 7.100 libavfilter 6. 84.101 / 6. 84.101 libswscale 4. 7.100 / 4. 7.100 libswresample 2. 8.100 / 2. 8.100 libpostproc 54. 6.100 / 54. 6.100Hyper fast Audio and Video encoder
I have tried all possible ways to use all my 16 cpus (x2 threads) to encode with libx265, but each time it fails...
-pools, --pools, -p, --numa-* ... and so on are not recognise at all
$ffmpeg -p 16 -i Ice.avi -c:v libx265 -preset veryslow -crf 28 -c:a aac -b:a 128k Ice_x265.mp4Unrecognized option 'p'.Error splitting the argument list: Option not found
Internet and my friend Google didn't provide me with any solution...
so how to proceed ???

It's a bug in libx265 that has been resolved some time ago. I guess
you're using the libx265 from your distribution (e.g. Fedora) and
they're behind. The solution is to compile libx265 yourself from git HEAD.
_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpeg.org with subject "unsubs

Moritz Barsnick

2017-04-10 10:14:34 UTC

Permalink

Post by fred fred
I have tried all possible ways to use all my 16 cpus (x2 threads) to encode with libx265, but each time it fails...
-pools, --pools, -p, --numa-* ... and so on are not recognise at all

Those are x265 options (of the command line tool), not ffmpeg+libx265
options.

First, please try ffmpeg's own "-threads 16" option. That should do the
right thing for you.

If you need to directly use those pool options of libx265, use a special
ffmpeg option:
"-x265-params pools=8" (or something similar)

Also observe the info ffmpeg's libx265 encoder outputs when beginning

Post by fred fred
x265 [info]: Thread pool created using 4 threads
x265 [info]: frame threads / pool features : 2 / wpp(4 rows)

fred fred

2017-04-10 12:43:22 UTC

Permalink

thank you Moritz !
anyway, even with -threads 0
x265 [info]: Thread pool created using 24 threadsx265 [info]: Slices : 1x265 [info]: frame threads / pool features : 5 / wpp(5 rows)
I see with top command between 400% - 500% CPU on ffmpeg ... I was expected like 2400% or 24 ffmpeg workers with 100%
where am I wring again ??
as for my last email, I write it with yahoo email web interface, I guess it's not fully compatible with the list managing software. Sorry
RegardsFred

Post by fred fred
x265 [info]: Thread pool created using 4 threads
x265 [info]: frame threads / pool features : 2 / wpp(4 rows)

BTW, your email's formatting is terribly broken:
http://ffmpeg.org/pipermail/ffmpeg-user/2017-April/035824.html

Cheers,
Moritz
_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpeg.org with subj

Reindl Harald

2017-04-10 12:53:49 UTC

Permalink

Post by fred fred
thank you Moritz !
anyway, even with -threads 0
x265 [info]: Thread pool created using 24 threadsx265
I see with top command between 400% - 500% CPU on ffmpeg ...
I was expected like 2400% or 24 ffmpeg workers with 100%

Erik Slagter

2017-04-10 12:55:57 UTC

Permalink

Post by Reindl Harald

Post by fred fred
thank you Moritz !
anyway, even with -threads 0
x265 [info]: Thread pool created using 24 threadsx265 I see with top
command between 400% - 500% CPU on ffmpeg ... I was expected like
2400% or 24 ffmpeg workers with 100%

On the other hand, both libx264 and libx265 do scale quite well by
smartly dividing the work over all available threads. Yes, you can
really have up to 800% cpu usage on a 4x2 cpu (which is quite
impressive), with both of them.
_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpe

Reindl Harald

2017-04-10 12:58:33 UTC

Permalink

Post by Erik Slagter

Post by Reindl Harald

Post by fred fred
thank you Moritz !
anyway, even with -threads 0
x265 [info]: Thread pool created using 24 threadsx265 I see with top
command between 400% - 500% CPU on ffmpeg ... I was expected like
2400% or 24 ffmpeg workers with 100%

but you can't expect that that scales up to every core count
_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user

Erik Slagter

2017-04-10 13:02:43 UTC

Permalink

Post by Reindl Harald

Post by Erik Slagter

Post by Reindl Harald
how did you come to the conclusion that parallelization scales magically
and with no limits? if that would be realistic just throwing enough CPU
cores on whatever problem would be the soplution - but that is not how
computers are working

but you can't expect that that scales up to every core count

fred fred

2017-04-10 13:08:47 UTC

Permalink

Erik, can you teach me how to reach maximum load ? at least what is your command line use for that ?

Post by Reindl Harald

Post by Erik Slagter

but you can't expect that that scales up to every core count

But at least to more than eight cores (see the documentation), libx264
is (yet) a bit more scalable, but I guess libx265 will get better at it
as well. On a 2x2x12 core machine I had to start two concurrent
encodings to get all of the cpu's busy all of the time (libx265), but I
also must say it didn't really add that much to the total encoding frame
rate.
_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpeg.org with subject "unsub

Erik Slagter

2017-04-10 13:12:40 UTC

Permalink

Erik, can you teach me how to reach maximum load ? at least what is your command line use for that ?

Just like the documentation says, leave all threading options like they
are, don't try to outsmart libx265, it will assign threads optimally.

If you think your cores aren't used optimally, you can always start a
second transcoding session.

I find libx265 (from ffmpeg) refreshingly simple to configure for
optimal results, almost nothing needs to be tweaked, contrary to libx264.
_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-use

fred fred

2017-04-10 12:59:40 UTC

Permalink

hello Harald, it is not a conclusion, it's an expectation ! and a question : for one file from some format to x265, what it the best efficient way to use my 24 cores ? can you contribute ?BestFred

how did you come to the conclusion that parallelization scales magically
and with no limits? if that would be realistic just throwing enough CPU
cores on whatever problem would be the soplution - but that is not how
computers are working
_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-

Reindl Harald

2017-04-10 13:05:51 UTC

Permalink

Post by fred fred
hello Harald, it is not a conclusion, it's an expectation ! and a question : for one file from some format to x265, what it the best efficient way to use my 24 cores ? can you contribute ?BestFred

parallelization depends on a lot of things and you always have sharded
ressources like memory, IO, thread-synchronisation - for one video task
probably there is just no way to get 24 core to 100% CPU usage

with 3 different parallel tasks each using 8 cores probably better
because each one has it's own thread-synchronisation and so on

you just can't expect that throwing enough CPU cores on a problem will
solve it faster the same as 9 girls can't make a child in one month :-)

Post by fred fred

_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg

Erik Slagter

2017-04-10 13:08:27 UTC

Permalink

Post by Reindl Harald
you just can't expect that throwing enough CPU cores on a problem will
solve it faster the same as 9 girls can't make a child in one month :-)

Also don't forget the speed at which the storage can supply and store
the streams!
_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpeg.org with su

fred fred

2017-04-10 13:11:09 UTC

Permalink

get it, but then how to reach 800% for one task at least ?as for the girls, well, I strongly believe in practical experiments ;)

Post by fred fred

_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-***@ffmpeg.org with subject "

Carl Eugen Hoyos

2017-04-14 05:40:01 UTC

Permalink

Post by fred fred
anyway, even with -threads 0
x265 [info]: Thread pool created using 24 threadsx265 [info]: Slices : 1x265 [info]: frame threads / pool features : 5 / wpp(5 rows)
I see with top command between 400% - 500% CPU on ffmpeg ...
I was expected like 2400% or 24 ffmpeg workers with 100%

It is possible to change the x265 code so that you get something like
2400% - the disadvantage is that encoding takes longer in that case
compared to the 500% you see now;-(

You have to understand that for "real" (not intra-only) codecs
performance cannot be increased by simply adding more cpu cores.

Post by fred fred
what it the best efficient way to use my 24 cores ?

That is easy to answer: If there are no other bottlenecks (harddisk
and memory come to mind), starting 24 transcoding processes
with -threads 1 each is most efficient.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
ffmpeg-***@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-req