How to make sort using multiple threads to run faster?

sort supports --parallel N to run N thread. However, it seems it only uses around 100% CPU as I observed although the command specified that N threads can be used.

The command is as follows

cat large-file | sort --parallel `nproc`

where I have 16 from nproc.

How to make sort use multiple threads to run faster?

asked Apr 28 by anonymous

1 Answer

It is possibly caused by the pipe |: data are sent out as a stream. You can possibly make parallel take more effect by setting its buffer for sort like

cat large-file | sort --parallel=`nproc` -S 20G
-S, --buffer-size=SIZE
    use SIZE for main memory buffer

as from sort manual https://www.systutorials.com/docs/linux/man/1-sort/

answered Jun 21 by Eric Z Ma (42,870 points)

Please log in or register to answer this question.

Copyright © SysTutorials. User contributions licensed under cc-wiki with attribution required.
Hosted on Dreamhost

...