How to make sort using multiple threads to run faster?

sort supports --parallel N to run N thread. However, it seems it only uses around 100% CPU as I observed although the command specified that N threads can be used.

The command is as follows

cat large-file | sort --parallel `nproc`

where I have 16 from nproc.

How to make sort use multiple threads to run faster?

asked Apr 27, 2017 by anonymous

1 Answer

It is possibly caused by the pipe |: data are sent out as a stream. You can possibly make parallel take more effect by setting its buffer for sort like

cat large-file | sort --parallel=`nproc` -S 20G
-S, --buffer-size=SIZE
    use SIZE for main memory buffer

as from sort manual

answered Jun 21, 2017 by Eric Z Ma (44,280 points)

Please log in or register to answer this question.

Copyright © SysTutorials. User contributions licensed under cc-wiki with attribution required.
Hosted on Dreamhost