Using ThreadJob for Performance

Parallelizing work in PowerShell has been a drag. Runspaces are too complicated to spin up and background jobs are too resource intensive. There is a parallel foreach parameter when using workflows but I’ve never been able to figure workflows out. I’ve always used Invoke-Parallel and PoshRSJob to accomplish most speed related tasks but now there is a new module in town. The PowerShell team has just released to the PowerShell Gallery a new module called ThreadJob to handle spinning up multiple, lightweight jobs in PowerShell. Background jobs themselves tend to take a lot of CPU cycles to initialize and can cause great strains on memory. Consider the following code:

Get-Job | Remove-Job -Force

Measure-Command { 1..10 | ForEach-Object {
        Start-Job -ArgumentList $_ -ScriptBlock {
            Param($Arg) 

            $arg % 2
        }
    }
    Get-Job | Receive-Job -Wait
}

First, remove all previous jobs running in this session. Then wrap everything in a measure-command script block to test the speed. Create 10 integers and throw them into the pipeline. From there, start a job where the job performs the modulus operation on an argument. Simple stuff. If you open task manager jobs spin up their own PowerShell instance to complete the work.

Threads, on the other hand, do not open multiple processes. The ThreadJob module leverages the process that spawned them. The code is exactly the same, just substituting Start-Job with Start-ThreadJob. Once the ThreadJob starts you interact with it like any other job.

Get-Job | Remove-Job -Force
Measure-Command { 1..10 | ForEach-Object {
        Start-ThreadJob -ArgumentList $_ -ScriptBlock {
            Param($Arg) 

            $arg % 2
        }
    }
Get-Job | Receive-Job -Wait

The results are incredible. Even with such a small sample test, ThreadJob took 1/10th the time that background jobs use. Give 100 jobs a try. My computer couldn’t handle 100 background jobs in a reasonable amount of time.

Currently, there are a few drawbacks to using a ThreadJob over a background job. If a background job hangs, only that process hangs. All other jobs keep chugging away. If you have a job that hangs with ThreadJob the entire queue is affected. The team also hasn’t implemented the $Using variable so make sure you structure your parameter blocks to give your scripts all the parameters it needs.

Example code can be found at the following gist.