Copyright (c) 2014 The NetBSD Foundation, Inc.
All rights reserved.
This code is derived from software contributed to The NetBSD Foundation
by Taylor R. Campbell.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
.Dd December 26, 2018 .Dt THREADPOOL 9 .Os
.Sh NAME .Nm threadpool .Nd shared pools of kthreads """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
.Sh SYNOPSIS n sys/threadpool.h """""""""""""""""""""""""""""""""""
.Vt typedef void threadpool_job_fn_t(struct threadpool_job *); """""""""""""""""""""""""""""""""""
.Ft int .Fn threadpool_get "struct threadpool **poolp" "pri_t pri"
.Ft void .Fn threadpool_put "struct threadpool *pool" "pri_t pri" """""""""""""""""""""""""""""""""""
.Ft int .Fn threadpool_percpu_get "struct threadpool_percpu **pool_percpup" "pri_t pri"
.Ft void .Fn threadpool_percpu_put "struct threadpool_percpu *pool_percpu" "pri_t pri"
.Ft struct threadpool * .Fn threadpool_percpu_ref "struct threadpool_percpu *pool"
.Ft struct threadpool * .Fn threadpool_percpu_ref_remote "struct threadpool_percpu *pool" "struct cpu_info *ci" """""""""""""""""""""""""""""""""""
.Ft void .Fn threadpool_job_init "struct threadpool_job *job" "threadpool_job_fn_t fn" "kmutex_t *interlock" "const char *fmt" "..."
.Ft void .Fn threadpool_job_destroy "struct threadpool_job *job"
.Ft void .Fn threadpool_job_done "struct threadpool_job *job" """""""""""""""""""""""""""""""""""
.Ft void .Fn threadpool_schedule_job "struct threadpool *pool" "struct threadpool_job *job"
.Ft void .Fn threadpool_cancel_job "struct threadpool *pool" "struct threadpool_job *job"
.Ft bool .Fn threadpool_cancel_job_async "struct threadpool *pool" "struct threadpool_job *job" """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
.Sh DESCRIPTION The .Nm abstraction is provided to share a pool of .Xr kthread 9 kernel threads for medium- to long-term actions, called jobs, which can be scheduled from contexts that do not allow sleeping.
p For each priority level, there is one unbound thread pool, and one collection of per-CPU thread pools. Access to the unbound thread pools is provided by .Fn threadpool_get and .Fn threadpool_put . Access to the per-CPU thread pools is provided by .Fn threadpool_percpu_get and .Fn threadpool_percpu_put .
p Job state is stored in the .Vt threadpool_job structure. Callers of the .Nm abstraction must allocate memory for .Vt threadpool_job structures, but should consider them opaque, and should not inspect or copy them. Each job represented by a .Vt threadpool_job structure will be run only once at a time, until the action associated with it calls .Fn threadpool_job_done .
p
Jobs are run in thread context and may take arbitrarily long to run or
sleep arbitrarily long.
The
.Nm
abstraction is intended as a building block for cheaper abstractions,
namely
.Xr task 9
and
.Xr workqueue 9 .
It should generally not be used directly.
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
.Sh FUNCTIONS
l -tag -width abcd """""""""""""""""""""""""""""""""""
t Fn threadpool_get "poolp" "pri" Obtain a reference to the unbound thread pool at priority
.Fa pri
and store it in
.Fa poolp .
p
May sleep.
t Fn threadpool_put "pool" "pri" Release the reference to the unbound thread pool
.Fa pool
at priority
.Fa pri ,
which must be the same as the priority that was passed to
.Fn threadpool_get
to obtain
.Fa pool .
p May sleep.
p
Do not use
.Fn threadpool_put
with thread pools obtained from
.Fn threadpool_percpu_ref
or
.Fn threadpool_percpu_ref_remote .
"""""""""""""""""""""""""""""""""""
t Fn threadpool_percpu_get "pool_percpup" "pri" Obtain a reference to the per-CPU thread pool at priority
.Fa pri
and store it in
.Fa pool_percpup .
p Use .Fn threadpool_percpu_ref or .Fn threadpool_percpu_ref_remote with it to get at the thread pool for a particular CPU.
p
May sleep.
t Fn threadpool_percpu_put "pool_percpu" "pri" Release a reference to the per-CPU thread pool
.Fa pool_percpu
at priority
.Fa pri .
p
May sleep.
t Fn threadpool_percpu_ref "pool_percpu" Return the thread pool in
.Fa pool_percpu
for the current CPU.
p The resulting thread pool pointer is stable until .Fa pool_percpu is released with .Fn threadpool_percpu_put . Using it to schedule or cancel a job does not require being on the same CPU.
p
Do not use
.Fn threadpool_put
with thread pools obtained from
.Fn threadpool_percpu_ref .
t Fn threadpool_percpu_ref_remote "pool_percpu" "ci" Return the thread pool in
.Fa pool_percpu
for the CPU whose
.Vt struct cpu_info
is given by
.Fa ci .
p The resulting thread pool pointer is stable until .Fa pool_percpu is released with .Fn threadpool_percpu_put . Using it to schedule or cancel a job does not require being on the same CPU, but it is faster and friendlier to the cache to use .Fn threadpool_percpu_ref and use the resulting thread pool only on the same CPU.
p
Do not use
.Fn threadpool_put
with thread pools obtained from
.Fn threadpool_percpu_ref_remote .
"""""""""""""""""""""""""""""""""""
t Fn threadpool_job_init "job" "fn" "interlock" "fmt" "..." Initialize the threadpool job
.Fa job
to run
.Fa fn
when scheduled and to interlock with
.Fa interlock .
The argument
.Fa fmt
is a
.Xr printf 9
format string for the job's name.
p The mutex .Fa interlock is used to synchronize job scheduling and completion. The action .Fa fn is required to eventually call .Fn threadpool_job_done , with .Fa interlock held. This is so that while the job is running and may be waiting for work to do, scheduling the job has no effect, but as soon as the job is done, scheduling the job will cause it to run again.
p
To change the action of a job, you must use
.Fn threadpool_job_destroy
first and then call
.Fn threadpool_job_init
again.
t Fn threadpool_job_destroy "job" Destroy the threadpool job
.Fa job .
.Fa job
must not currently be scheduled to run.
If it may still be scheduled, you can use
.Fn threadpool_cancel_job
to cancel it.
However,
.Fn threadpool_cancel_job_async
is not enough.
t Fn threadpool_job_done "job" Notify that
.Fa job
is done, so that subsequent calls to
.Fn threadpool_schedule_job
will cause it to re-run its action.
p
.Fn threadpool_job_done
must be called exactly once by a job's action, and may not be called in
any other context.
"""""""""""""""""""""""""""""""""""
t Fn threadpool_schedule_job "pool" "job" Schedule
.Fa job
to run in a thread in
.Fa pool
as soon as possible, creating a new thread if necessary.
p Caller must hold the interlock of .Fa job .
p
.Fn threadpool_schedule_job
may be called in any context, including hard interrupt context, except
at interrupt priority levels above
.Vt IPL_VM .
t Fn threadpool_cancel_job "pool" "job" Cancel
.Fa job
if it has been scheduled but has not yet been assigned a thread, or
wait for it to complete if it has.
p Caller must hold the interlock of .Fa job , which may be released in order to wait for completion.
p If .Fa job has not been scheduled, .Fn threadpool_cancel_job returns immediately. If .Fa job has been scheduled, it must have been scheduled in .Fa pool , not in any other thread pool.
p
May sleep.
t Fn threadpool_cancel_job_async "pool" "job" Try to cancel
.Fa job
like
.Fn threadpool_cancel_job ,
but if it is already running, return
.Vt false
instead of waiting;
otherwise, if it was not scheduled, or if it was scheduled and has not
yet begun to run, return
.Vt true .
p Caller must hold the interlock of .Fa job .
p
.Fn threadpool_cancel_job_async
may be called in any context, including hard interrupt context, except
at interrupt priority levels above
.Vt IPL_VM .
"""""""""""""""""""""""""""""""""""
.El
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
.Sh CODE REFERENCES
The
.Nm
abstraction is implemented in
a sys/kern/kern_threadpool.c .
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
.Sh SEE ALSO
.Xr kthread 9 ,
.Xr softint 9 ,
.Xr task 9 ,
.Xr workqueue 9
Indexes created Sat Sep 20 22:09:52 GMT 2025