Home | History | Annotate | Download | only in kern
History log of /src/sys/kern/kern_malloc.c
RevisionDateAuthorComments
 1.158  14-Nov-2019  maxv Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
- "shad", to track uninitialized memory with a bit granularity (1:1).
Each bit set to 1 in the shad corresponds to one uninitialized bit of
real kernel memory.
- "orig", to track the origin of the memory with a 4-byte granularity
(1:1). Each uint32_t cell in the orig indicates the origin of the
associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
- a code designating the type of memory (Stack, Pool, etc), and
- a compressed pointer, which points either (1) to a string containing
the name of the variable associated with the cell, or (2) to an area
in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.
 1.157  07-Apr-2019  maxv Provide a code argument in kasan_mark(), and give a code to each caller.
Five codes used: GenericRedZone, MallocRedZone, KmemRedZone, PoolRedZone,
and PoolUseAfterFree.

This can greatly help debugging complex memory corruptions.
 1.156  07-Mar-2019  maxv Mmh, fix len, mh_size includes the malloc header, but we don't redzone it.
 1.155  23-Dec-2018  maxv Simplify the KASAN API, use only kasan_mark() and explain briefly. The
alloc/free naming was too confusing.
 1.154  20-Oct-2018  martin Do not assume size_t == unsigned long
 1.153  22-Aug-2018  christos - opt_kasan.h is included from <sys/asan.h>
- now that we are not using inlines, we need one more ifdef.
 1.152  22-Aug-2018  maxv Add back the KASAN ifdefs in kern_malloc until we sort out the type issue,
and fix sys/asan.h. Tested on i386, amd64 and amd64-kasan.
 1.151  22-Aug-2018  maxv Reduce the number of KASAN ifdefs, suggested by Christos/Taylor.
 1.150  21-Aug-2018  maxv Need to keep track of the requested size, when realloc is used under kASan.
Maybe we could use mh_rqsz by default.
 1.149  21-Aug-2018  pgoyette Conditionalize inclusion of kasan.h so that rump can build.
 1.148  20-Aug-2018  maxv Add support for kASan on amd64. Written by me, with some parts inspired
from Siddharth Muralee's initial work. This feature can detect several
kinds of memory bugs, and it's an excellent feature.

It can be enabled by uncommenting these three lines in GENERIC:

#makeoptions KASAN=1 # Kernel Address Sanitizer
#options KASAN
#no options SVS

The kernel is compiled without SVS, without DMAP and without PCPU area.
A shadow area is created at boot time, and it can cover the upper 128TB
of the address space. This area is populated gradually as we allocate
memory. With this design the memory consumption is kept at its lowest
level.

The compiler calls the __asan_* functions each time a memory access is
done. We verify whether this access is legal by looking at the shadow
area.

We declare our own special memcpy/memset/etc functions, because the
compiler's builtins don't add the __asan_* instrumentation.

Initially all the mappings are marked as valid. During dynamic
allocations, we add a redzone, which we mark as invalid. Any access on
it will trigger a kASan error message. Additionally, the compiler adds
a redzone on global variables, and we mark these redzones as invalid too.
The illegal-access detection works with a 1-byte granularity.

For now, we cover three areas:

- global variables
- kmem_alloc-ated areas
- malloc-ated areas

More will come, but that's a good start.
 1.147  20-Aug-2018  maxv Compute the pointer earlier, not in the return statement. No functional
change.
 1.146  28-Jul-2017  martin branches: 1.146.2; 1.146.4;
Avoid integer overflow in kern_malloc(). Reported by Ilja Van Sprundel.
XXX Time to kill malloc() completely!
 1.145  06-Feb-2015  maxv branches: 1.145.10;
Don't include <uvm/uvm_extern.h>
 1.144  06-Feb-2015  maxv Kill kmeminit().
 1.143  19-May-2014  rmind branches: 1.143.2; 1.143.4;
- Split off PRU_ATTACH and PRU_DETACH logic into separate functions.
- Replace malloc with kmem and eliminate M_PCB while here.
- Sprinkle more asserts.
 1.142  30-Apr-2012  rmind branches: 1.142.2; 1.142.4; 1.142.12;
- Replace some malloc(9) uses with kmem(9).
- G/C M_IPMOPTS, M_IPMADDR and M_BWMETER.
 1.141  29-Apr-2012  dsl Remove everything to do with 'struct malloc_type' and the malloc link_set.
To make code in 'external' (etc) still compile, MALLOC_DECLARE() still
has to generate something of type 'struct malloc_type *', with
normal optimisation gcc generates a compile-time 0.
MALLOC_DEFINE() and friends have no effect.
Fix one or two places where the code would no longer compile.
 1.140  29-Apr-2012  dsl Remove the unused 'struct malloc_type' args to kern_malloc/realloc/free
The M_xxx arg is left on the calls to malloc() and free(),
maybe they could be converted to an enumeration and just saved in
the malloc header (for deep diag use).
Remove the malloc_type from mbuf extension.
Fixes rump build as well.
Welcome to 6.99.6
 1.139  28-Apr-2012  rmind Remove MALLOC_DEBUG and MALLOCLOG, which is dead code after malloc(9) move
to kmem(9). Note: kmem(9) has debugging facilities under DEBUG/DIAGNOSTIC.
However, expensive kmguard and debug_freecheck have to be enabled manually.
 1.138  06-Feb-2012  drochner branches: 1.138.2; 1.138.6; 1.138.8;
align allocations >=pagesize at a page boundary, to preserve traditional
malloc(9) semantics
fixes dri mappings shared per mmap (at least on i945)
approved by releng
 1.137  30-Jan-2012  mrg make sure that the 'struct malloc' header on allocations is properly
aligned to (ALIGNBYTES+1). this ensures that the memory that malloc(9)
returns is correctly aligned for the platform. this change has an
effect on hppa, ia64, sparc and sparc64.

necessary on sparc for 8-byte load/store instructions. with this my
SS20 boots multiuser again.
 1.136  30-Jan-2012  rmind - kern_realloc: fix a recent regression, use correct size of current allocation.
- kern_malloc: constify.
 1.135  28-Jan-2012  rmind - Instead of kmem_cache_max, calculate max index and avoid a shift.
- Use __read_mostly and __cacheline_aligned.
- Make kmem_{intr_alloc,free} public.
- Misc.
 1.134  27-Jan-2012  para extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged
 1.133  15-Oct-2011  christos branches: 1.133.2; 1.133.6;
make buckets u_int. Fixes diagnostic panic when we allocate more than 65K
active elements in a bucket.
 1.132  31-Aug-2011  plunky NULL does not need a cast
 1.131  05-May-2010  christos give the name of the dup initialization.
 1.130  05-Apr-2010  he Follow christos' suggestions, and make ks_active a u_short, and
also only use 16 u_shorts instead of 32 ints. Also add panic()
calls for under- and overflow of the ks_active members under
DIAGNOSTIC. The MAXBUCKET constant ended up in sys/mallocvar.h
and not sys/param.h, as the latter caused build problems.

Ride the kernel revision bump of my previous change.
 1.129  05-Apr-2010  he Extend struct malloc_type to count the number of active allocations
per size, and make vmstat report this information under the "Memory
statistics by type" display, which is only printed when the kernel
has been compiled with KMEMSTATS defined, like this:

Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
wapbl 15 4192K 4192K 78644K 376426 0 0 32:0,256:3,512:6,131072:1,262144:2,524288:3

Since struct malloc_type is user-visible and is changed, bump kernel
revision to 5.99.26.

While it is true that malloc(9) is in general on the path of slowly
being replaced by kmem(9) (kmem_alloc/kmem_free), there remains a
lot of points of usage of malloc/free, and this could aid in finding
any leaks. (It helped finding the leak fixed in PR#42661.)

This was discussed with and somewhat hestitantly OKed by rmind@
 1.128  22-Jan-2010  hubertf branches: 1.128.2; 1.128.4;
Let kernel build when MALLOCLOG is defined but DIAGNOSTIC is not.
Else, hitmlog() is defined but not used, which triggers a warning.
 1.127  13-Sep-2009  pooka Wipe out the last vestiges of POOL_INIT with one swift stroke. In
most cases, use a proper constructor. For proplib, give a local
equivalent of POOL_INIT for the kernel object implementation. This
way the code structure can be preserved, and a local link set is
not hazardous anyway (unless proplib is split to several modules,
but that'll be the day).

tested by booting a kernel in qemu and compile-testing i386/ALL
 1.126  07-Jan-2009  pooka Missed realloc in previous. Ride 5.99.6 bump.
 1.125  05-Jan-2009  pooka Rename malloc() to kern_malloc() to avoid name conflict with libc.
Now it is possible to use the kernel linker with rump.
 1.124  21-Dec-2008  cegger undo turn malloc flags to enums.
Requested by christos.

Keep malloc flag defines in kernel section.
 1.123  19-Dec-2008  cegger Turn malloc flags into an enum.
While here, move the flags definition into _KERNEL section.
 1.122  07-Dec-2008  cegger make this compile with MALLOC_DEBUG but w/o DEBUG
 1.121  26-Oct-2008  blymn branches: 1.121.2; 1.121.4; 1.121.10;
Extend the information that MALLOCLOG outputs to include the preceding
buffer. This can be helpful in the case where a buffer overrun in the
preceding region stomps on the following regions data.
 1.120  25-Sep-2008  pooka Move global malloc types from kern_malloc into a separate module.
 1.119  17-Mar-2008  ad branches: 1.119.4; 1.119.6; 1.119.10;
Make them compile again.
 1.118  17-Mar-2008  yamt - simplify ASSERT_SLEEPABLE.
- move it from proc.h to systm.h.
- add some more checks.
- make it a little more lkm friendly.
 1.117  03-Jan-2008  yamt branches: 1.117.2; 1.117.6;
malloc: fix freelist corruption.
 1.116  05-Dec-2007  ad branches: 1.116.2; 1.116.4;
Match the docs: MUTEX_DRIVER/SPIN are now only for porting code written
for Solaris.
 1.115  26-Nov-2007  yamt branches: 1.115.2;
free: fix size passed to LOCKDEBUG_MEM_CHECK.
 1.114  11-Nov-2007  matt Change some initialization of static queues to compile time.
(xxx_INIT to xxx_HEAD_INITIALIZER). Drop code which inits
non-auto (global or static) variables to 0 since that's
already implied by being non-auto. Init some static/global
cpu_simple_locks at compile time.
 1.113  06-Nov-2007  ad Merge from vmlocking.
 1.112  11-Oct-2007  ad branches: 1.112.2; 1.112.4;
Merge from vmlocking:

- G/C spinlockmgr() and simple_lock debugging.
- Always include the kernel_lock functions, for LKMs.
- Slightly improved subr_lockdebug code.
- Keep sizeof(struct lock) the same if LOCKDEBUG.
 1.111  19-Apr-2007  yamt branches: 1.111.6; 1.111.8; 1.111.10;
malloc: fix a deadlock.
 1.110  12-Mar-2007  ad branches: 1.110.2;
Pass an ipl argument to pool_init/POOL_INIT to be used when initializing
the pool's lock.
 1.109  12-Mar-2007  ad Use mutexes.
 1.108  04-Mar-2007  christos branches: 1.108.2;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.
 1.107  22-Feb-2007  thorpej TRUE -> true, FALSE -> false
 1.106  09-Feb-2007  ad branches: 1.106.2;
Merge newlock2 to head.
 1.105  01-Nov-2006  yamt branches: 1.105.2; 1.105.4;
remove some __unused from function parameters.
 1.104  12-Oct-2006  christos - sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386
 1.103  02-Oct-2006  chs remove details of the kernel malloc() implementation from header files:
- change MALLOC() and FREE() to just call their function equivalents.
- remove references to other malloc()-related constants.
 1.102  21-Jul-2006  yamt branches: 1.102.4; 1.102.6;
use ASSERT_SLEEPABLE where appropriate.
 1.101  11-Dec-2005  christos branches: 1.101.4; 1.101.8;
merge ktrace-lwp.
 1.100  05-Aug-2005  jmmv Fix wording in a comment.
 1.99  30-May-2005  chs branches: 1.99.2;
rename "kbucket" to "kmembuckets", for greater clarity.
don't forget to update vmstat this time.
 1.98  30-May-2005  christos finish bucket -> kbucket change
 1.97  01-Apr-2005  yamt merge yamt-km branch.
- don't use managed mappings/backing objects for wired memory allocations.
save some resources like pv_entry. also fix (most of) PR/27030.
- simplify kernel memory management API.
- simplify pmap bootstrap of some ports.
- some related cleanups.
 1.96  26-Feb-2005  perry nuke trailing whitespace
 1.95  14-Jan-2005  christos branches: 1.95.2; 1.95.4;
Cast nkmempages to vsize_t before shifting to avoid overflow. Requested
by soda.
XXX: should be pulled up to 2.0.
 1.94  13-Jan-2005  yamt kmeminit_nkmempages: don't limit the size of kmem_map to physmem/4
because, while there's little benefit to do so, it easily causes
"out of space in kmem_map" panic on machines with small memory.
 1.93  01-Jan-2005  yamt in the case of !PMAP_MAP_POOLPAGE, gather pool backend allocations to
large chunks for kernel_map and kmem_map to ease kva fragmentation.
 1.92  01-Jan-2005  yamt introduce vm_map_kernel, a subclass of vm_map, and
move some kernel-only members of vm_map to it.
 1.91  01-Jan-2005  simonb Spell "available" correctly.
 1.90  04-Sep-2004  manu IPv4 PIM support, based on a submission from Pavlin Radoslavov posted on
tech-net@
 1.89  30-Oct-2003  simonb Remove some assigned-to but otherwise unused variables.
 1.88  27-Sep-2003  mycroft A nit I ran into while copying this code for something else...
 1.87  23-Sep-2003  thorpej Avoid a type punning warning in the MALLOC_DEBUG case.
 1.86  03-Sep-2003  ragge Do correct pointer casts.
 1.85  31-Aug-2003  fvdl Fix error in previous commit: the first vaddr_t argument to uvm_km_suballoc
is in/out, so it must be zeroed out before passing it, otherwise the
uvm_km_suballoc call would get stack garbage, and fail randomly.
 1.84  30-Aug-2003  ragge Do not cast a pointer to a pointer with different basic type when given
as function argument. It makes assumptions about pointer internals
that do not necessarily have to be true.
 1.83  28-Aug-2003  enami Use vm_map_{min,max}() rather than accessing struct vm_map.header.{start,end}
directly.
 1.82  26-Aug-2003  manu Added a malloc freelist sanity check function, for debugging purposes
 1.81  07-Aug-2003  agc Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.
 1.80  02-Aug-2003  manu Build again with MALLOCLOG
 1.79  06-May-2003  fvdl branches: 1.79.2;
In the KMEMSTATS && DIAGNOSTIC case, add a check for inuse already being 0
at free() time. This will at least catch an alloc/free mismatch early in
boot, rather than having it hang hard because of a wrapped ks_limit.
 1.78  14-Feb-2003  pk Make the memory allocation code MP-safe.
 1.77  01-Feb-2003  thorpej Add extensible malloc types, adapted from FreeBSD. This turns
malloc types into a structure, a pointer to which is passed around,
instead of an int constant. Allow the limit to be adjusted when the
malloc type is defined, or with a function call, as suggested by
Jonathan Stone.
 1.76  10-Nov-2002  thorpej Avoid signed/unsigned comparison and strict alias warnings.
 1.75  27-Sep-2002  provos remove trailing \n in panic(). approved perry.
 1.74  25-Sep-2002  thorpej Don't include <sys/map.h>.
 1.73  15-Sep-2002  chs add a new km flag UVM_KMF_CANFAIL, which causes uvm_km_kmemalloc() to
return failure if swap is full and there are no free physical pages.
have malloc() use this flag if M_CANFAIL is passed to it.
use M_CANFAIL to allow amap_extend() to fail when memory is scarce.
this should prevent most of the remaining hangs in low-memory situations.
 1.72  25-Aug-2002  thorpej Fix a signed/unsigned comparison warning from GCC 3.3.
 1.71  03-Apr-2002  fvdl branches: 1.71.2; 1.71.4;
Include opt_malloc_debug.h so that debug_malloc will actually be used
when enabled, thankyouverymuch.
 1.70  05-Dec-2001  enami branches: 1.70.4;
Introduce new function malloc_roundup(), suggested by Bill Sommerfeld
on tech-kern.
 1.69  04-Dec-2001  enami KNF and other cosmetic changes.
 1.68  04-Dec-2001  jdolecek Add M_CANFAIL malloc(9) flag. This flag changes behaviour in M_WAITOK
case when the requested memory size can't ever be granted - instead
of panic, malloc(9) would return failure (NULL).
Note kernel code should do proper bound checking, rather than
depend on M_CANFAIL. This flag is only supposed to be used in very
special cases, where common bound checking is not appropriate.

Discussed on tech-kern@, name ``M_CANFAIL'' suggested by Chuck Cranor.
 1.67  30-Nov-2001  enami Since we know that `va' is non-NULL here, no need to test if it is NULL.
 1.66  21-Nov-2001  enami Use round_page() instead of roundup(, PAGE_SIZE).
Suggested by chs@netbsd.org.
 1.65  17-Nov-2001  lukem Add new malloc(9) flag M_ZERO - zeros memory before returning.
From Poul-Henning Kamp's equivalent enhancement in FreeBSD.
 1.64  12-Nov-2001  lukem add RCSIDs
 1.63  15-Sep-2001  chs branches: 1.63.2;
a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.
 1.62  17-Aug-2001  thorpej branches: 1.62.2;
Artur Grabowski's simple brute-force malloc debugger, which uses
guard pages. Can only debug one malloc type at a time, and nothing
larger than 1 page. But can be useful for debugging certain types
of "data modified on freelist" type problems.

Modified from code in OpenBSD.
 1.61  26-Jun-2001  thorpej branches: 1.61.2;
Reduce some complexity in the fault path -- Rather than maintaining
an spl-protected "interrupt safe map" list, simply require that callers
of uvm_fault() never call us in interrupt context (MD code must make
the assertion), and check for interrupt-safe maps in uvmfault_lookup()
before we lock the map.
 1.60  26-Jun-2001  thorpej Make the kmemstats limits unsigned, and use unsigned arithmetic
to compute them. Prevents overflows when using a very large number
of kmem pages.

From chopps@netbsd.org.
 1.59  05-Jun-2001  thorpej Assert that no locks are held if we're called with M_WAITOK.
 1.58  02-Jun-2001  chs replace vm_map{,_entry}_t with struct vm_map{,_entry} *.
 1.57  18-Jan-2001  jdolecek branches: 1.57.2;
constify
 1.56  14-Jan-2001  thorpej Use splvm(), not the home-grown splmem().
 1.55  24-Nov-2000  chs ifdef DEBUG, increase the max size we'll overwrite in freed memory blocks
to detect modifications after frees from 32 bytes to 1 page.
 1.54  27-Jun-2000  mrg remove include of <vm/vm.h>
 1.53  26-Jun-2000  mrg remove/move more mach vm header files:

<vm/pglist.h> -> <uvm/uvm_pglist.h>
<vm/vm_inherit.h> -> <uvm/uvm_inherit.h>
<vm/vm_kern.h> -> into <uvm/uvm_extern.h>
<vm/vm_object.h> -> nothing
<vm/vm_pager.h> -> into <uvm/uvm_pager.h>

also includes a bunch of <vm/vm_page.h> include removals (due to redudancy
with <vm/vm.h>), and a scattering of other similar headers.
 1.52  26-May-2000  sommerfeld Improve #error message slightly.
 1.51  08-May-2000  thorpej __predict_false() the check for bogus malloc type, running out of space
in kmem_map, and the DIAGNOSTIC error checks.
 1.50  30-Mar-2000  augustss Get rid of register declarations.
 1.49  11-Feb-2000  thorpej Add some very simple code to auto-size the kmem_map. We take the
amount of physical memory, divide it by 4, and then allow machine
dependent code to place upper and lower bounds on the size. Export
the computed value to userspace via the new "vm.nkmempages" sysctl.

NKMEMCLUSTERS is now deprecated and will generate an error if you
attempt to use it. The new option, should you choose to use it,
is called NKMEMPAGES, and two new options NKMEMPAGES_MIN and
NKMEMPAGES_MAX allow the user to configure the bounds in the kernel
config file.
 1.48  01-Feb-2000  thorpej Add a sanity check to ensure that a region being free()'d is actually
within the KVA range malloc() would return.
 1.47  03-Dec-1999  ragge First round of discarding the CL* macros.
 1.46  15-Nov-1999  fvdl Add Kirk McKusick's soft updates code to the trunk. Not enabled by
default, as the copyright on the main file (ffs_softdep.c) is such
that is has been put into gnusrc. options SOFTDEP will pull this
in. This code also contains the trickle syncer.

Bump version number to 1.4O
 1.45  19-Jul-1999  chs branches: 1.45.2; 1.45.4; 1.45.8;
remove simplelockrecurse, it's no longer needed.
 1.44  04-Jun-1999  thorpej Keep interrupt-safe maps on an additional queue. In uvm_fault(), if we're
looking up a kernel address, check to see if the address is on this
"interrupt-safe" list. If so, return failure immediately. This prevents
a locking screw if a page fault is taken on an interrupt-safe map in or
out of interrupt context.
 1.43  28-May-1999  thorpej Don't use a read lock on an intrsafe map; these maps can only use exclusive,
non-sleeping locks.
 1.42  26-May-1999  thorpej Change the vm_map's "entries_pageable" member to a r/o flags member, which
has PAGEABLE and INTRSAFE flags. PAGEABLE now really means "pageable",
not "allocate vm_map_entry's from non-static pool", so update all map
creations to reflect that. INTRSAFE maps are maps that are used in
interrupt context (e.g. kmem_map, mb_map), and thus use the static
map entry pool (XXX as does kernel_map, for now). This will eventually
change now these maps are locked, as well.
 1.41  24-Mar-1999  mrg branches: 1.41.4;
completely remove Mach VM support. all that is left is the all the
header files as UVM still uses (most of) these.
 1.40  22-Jan-1999  chs avoid math on void *.
 1.39  02-Dec-1998  thorpej Add dump_kmemstats(), callable from DDB to show kmem statistics. From
Chuck Silvers via Chuck Cranor.
 1.38  04-Nov-1998  chs LOCKDEBUG enhancements for non-MP:
keep a list of locked locks.
use this to print where the lock was locked
when we either go to sleep with a lock held
or try to free a locked lock.
 1.37  12-Sep-1998  christos Make copyrights consistent; fix weird/trailing spaces add missing (c) etc.
 1.36  18-Aug-1998  thorpej Add some braces to make egcs happy (ambiguous else warning).
 1.35  13-Aug-1998  eeh Merge paddr_t changes into the main branch.
 1.34  04-Aug-1998  perry Abolition of bcopy, ovbcopy, bcmp, and bzero, phase one.
bcopy(x, y, z) -> memcpy(y, x, z)
ovbcopy(x, y, z) -> memmove(y, x, z)
bcmp(x, y, z) -> memcmp(x, y, z)
bzero(x, y) -> memset(x, 0, y)
 1.33  20-May-1998  thorpej branches: 1.33.2;
defopt LOCKDEBUG
 1.32  01-Mar-1998  fvdl Merge with Lite2 + local changes
 1.31  10-Feb-1998  mrg - add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.
 1.30  08-Feb-1998  thorpej Allow callers of uvm_km_suballoc() to specify where the base of the
submap _must_ begin, by adding a "fixed" boolean argument.
 1.29  07-Feb-1998  chs fix a locking problem by replacing the call to uvm_kernacc()
with an inlined version that uses kmem_map instead of kernel_map.
 1.28  05-Feb-1998  mrg initial import of the new virtual memory system, UVM, into -current.

UVM was written by chuck cranor <chuck@maria.wustl.edu>, with some
minor portions derived from the old Mach code. i provided some help
getting swap and paging working, and other bug fixes/ideas. chuck
silvers <chuq@chuq.com> also provided some other fixes.

this is the rest of the MI portion changes.

this will be KNF'd shortly. :-)
 1.27  21-Jan-1998  thorpej Implement an activity log for malloc() and free(), useful for tracking
down "Data modified on freelist" and "muliple free" problems.

The log is activated by the MALLOCLOG option, and the size of the
event ring buffer is controlable via the MALLOGLOGSIZE option (default
is 100000 entries).

From Chris Demetriou, cleaned up a little by me per suggestions in the
e-mail from Chris that contained the code.
 1.26  09-Oct-1997  mycroft Add another bit of const poisoning.
 1.25  09-Oct-1997  mycroft Make memname[] const (but only shallow, since the array may be modified).
 1.24  02-Feb-1997  thorpej branches: 1.24.8;
KMEMSTATS is a header option; pull in "opt_kmemstats.h".
 1.23  30-Jan-1997  tls add #ifdef KMEMSTATS around variable otherwise unused; now compiles with KMEMSTATS turned off.
 1.22  13-Oct-1996  christos backout previous kprintf change
 1.21  10-Oct-1996  christos printf -> kprintf, sprintf -> ksprintf
 1.20  27-Aug-1996  cgd implement realloc(), just like the user-land version, except it takes
"type" and "flags" arguments a la kernel malloc().
 1.19  13-Aug-1996  thorpej Inside some DIAGNOSTIC code, cast some pointer arithmetic results
to long to quiet the compiler.
 1.18  10-Jul-1996  cgd print difference between pointers with %ld, so that -Wformat works
on the Alpha and for consistency. Also, other minor formatting cleanups.
 1.17  13-Jun-1996  cgd handle the case where kmem_malloc() returns NULL in the 'can wait' case.
Right now, this code just panic()s (same as kmem_malloc() used to do
before, but different message), but in the future it should be modified
to try to reclaim wasted memory.
 1.16  06-Jun-1996  cgd Fix a bug where free()ing an already-free block, or even a block that looked
like it might have already been freed, would cause a crash because of a bad
pointer dereference. Pointed out by Brian Noble <bnoble@cs.cmu.edu>.
 1.15  16-Mar-1996  christos branches: 1.15.4;
Fix printf() formats.
 1.14  20-Feb-1996  cgd when printing data modified on the free list:
(1) do not cast it to (void *), and
(2) print it as 0x%x, rather than %p.
This is not perfect (because the data being printed is "int32_t"-sized), but
is more correct than printing it as a pointer because the data is _not_ a
pointer, it is data to be printed in hex, and on some systems, pointers are
wider than the data items being printed, which leads to excess and misleading
output. The only 'right' solution to this is to have a printf specifier
that prints the fixed-sized types the right way, and that's not really
practical.
 1.13  09-Feb-1996  christos More proto fixes
 1.12  04-Feb-1996  christos First pass at prototyping
 1.11  01-May-1995  cgd make this work sanely on a 64-bit machine. Sacrifices a small bit of
error checking in the DIAGNOSTIC case. These changes might be backed out,
if it's decided that MINBUCKET should be 5 (rather than 4) on the alpha.
However, doing that has its own set of nasty consequences.
 1.10  19-Mar-1995  mycroft Use %p.
 1.9  29-Jun-1994  cgd New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'
 1.8  13-May-1994  cgd new kernel malloc. much better (but slower) diagnostic checking
 1.7  18-Dec-1993  mycroft Canonicalize all #includes.
 1.6  15-Jul-1993  cgd branches: 1.6.4;
get rid of bogus retrymalloc code (and it was bogus)...
 1.5  27-Jun-1993  andrew ANSIfications - removed all implicit function return types and argument
definitions. Ensured that all files include "systm.h" to gain access to
general prototypes. Casts where necessary.
 1.4  27-May-1993  deraadt patches to make kernel malloc() retryable. from <MDICKSON@CSI.compuserve.com>
 1.3  25-May-1993  cgd fix dependencies on M_LAST's old off-by-one error...
(arguably, the old error wasn't an error, but, given some
of the old usage of M_LAST and its name, i say it was)
 1.2  20-May-1993  cgd add $Id$ strings, and clean up file headers where necessary
 1.1  21-Mar-1993  cgd branches: 1.1.1;
Initial revision
 1.1.1.3  01-Mar-1998  fvdl Import 4.4BSD-Lite2
 1.1.1.2  01-Mar-1998  fvdl Import 4.4BSD-Lite for reference
 1.1.1.1  21-Mar-1993  cgd initial import of 386bsd-0.1 sources
 1.6.4.2  14-Nov-1993  mycroft Canonicalize all #includes.
 1.6.4.1  24-Sep-1993  mycroft Make all files using spl*() #include cpu.h. Changes from trunk.
init_main.c: New method of pseudo-device of initialization.
kern_clock.c: hardclock() and softclock() now take a pointer to a clockframe.
softclock() only does callouts.
kern_synch.c: Remove spurious declaration of endtsleep(). Adjust uses of
averunnable for new struct loadav.
subr_prf.c: Allow printf() formats in panic().
tty.c: averunnable changes.
vfs_subr.c: va_size and va_bytes are now quads.
 1.15.4.2  13-Jun-1996  cgd pull up from trunk:
>handle the case where kmem_malloc() returns NULL in the 'can wait' case.
>Right now, this code just panic()s (same as kmem_malloc() used to do
>before, but different message), but in the future it should be modified
>to try to reclaim wasted memory.
 1.15.4.1  06-Jun-1996  cgd pull up from trunk:
>Fix a bug where free()ing an already-free block, or even a block that looked
>like it might have already been freed, would cause a crash because of a bad
>pointer dereference. Pointed out by Brian Noble <bnoble@cs.cmu.edu>.
 1.24.8.1  14-Oct-1997  thorpej Update marc-pcmcia branch from trunk.
 1.33.2.1  30-Jul-1998  eeh Split vm_offset_t and vm_size_t into paddr_t, psize_t, vaddr_t, and vsize_t.
 1.41.4.2  02-Aug-1999  thorpej Update from trunk.
 1.41.4.1  21-Jun-1999  thorpej Sync w/ -current.
 1.45.8.1  27-Dec-1999  wrstuden Pull up to last week's -current.
 1.45.4.1  19-Oct-1999  fvdl Bring in Kirk McKusick's FFS softdep code on a branch.
 1.45.2.4  11-Feb-2001  bouyer Sync with HEAD.
 1.45.2.3  18-Jan-2001  bouyer Sync with head (for UBC+NFS fixes, mostly).
 1.45.2.2  08-Dec-2000  bouyer Sync with HEAD.
 1.45.2.1  20-Nov-2000  bouyer Update thorpej_scsipi to -current as of a month ago
 1.57.2.10  11-Nov-2002  nathanw Catch up to -current
 1.57.2.9  18-Oct-2002  nathanw Catch up to -current.
 1.57.2.8  17-Sep-2002  nathanw Catch up to -current.
 1.57.2.7  27-Aug-2002  nathanw Catch up to -current.
 1.57.2.6  17-Apr-2002  nathanw Catch up to -current.
 1.57.2.5  08-Jan-2002  nathanw Catch up to -current.
 1.57.2.4  14-Nov-2001  nathanw Catch up to -current.
 1.57.2.3  21-Sep-2001  nathanw Catch up to -current.
 1.57.2.2  24-Aug-2001  nathanw Catch up with -current.
 1.57.2.1  21-Jun-2001  nathanw Catch up to -current.
 1.61.2.5  10-Oct-2002  jdolecek sync kqueue with -current; this includes merge of gehenna-devsw branch,
merge of i386 MP branch, and part of autoconf rototil work
 1.61.2.4  06-Sep-2002  jdolecek sync kqueue branch with HEAD
 1.61.2.3  23-Jun-2002  jdolecek catch up with -current on kqueue branch
 1.61.2.2  10-Jan-2002  thorpej Sync kqueue branch with -current.
 1.61.2.1  25-Aug-2001  thorpej Merge Aug 24 -current into the kqueue branch.
 1.62.2.1  01-Oct-2001  fvdl Catch up with -current.
 1.63.2.1  12-Nov-2001  thorpej Sync the thorpej-mips-cache branch with -current.
 1.70.4.1  22-Mar-2002  thorpej Protect malloc internals with a spin mutex at IPL_VM.
 1.71.4.1  02-Jun-2003  tron Pull up revision 1.73 (requested by skrll):
add a new km flag UVM_KMF_CANFAIL, which causes uvm_km_kmemalloc() to
return failure if swap is full and there are no free physical pages.
have malloc() use this flag if M_CANFAIL is passed to it.
use M_CANFAIL to allow amap_extend() to fail when memory is scarce.
this should prevent most of the remaining hangs in low-memory situations.
 1.71.2.1  29-Aug-2002  gehenna catch up with -current.
 1.79.2.7  10-Nov-2005  skrll Sync with HEAD. Here we go again...
 1.79.2.6  01-Apr-2005  skrll Sync with HEAD.
 1.79.2.5  04-Mar-2005  skrll Sync with HEAD.

Hi Perry!
 1.79.2.4  17-Jan-2005  skrll Sync with HEAD.
 1.79.2.3  21-Sep-2004  skrll Fix the sync with head I botched.
 1.79.2.2  18-Sep-2004  skrll Sync with HEAD.
 1.79.2.1  03-Aug-2004  skrll Sync with HEAD
 1.95.4.2  19-Mar-2005  yamt sync with head. xen and whitespace. xen part is not finished.
 1.95.4.1  25-Jan-2005  yamt convert to new apis.
 1.95.2.1  29-Apr-2005  kent sync with -current
 1.99.2.10  24-Mar-2008  yamt sync with head.
 1.99.2.9  17-Mar-2008  yamt sync with head.
 1.99.2.8  21-Jan-2008  yamt sync with head
 1.99.2.7  07-Dec-2007  yamt sync with head
 1.99.2.6  15-Nov-2007  yamt sync with head.
 1.99.2.5  27-Oct-2007  yamt sync with head.
 1.99.2.4  03-Sep-2007  yamt sync with head.
 1.99.2.3  26-Feb-2007  yamt sync with head.
 1.99.2.2  30-Dec-2006  yamt sync with head.
 1.99.2.1  21-Jun-2006  yamt sync with head.
 1.101.8.1  11-Aug-2006  yamt sync with head
 1.101.4.1  09-Sep-2006  rpaulo sync with head
 1.102.6.2  10-Dec-2006  yamt sync with head.
 1.102.6.1  22-Oct-2006  yamt sync with head
 1.102.4.2  19-Jan-2007  ad Add some DEBUG code to check that items being freed were previously
allocated from the same source. Needs to be enabled via DDB.
 1.102.4.1  18-Nov-2006  ad Sync with head.
 1.105.4.1  03-Jun-2008  skrll Sync with netbsd-4.
 1.105.2.1  22-Feb-2008  bouyer Pull up following revision(s) (requested by yamt in ticket #1076):
sys/kern/kern_malloc.c: revision 1.117 via patch
malloc: fix freelist corruption.
 1.106.2.4  07-May-2007  yamt sync with head.
 1.106.2.3  24-Mar-2007  yamt sync with head.
 1.106.2.2  12-Mar-2007  rmind Sync with HEAD.
 1.106.2.1  27-Feb-2007  yamt - sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.
 1.108.2.5  26-Oct-2007  ad - malloc_lock must always be a spinlock.
- Remove spl calls.
 1.108.2.4  29-Jul-2007  ad Trap free() of areas that contain undestroyed locks. Not a major problem
but it helps to catch bugs.
 1.108.2.3  08-Jun-2007  ad Sync with head.
 1.108.2.2  21-Mar-2007  ad GC the simplelock/spinlock debugging stuff.
 1.108.2.1  13-Mar-2007  ad Sync with head.
 1.110.2.1  11-Jul-2007  mjf Sync with head.
 1.111.10.1  14-Oct-2007  yamt sync with head.
 1.111.8.4  23-Mar-2008  matt sync with HEAD
 1.111.8.3  09-Jan-2008  matt sync with HEAD
 1.111.8.2  08-Nov-2007  matt sync with -HEAD
 1.111.8.1  06-Nov-2007  matt sync with HEAD
 1.111.6.5  09-Dec-2007  jmcneill Sync with HEAD.
 1.111.6.4  27-Nov-2007  joerg Sync with HEAD. amd64 Xen support needs testing.
 1.111.6.3  14-Nov-2007  joerg Sync with HEAD.
 1.111.6.2  06-Nov-2007  joerg Sync with HEAD.
 1.111.6.1  26-Oct-2007  joerg Sync with HEAD.

Follow the merge of pmap.c on i386 and amd64 and move
pmap_init_tmp_pgtbl into arch/x86/x86/pmap.c. Modify the ACPI wakeup
code to restore CR4 before jumping back into kernel space as the large
page option might cover that.
 1.112.4.3  18-Feb-2008  mjf Sync with HEAD.
 1.112.4.2  08-Dec-2007  mjf Sync with HEAD.
 1.112.4.1  19-Nov-2007  mjf Sync with HEAD.
 1.112.2.1  13-Nov-2007  bouyer Sync with HEAD
 1.115.2.1  08-Dec-2007  ad Sync with head.
 1.116.4.1  08-Jan-2008  bouyer Sync with HEAD
 1.116.2.1  10-Dec-2007  yamt - separate kernel va allocation (kernel_va_arena) from
in-kernel fault handling (kernel_map).
- add vmem bootstrap code. vmem doesn't rely on malloc anymore.
- make kmem_alloc interrupt-safe.
- kill kmem_map. make malloc a wrapper of kmem_alloc.
 1.117.6.3  17-Jan-2009  mjf Sync with HEAD.
 1.117.6.2  28-Sep-2008  mjf Sync with HEAD.
 1.117.6.1  03-Apr-2008  mjf Sync with HEAD.
 1.117.2.1  24-Mar-2008  keiichi sync with head.
 1.119.10.2  13-Dec-2008  haad Update haad-dm branch to haad-dm-base2.
 1.119.10.1  19-Oct-2008  haad Sync with HEAD.
 1.119.6.1  10-Oct-2008  skrll Sync with HEAD.
 1.119.4.4  11-Aug-2010  yamt sync with head.
 1.119.4.3  11-Mar-2010  yamt sync with head
 1.119.4.2  16-Sep-2009  yamt sync with head
 1.119.4.1  04-May-2009  yamt sync with head.
 1.121.10.3  14-Mar-2014  matt Deal with the issue that VM_MAP_INTRSAFE km_suballoc returns more than asked
for so the space needs for kmemusage accounting has to needs adjusted as well.
 1.121.10.2  14-Mar-2014  matt Make sure uvm_km_suballoc returned the correctly sized submap.
 1.121.10.1  21-Apr-2010  matt sync to netbsd-5
 1.121.4.1  14-Feb-2010  bouyer Pull up following revision(s) (requested by hubertf in ticket #1292):
sys/kern/kern_malloc.c: revision 1.128
Let kernel build when MALLOCLOG is defined but DIAGNOSTIC is not.
Else, hitmlog() is defined but not used, which triggers a warning.
 1.121.2.1  19-Jan-2009  skrll Sync with HEAD.
 1.128.4.1  30-May-2010  rmind sync with head
 1.128.2.2  17-Aug-2010  uebayasi Sync with HEAD.
 1.128.2.1  30-Apr-2010  uebayasi Sync with HEAD.
 1.133.6.3  02-Jun-2012  mrg sync to latest -current.
 1.133.6.2  29-Apr-2012  mrg sync to latest -current.
 1.133.6.1  18-Feb-2012  mrg merge to -current.
 1.133.2.2  23-May-2012  yamt sync with head.
 1.133.2.1  17-Apr-2012  yamt sync with head
 1.138.8.1  18-Aug-2017  snj Pull up following revision(s) (requested by martin in ticket #1465):
sys/kern/kern_malloc.c: revision 1.146
Avoid integer overflow in kern_malloc(). Reported by Ilja Van Sprundel.
 1.138.6.1  18-Aug-2017  snj Pull up following revision(s) (requested by martin in ticket #1465):
sys/kern/kern_malloc.c: revision 1.146
Avoid integer overflow in kern_malloc(). Reported by Ilja Van Sprundel.
 1.138.2.1  18-Aug-2017  snj Pull up following revision(s) (requested by martin in ticket #1465):
sys/kern/kern_malloc.c: revision 1.146
Avoid integer overflow in kern_malloc(). Reported by Ilja Van Sprundel.
 1.142.12.1  10-Aug-2014  tls Rebase.
 1.142.4.1  28-Aug-2013  rmind Checkpoint work in progress:
- Initial split of the protocol user-request method into the following
methods: pr_attach, pr_detach and pr_generic for old the pr_usrreq.
- Adjust socreate(9) and sonewconn(9) to call pr_attach without the
socket lock held (as a preparation for the locking scheme adjustment).
- Adjust all pr_attach routines to assert that PCB is not set.
- Sprinkle various comments, document some routines and their locking.
- Remove M_PCB, replace with kmem(9).
- Fix few bugs spotted on the way.
 1.142.2.2  03-Dec-2017  jdolecek update from HEAD
 1.142.2.1  20-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.143.4.2  28-Aug-2017  skrll Sync with HEAD
 1.143.4.1  06-Apr-2015  skrll Sync with HEAD
 1.143.2.2  09-Aug-2017  snj Pull up following revision(s) (requested by martin in ticket #1461):
sys/kern/kern_malloc.c: revision 1.146
Avoid integer overflow in kern_malloc(). Reported by Ilja Van Sprundel.
 1.143.2.1  25-Mar-2015  snj branches: 1.143.2.1.2; 1.143.2.1.6;
Pull up following revision(s) (requested by maxv in ticket #617):
sys/kern/kern_malloc.c: revision 1.144, 1.145
sys/kern/kern_pmf.c: revision 1.37
sys/rump/librump/rumpkern/rump.c: revision 1.316
sys/uvm/uvm_extern.h: revision 1.193
sys/uvm/uvm_km.c: revision 1.139
Don't include <uvm/uvm_extern.h>
--
Kill kmeminit().
--
Remove this MALLOC_DEFINE (M_PMF unused).
 1.143.2.1.6.1  09-Aug-2017  snj Pull up following revision(s) (requested by martin in ticket #1461):
sys/kern/kern_malloc.c: revision 1.146
Avoid integer overflow in kern_malloc(). Reported by Ilja Van Sprundel.
 1.143.2.1.2.1  09-Aug-2017  snj Pull up following revision(s) (requested by martin in ticket #1461):
sys/kern/kern_malloc.c: revision 1.146
Avoid integer overflow in kern_malloc(). Reported by Ilja Van Sprundel.
 1.145.10.1  01-Aug-2017  snj Pull up following revision(s) (requested by martin in ticket #168):
sys/kern/kern_malloc.c: revision 1.146
Avoid integer overflow in kern_malloc(). Reported by Ilja Van Sprundel.
 1.146.4.2  13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.146.4.1  10-Jun-2019  christos Sync with HEAD
 1.146.2.3  26-Dec-2018  pgoyette Sync with HEAD, resolve a few conflicts
 1.146.2.2  26-Nov-2018  pgoyette Sync with HEAD, resolve a couple of conflicts
 1.146.2.1  06-Sep-2018  pgoyette Sync with HEAD

Resolve a couple of conflicts (result of the uimin/uimax changes)

RSS XML Feed