really, don't install uvm_physseg.h!

Use uvm_physseg.h:uvm_page_physload() instead of uvm_extern.h

For this, include uvm_physseg.h in the build and include tree, make a
cosmetic modification to the prototype for uvm_page_physload().

branches: 1.9.134; 1.9.138;
remove the following options. no objections on tech-kern@.

UVM_PAGER_INLINE
UVM_AMAP_INLINE
UVM_PAGE_INLINE
UVM_MAP_INLINE

branches: 1.8.22; 1.8.34; 1.8.36; 1.8.38;
Remove KDIR=, since SYS_INCLUDE=symlinks and KDIR are not supported any more.

a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.6.2; 1.6.4; 1.6.6;
Rename VM_INHERIT_* to MAP_INHERIT_* and move them to sys/sys/mman.h as
discussed on tech-kern.
Retire sys/uvm/uvm_inherit.h, update man page for minherit(2).

install uvm_pmap.h

install uvm_param.h.

remove/move more mach vm header files:

<vm/pglist.h> -> <uvm/uvm_pglist.h>
<vm/vm_inherit.h> -> <uvm/uvm_inherit.h>
<vm/vm_kern.h> -> into <uvm/uvm_extern.h>
<vm/vm_object.h> -> nothing
<vm/vm_pager.h> -> into <uvm/uvm_pager.h>

also includes a bunch of <vm/vm_page.h> include removals (due to redudancy
with <vm/vm.h>), and a scattering of other similar headers.

<vm/vm_prot.h> becomes <uvm/uvm_prot.h>

branches: 1.1.14;
Rework the way kernel include files are installed. In the new method,
as with user-land programs, include files are installed by each directory
in the tree that has includes to install. (This allows more flexibility
as to what gets installed, makes 'partial installs' easier, and gives us
more options as to which machines' includes get installed at any given
time.) The old SYS_INCLUDES={symlinks,copies} behaviours are _both_
still supported, though at least one bug in the 'symlinks' case is
fixed by this change. Include files can't be build before installation,
so directories that have includes as targets (e.g. dev/pci) have to move
those targets into a different Makefile.

branches: 1.1.2;
file README.ubc was initially added on branch chs-ubc.

uvm(9): Enable swap encryption by default.

For machines where the performance impact of swapping before the
system has an opportunity to process `vm.swap_encrypt=0' in
/etc/sysctl.conf, you can disable it again by adding

options VMSWAP_DEFAULT_PLAINTEXT

to the kernel config.

branches: 1.36.6; 1.36.10;
G/C USE_TOPDOWN_VM. __USE_TOPDOWN_VM is used (and hidden)

uvm(9): Switch from legacy rijndael API to new aes API.

Add missing dependency.

Fixes builds with VM_SWAP but no other users of rijndael crypto code.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.32.2;
Redo the page allocator to perform better, especially on multi-core and
multi-socket systems. Proposed on tech-kern. While here:

- add rudimentary NUMA support - needs more work.
- remove now unused "listq" from vm_page.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Move all non-emulation-specific coredump code into the coredump module,
and remove all #ifdef COREDUMP conditional compilation. Now, the
coredump module is completely separated from the emulation modules, and
they can all be independently loaded and unloaded.

Welcome to 9.99.18 !

branches: 1.29.2;
Remove emap support. Unfortunately it never got to state where it would be
used and usable, due to reliability and limited & complicated MD support.

Going forward, we need to concentrate on interface which do not map anything
into kernel in first place (such as direct map or KVA-less I/O), rather
than making those mappings cheaper to do.

branches: 1.28.14;
"Make NetBSD great again!"

Introduce uvm_hotplug(9) to the kernel.

Many thanks, in no particular order to:

TNF, for funding the project.

Chuck Silvers - for multiple API reviews and feedback.
Nick Hudson - for testing on multiple architectures and bugfix patches.
Everyone who helped with boot testing.

KeK (http://www.kek.org.in) for hosting the primary developers.

allow the sizes of the maphist and pdhist to be set in the config
file via UVMHIST_MAPHIST_SIZE and UVMHIST_PDHIST_SIZE.

Make UVMHIST_PRINT work again by making it define KERNHIST_PRINT

branches: 1.25.2;
Several changes and improvements in KMEM_GUARD:
- merge uvm_kmguard.{c,h} into subr_kmem.c. It is only user there, and
makes it more consistent. Also, it allows us to enable KMEM_GUARD
without enabling DEBUG.
- rename uvm_kmguard_XXX to kmem_guard_XXX, for consistency
- improve kmem_guard_alloc() so that it supports allocations bigger than
PAGE_SIZE
- remove the canary value, and use directly the kmem header as underflow
pattern.
- fix some comments

(The UAF fifo is disabled for the moment; we actually need to register
the va and its size, and add a weight support not to consume too much
memory.)

UVM_RESERVED_PAGES_PER_CPU must be a param, not a flag.

Allow changing the per-cpu emergency page reservation via kernel config.

branches: 1.22.2;
Use opt_*.h to not polute CPPFLAGS.

Attribute dependency is not yet. Revert a definition.

Define "uvm" attribute and mark files.

branches: 1.20.4; 1.20.14;
fix the ordering and make UVMHIST enable KERNHIST automatically.

branches: 1.19.2;
Make UVM_PAGE_TRKOWN a real flag.

branches: 1.18.2;
rename the va0_disabled option and cpp conditional to "disable" as well,
for consistency, and document option and sysctl flag

Disable mapping of virtual address 0 by user programs per default.
This blocks an easy exploit of kernel bugs leading to dereference
of a NULL pointer on some architectures (eg i386).
The check can be disabled in various ways:
-by CPP definitions in machine/types.h (portmaster's choice)
-by a kernel config option USER_VA0_DISABLED_DEFAULT=0
-at runtime by sysctl vm.user_va0_disabled (cannot be cleared
at securelevel>0)

branches: 1.16.2;
Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Add [default] option to make UAREAs swappable. Disabling the option makes
them unswappable and therefore allocatable using KSEG/BAT/etc.

Ephemeral mapping (emap) implementation. Concept is based on the idea that
activity of other threads will perform the TLB flush for the processes using
emap as a side effect. To track that, global and per-CPU generation numbers
are used. This idea was suggested by Andrew Doran; various improvements to
it by me. Notes:

- For now, zero-copy on pipe is not yet enabled.
- TCP socket code would likely need more work.
- Additional UVM loaning improvements are needed.

Proposed on <tech-kern>, silence there.
Quickly reviewed by <ad>.

kernel memory guard for DEBUG kernels, proposed on tech-kern.
See kmem_alloc(9) for details.

branches: 1.12.4;
Make the emulations, exec formats, coredump, NFS, and the NFS server
into modules. By and large this commit:

- shuffles header files and ifdefs
- splits code out where necessary to be modular
- adds module glue for each of the components
- adds/replaces hooks for things that can be installed at runtime

branches: 1.11.4; 1.11.6; 1.11.12;
- Switch off the map evcnts by default.
- SAVE_HINT() doesn't need to be atomic.

branches: 1.10.16; 1.10.18; 1.10.20; 1.10.22;
defparam PAGER_MAP_SIZE.

branches: 1.9.6; 1.9.8; 1.9.12;
Add native mremap system call based on the UVM implementation for
Linux compat. Add code to enforce alignment of the new location.
Special thanks to wizd for helping with the man page.

branches: 1.8.8;
PR/34837: Mindaguas: Add SysV SHM dynamic reallocation and locking to the
physical memory

uobj_wirepages and uobj_unwirepages from Mindaugas. PR/34771.
(commented out in files.uvm for now because there is no user in tree.)

http://mail-index.netbsd.org/tech-kern/2006/09/24/0000.html
http://mail-index.netbsd.org/tech-kern/2006/10/10/0000.html

add ubc window hit/miss evcnts.

branches: 1.5.2;
merge yamt-pdpolicy branch.
- separate page replacement policy from the rest of kernel
- implement an alternative replacement policy

branches: 1.4.6; 1.4.16; 1.4.20;
implement compat_linux mremap.

branches: 1.3.2; 1.3.4;
merge yamt-readahead branch.

branches: 1.2.2;
read-ahead statistics.

Move UVM files to files.uvm

uvm(9): One rndsource for faults -- not one per CPU.

All relevant state is per-CPU anyway; the only substantive difference
this makes is how many entries appear in `rndctl -l' output and what
they are called -- formerly the somewhat confusing `cpuN', meaning
`page faults on cpuN', and now just `uvmfault'. I don't think
there's any real value in being able to enable or disable measurement
or counting of page faults on one CPU vs others, so although this
could be a minor compatibility change, it's hard to imagine it
matters much.

XXX kernel ABI change in struct cpu_info

- If the hardware provided NUMA info, then use it to decide how to set up
the allocator's buckets, instead of doing round robin distribution. There
are open questions here but this is better than doing nothing.

- Kernel reserve pages are for the kernel not realtime threads.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

remove the aiodoned thread. I originally added this to provide a thread context
for doing page cache iodone work, but since then biodone() has changed to
hand off all iodone work to a softint thread, so we no longer need the
special-purpose aiodoned thread.

branches: 1.73.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

Nothing uses uvm.cpus any more, and we can do the same with cpu_lookup(),
so get rid of it.

Redo the page allocator to perform better, especially on multi-core and
multi-socket systems. Proposed on tech-kern. While here:

- add rudimentary NUMA support - needs more work.
- remove now unused "listq" from vm_page.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

- Adjust uvmexp.swpgonly with atomics, and make uvm_swap_data_lock static.
- A bit more __cacheline_aligned on mutexes.

branches: 1.68.16;
Move sys/uvm/uvm_physseg.h inclusion to within _KERNEL only.

Use uvm_physseg.h:uvm_page_physload() instead of uvm_extern.h

For this, include uvm_physseg.h in the build and include tree, make a
cosmetic modification to the prototype for uvm_page_physload().

branches: 1.66.2;
Limit <sys/rndsource.h> include to kernel.

Convert remaining MI <sys/rnd.h> stragglers. Many MD ones left.

branches: 1.64.4;
Merge tls-earlyentropy branch into HEAD.

branches: 1.63.6; 1.63.20;
Entropy-pool implementation move and cleanup.

1) Move core entropy-pool code and source/sink/sample management code
to sys/kern from sys/dev.

2) Remove use of NRND as test for presence of entropy-pool code throughout
source tree.

3) Remove use of RND_ENABLED in device drivers as microoptimization to
avoid expensive operations on disabled entropy sources; make the
rnd_add calls do this directly so all callers benefit.

4) Fix bug in recent rnd_add_data()/rnd_add_uint32() changes that might
have lead to slight entropy overestimation for some sources.

5) Add new source types for environmental sensors, power sensors, VM
system events, and skew between clocks, with a sample implementation
for each.

ok releng to go in before the branch due to the difficulty of later
pullup (widespread #ifdef removal and moved files). Tested with release
builds on amd64 and evbarm and live testing on amd64.

branches: 1.62.4; 1.62.8;
move and rename the uvm history code out of uvm_stat to "kernhist".

rename "UVMHIST" option to enable the uvm histories.

TODO:
- make UVMHIST properly depend upon KERNHIST
- enable dynamic registration of histories. this is mostly just
allocating something in a bitmap, and is only for viewing multiple
histories in a merged form.


tested on amd64 and sparc64.

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

branches: 1.59.2; 1.59.4;
Make UVM_PAGE_TRKOWN a real flag.

Reduce memory spent on bookkeeping for large values of MAXCPUS.

branches: 1.57.2; 1.57.4;
Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Ephemeral mapping (emap) implementation. Concept is based on the idea that
activity of other threads will perform the TLB flush for the processes using
emap as a side effect. To track that, global and per-CPU generation numbers
are used. This idea was suggested by Andrew Doran; various improvements to
it by me. Notes:

- For now, zero-copy on pipe is not yet enabled.
- TCP socket code would likely need more work.
- Additional UVM loaning improvements are needed.

Proposed on <tech-kern>, silence there.
Quickly reviewed by <ad>.

branches: 1.55.12; 1.55.16;
Replace the global vm_page hash with a per vm_object rbtree.
Proposed on tech-kern@.

- vm_page: put listq, pageq into a union alongside a LIST_ENTRY, so we can
use both types of list.

- Make page coloring and idle zero state per-CPU.

- Maintain per-CPU page freelists. When freeing, put pages onto the local
CPU's lists and the global lists. When allocating, prefer to take pages
from the local CPU. If none are available take from the global list as
done now. Proposed on tech-kern@.

branches: 1.53.6; 1.53.8; 1.53.10; 1.53.12;
Merge vmlocking2 to head.

branches: 1.52.6; 1.52.12; 1.52.14; 1.52.18; 1.52.22;
Merge unobtrusive locking changes from the vmlocking branch.

branches: 1.51.2;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

Add a sysctl to disable swapout of kernel stacks. Discussed on tech-kern@.

branches: 1.49.4; 1.49.6;
Pick up some additional files that were missed before due to conflicts
with newlock2 merge:

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

uvm_kick_scheduler(): do nothing until the swap subsystem is initialized.

branches: 1.46.2;
Add uvm_kick_scheduler() (MP safe) to replace wakeup(&proc0).

merge yamt-splraiseipl branch.

- finish implementing splraiseipl (and makeiplcookie).
http://mail-index.NetBSD.org/tech-kern/2006/07/01/0000.html
- complete workqueue(9) and fix its ipl problem, which is reported
to cause audio skipping.
- fix netbt (at least compilation problems) for some ports.
- fix PR/33218.

branches: 1.44.2;
merge yamt-pdpolicy branch.
- separate page replacement policy from the rest of kernel
- implement an alternative replacement policy

branches: 1.43.2; 1.43.14;
remove the following options. no objections on tech-kern@.

UVM_PAGER_INLINE
UVM_AMAP_INLINE
UVM_PAGE_INLINE
UVM_MAP_INLINE

branches: 1.42.2; 1.42.4; 1.42.6;
read-ahead statistics.

branches: 1.41.2;
don't include uvm_*_i.h unless needed,
to reduce bogus header dependencies.

branches: 1.40.2; 1.40.4;
allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

for in-kernel maps,
- allocate kva for vm_map_entry from the map itsself and
remove the static limit, MAX_KMAPENT.
- keep merged entries for later splitting to fix allocate-to-free problem.
PR/24039.

introduce UVMHIST_LOANHIST and sprinkle UVMHIST_LOGs.

Back out the changes in
http://mail-index.netbsd.org/source-changes/2004/01/29/0027.html
since they don't really fix the problem.

Incorpate one fix: Mark uvm_map_entry's that were created with
UVM_FLAG_NOMERGE so that they will not be used as future merge
candidates.

- split uvm_map() into two functions for the followings.
- for in-kernel maps, disable map entry merging so that
unmap operations won't block. (workaround for PR/24039)
- for in-kernel maps, allocate kva for vm_map_entry from
the map itsself and eliminate MAX_KMAPENT and
uvm_map_entry_kmem_pool.

branches: 1.35.6;
Reorder things so that with multiple inclusion protection that optional
definitions are outside the protection checks.

gah. reversed a test.

/*CONTCOND*/, and protect UVMHIST_DECL with #ifdef UVMHIST

Protect "struct uvm" with _KERNEL.

branches: 1.31.6; 1.31.12;
a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.30.2; 1.30.4;
Macro'ize the code that checks the free and inactive thresholds and
wakes the pagedaemon.

replace vm_map{,_entry}_t with struct vm_map{,_entry} *.

use _KERNEL_OPT

replace {simple_,}lock{_data,}_t with struct {simple,}lock {,*}.

Merge the swap-backed and object-backed inactive lists.

Implement page coloring, using a round-robin bucket selection
algorithm (Solaris calls this "Bin Hopping").

This implementation currently relies on MD code to define a
constant defining the number of buckets. This will change
reasonably soon (MD code will be able to dynamically size
the bucket array).

branches: 1.24.2;
Initial integration of the Unified Buffer Cache project.

remove/move more mach vm header files:

<vm/pglist.h> -> <uvm/uvm_pglist.h>
<vm/vm_inherit.h> -> <uvm/uvm_inherit.h>
<vm/vm_kern.h> -> into <uvm/uvm_extern.h>
<vm/vm_object.h> -> nothing
<vm/vm_pager.h> -> into <uvm/uvm_pager.h>

also includes a bunch of <vm/vm_page.h> include removals (due to redudancy
with <vm/vm.h>), and a scattering of other similar headers.

Change UVM_UNLOCK_AND_WAIT() to use ltsleep() (it is now atomic, as
advertised). Garbage-collect uvm_sleep().

branches: 1.21.2;
Changes necessary to implement pre-zero'ing of pages in the idle loop:
- Make page free lists have two actual queues: known-zero pages and
pages with unknown contents.
- Implement uvm_pageidlezero(). This function attempts to zero up to
the target number of pages until the target has been reached (currently
target is `all free pages') or until whichqs becomes non-zero (indicating
that a process is ready to run).
- Define a new hook for the pmap module for pre-zero'ing pages. This is
used to zero the pages using uncached access. This allows us to zero
as many pages as we want without polluting the cache.

In order to use this feature, each platform must add the appropropriate
glue in their idle loop.

tidy.

Instead of checking vm_physmem[<physseg>].pgs to determine if
uvm_page_init() has completed, add a boolean uvm.page_init_done,
and test against that. Use this same boolean (rather than
pmap_initialized) in pmap_growkernel() to determine if we are
being called via uvm_page_init() to grow the kernel address space.

This fixes a problem on some i386 configurations where pmap_init()
itself was needing to have the kernel page table grown, and since
pmap_initialized was not yet set to TRUE, pmap_growkernel() was
choosing the wrong code path.

Fix tested by Havard Eidnes.

Always pass all arguments to uvm_sleep().

branches: 1.17.2; 1.17.4; 1.17.8;
Garbage collect thread_sleep()/thread_wakeup() left over from the old
Mach VM code. Also nuke iprintf(), which was no longer used anywhere.

Add proclist locking where appropriate.

Protect prototypes, certain macros, and inlines from userland.

branches: 1.15.2; 1.15.4;
add uvmexp.swpgonly and use it to detect out-of-swap conditions.

remove now >1 year old pre-release message.

branches: 1.13.2;
remove unused share map code from UVM:
dump UVM_ET_MAP/UVM_ET_ISMAP. if you need to detect a submap use
UVM_ET_SUBMAP/UVM_ET_ISSUBMAP.

NCPU > 1 -> MULTIPROCESSOR

Merge paddr_t changes into the main branch.

branches: 1.10.2;
Add support for multiple memory free lists. There is at least one
default free list, and 0 - N additional free list, in order of descending
priority.

A new page allocation function, uvm_pagealloc_strat(), has been added,
providing three page allocation strategies:

- normal: high -> low priority free list walk, taking the
page off the first free list that has one.

- only: attempt to allocate a page only from the specified free
list, failing if that free list has none available.

- fallback: if `only' fails, fall back on `normal'.

uvm_pagealloc(...) is provided for normal use (and is a synonym for
uvm_pagealloc_strat(..., UVM_PGA_STRAT_NORMAL, 0); the free list argument
is ignored for the `normal' case).

uvm_page_physload() now specified which free list the pages will be
loaded onto. This means that some platforms which have multiple physical
memory segments may define additional vm_physsegs if they wish to break
individual physical segments into differing priorities.

Machine-dependent code must define _at least_ the following constants
in <machine/vmparam.h>:

VM_NFREELIST: the number of free lists the system will have

VM_FREELIST_DEFAULT: the default freelist (should always be 0,
but is defined in machdep code so that it's with all of the
other free list-related constants).

Additional free list names may be defined by machine-dependent code, but
they will only be used by machine-dependent code (e.g. for loading the
vm_physsegs).

Shield `#include opt_*.h'.

defopt LOCKDEBUG

No dummy locks if LOCKDEBUG.

Merge with Lite2 + local changes

- add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.

add/cleanup multiple inclusion protection.

restore rcsids

RCS ID police.

branches: 1.1.1;
Initial revision

Align uvm_amap to COHERENCY_UNIT.

s/value value/value/ in comment. No functional change.

uvm(9): KASSERT(A && B) -> KASSERT(A); KASSERT(B)

Consistently use %#jx instead of 0x%jx or just %jx in UVMHIST_LOG formats

branches: 1.125.2;
the previous fix for PR 55366 in uvm_amap.c 1.124 was incomplete:
- amap_adjref_anons() must also ignore AMAP_REFALL when updating
the ppref, not just when deciding whether or not to initialize ppref.
- UVM_EXTRACT_QREF relies on AMAP_REFALL to work properly,
and since we can't use AMAP_REFALL then we can't use QREF either.

Effectively disable the AMAP_REFALL flag because it is unsafe.
This flag tells the amap code that it does not need to allocate ppref
as part of adding or removing a reference, but that is only correct
if the range of the reference being added or removed is the same
as the range of all other references to the amap, and the point of
this flag is exactly to try to optimize the case where the range is
different and thus this flag would not be correct to use.
Fixes PR 55366.

fix amap_extend() to handle amaps where we previously failed to allocate
the ppref memory.

Consistently use UVMHIST(__func__)

Convert UVMHIST_{CALLED,LOG} into UVMHIST_CALLARGS

Trailing whitespace

Mark amappl with PR_LARGECACHE.

Go back to freeing struct vm_anon one by one. There may have been an
advantage circa ~2008 but there isn't now.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

- Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
any priority boost gained earlier from blocking.

0x%#x --> %#x for non-external codes.
Also, stop mixing up 0x%x and %#x in single files as far as possible.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

branches: 1.114.2;
Back out the amap allocation changes from earlier today - have seen a panic
with them. Retain the lock changes.

- Start trying to reduce the high cache miss rate observed around vm_amap.
On _LP64, pad struct vm_amap to 128 bytes and use the additional space to
hold the arrays for tiny amaps which are common. Carefully size the array
allocations to avoid false sharing, and for smaller amaps try to share
allocated cache lines.

- Eliminate most contention due to amap_list: maintain the list in the pool
cache constructor / destructor like we do for struct file. Cache the
mutexes we allocate here.

- Don't do PR_WAITOK mutex allocations when NOWAIT has been specified.

PR kern/54821: 9.99.32 assertion in uvm_pageactivate

Looks like I forgot to commit this file yesterday.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

Activate pages in batch instead of acquring uvm_pageqlock a zillion times.

branches: 1.109.4;
Rename 'slotspace' -> 'slotarea' in UVM, to avoid (future) collision with
the x86 slotspace structure.

branches: 1.108.2; 1.108.4;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

branches: 1.107.2; 1.107.32;
initialize amap per-page reference counts before changing the amap's
overall reference count. this fixes the crashes seen for the last 9 months
with web browers and plugins, which was also the cause of PR 46193.

adjust amap_cow_now() to make UVM_PAGE_TRKOWN happy.

branches: 1.105.2;
extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

branches: 1.104.2; 1.104.6;
assertion

amap_cow_now: just free the fresh anon on error, no need to dispose it.

- Rework uvm_anfree() into uvm_anon_freelst(), which always drops the lock.
- Free anons in uvm_anon_freelst() without lock held.
- Mechanic sync to unused loaning code.

reduce the number of atomic ops in common cases. it's exceptional for
anons to remain longer than amap.

amap_copy(): Keep the source amap locked until its lock has been copied.

Kernel assertion "anon->an_lock == amap->am_lock" no longer fails.

Ok: Mindaugas Rasiukevicius <rmind@netbsd.org>

amap_copy: fix one more regression, thanks to enami@.

Fix uvmplock regression - a lock against oneself case in amap_swap_off().
Happens since amap is NULL in uvmfault_anonget(), so uvmfault_unlockall()
keeps anon locked, when it should unlock it.

amap_pp_adjref: fix regression, spotted by nonaka@.

Clean-up, add asserts, slightly simplify.

Clean up, sprinkle asserts, consify, use unsigned, use kmem_zalloc instead
of memset, reduce the scope of some variables, improve some comments.

No functional change intended.

amap_add/amap_unadd: clean up slightly, use unsigned, add asserts.

Add amap_adjref_anons() helper and simplify amap_ref()/amap_unref().

amap_lookup{s}: add assert, clean-up slightly.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.90.2;
Replace "malloc" in comments, remove unnecessary header inclusions.

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

branches: 1.88.4; 1.88.6; 1.88.8;
Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

assertions

Convert some panic() checks to KASSERT()s.
This code is stable and there is no reason to enforce checks.

branches: 1.85.4;
Make adjustment of uvm_extrapages atomic since it's done without a lock.
XXX This is still a hack.

branches: 1.84.6; 1.84.10; 1.84.16; 1.84.18; 1.84.20;
Merge vmlocking2 to head.

branches: 1.83.4;
Allocate amaps from a pool_cache.

branches: 1.82.4; 1.82.6; 1.82.12; 1.82.14; 1.82.16;
Merge unobtrusive locking changes from the vmlocking branch.

branches: 1.81.2;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

branches: 1.80.2; 1.80.4;
Pass an ipl argument to pool_init/POOL_INIT to be used when initializing
the pool's lock.

branches: 1.79.4;
TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.77.2;
Merge newlock2 to head.

remove some __unused from function parameters.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.74.4; 1.74.6;
make amap use kmem_alloc, rather than malloc.
(ie. make it use kernel_map, rather than kmem_map.)
kmem_map is more restricted than kernel_map,
and there's no point for amap to use it.

branches: 1.73.4;
amap_splitref: assert that origref->ar_amap is initialized
by caller beforehand.

branches: 1.72.2; 1.72.4; 1.72.6;
- amap_copy: take a "flags" argument instead of booleans.
- add AMAP_COPY_NOMERGE flag, and use it for uvm_map_extract.
PR/32806 from Julio M. Merino Vidal.

remove the following options. no objections on tech-kern@.

UVM_PAGER_INLINE
UVM_AMAP_INLINE
UVM_PAGE_INLINE
UVM_MAP_INLINE

branches: 1.70.2; 1.70.4;
- uvm_fault: move a common code of 1B and 2B to a new function.
don't attempt to allocate anons with kernel_map locked. PR/32543.
- amap_copy: add an assertion.

in amap_alloc(), only put the amap on the list of amaps if we succeeded
in allocating it.

branches: 1.68.2;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

in amap_cow_now(), handle the case where we have to sleep and some of the
already-copied pages are paged out. anons that have already been copied
will have refcount == 1, whereas anons that still need to be copied will
have refcount > 1. fixes PR 25392, PR 30257, PR 31924.

wrap swap related code by #ifdef VMSWAP. always #define VMSWAP for now.

revert "defflag VMSWAP" changes for now.
there seems to be far more people who don't want to edit
their kernel config files than i thought.

defflag VMSWAP.

branches: 1.62.2;
Use ANSI function decls.

(try to) merge map entries in fault handler.

allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

- amap_extend: don't extend amap beyond UVM_AMAP_LARGE.
- uvm_map_enter: if we fail to extend amap, just give up merging instead of
bailing out immediately.

amap_wipeout: remove a comment which is no longer true.
despite of what comment said, i left preempt() call
because i don't think of any bad effects.

branches: 1.57.4;
hack around a UVM problem that causes hangs when large processes fork.
see PR 26908 for details.

branches: 1.56.2; 1.56.4;
for in-kernel maps,
- allocate kva for vm_map_entry from the map itsself and
remove the static limit, MAX_KMAPENT.
- keep merged entries for later splitting to fix allocate-to-free problem.
PR/24039.

add assertions.

Initialise (most) pools from a link set instead of explicit calls
to pool_init. Untouched pools are ones that either in arch-specific
code, or aren't initialiased during initial system startup.

Convert struct session, ucred and lockf to pools.

branches: 1.53.2; 1.53.4;
- Nuke __P().
- Drop trailing spaces.

branches: 1.52.2;
Add extensible malloc types, adapted from FreeBSD. This turns
malloc types into a structure, a pointer to which is passed around,
instead of an int constant. Allow the limit to be adjusted when the
malloc type is defined, or with a function call, as suggested by
Jonathan Stone.

amap_copy: remove stray amap_unlock().

Merge the nathanw_sa branch.

Properly set page references counts at the start of the newly
allocated ppref data to zero in the case of an amap that has empty
space at the front.

Don't set anything in the ppref array if "len" is zero.

Many thanks to Sami Kantoluoto for providing gdb access to a machine
that would reliably crash with problems related to the above, and to
Stephan Thesing for corroborating that the patch properly addressed
the problem.

Note that the ar_pageoff (and related variables) types must be changed
soon. The use of "int" here is not theoretically sufficient.

Change uvm_km_kmemalloc() to accept flag UVM_KMF_NOWAIT and pass it to
uvm_map(). Change uvm_map() to honnor UVM_KMF_NOWAIT. For this, change
amap_extend() to take a flags parameter instead of just boolean for
direction, and introduce AMAP_EXTEND_FORWARDS and AMAP_EXTEND_NOWAIT flags
(AMAP_EXTEND_BACKWARDS is still defined as 0x0, to keep the code easier to
read).
Add a flag parameter to uvm_mapent_alloc().
This solves a problem a pool_get(PR_NOWAIT) could trigger a pool_get(PR_WAITOK)
in uvm_mapent_alloc().
Thanks to Chuck Silvers, enami tsugutomo, Andrew Brown and Jason R Thorpe
for feedback.

Properly free "newppref", instead of "amap->am_ppref" (oops), and
delay freeing the old am_ppref so that if we bail early due to
malloc() failures, valid ppref data hasn't been freed for no reason.

Based on comments from enami.

Implement backwards extension of amaps. There are three cases to deal
with:

Case #1 -- adjust offset: The slot offset in the aref can be
decremented to cover the required size addition.

Case #2 -- move pages and adjust offset: The slot offset is not large
enough, but the amap contains enough inactive space *after* the mapped
pages to make up the difference, so active slots are slid to the "end"
of the amap, and the slot offset is, again, adjusted to cover the
required size addition. This optimizes for hitting case #1 again on
the next small extension.

Case #3 -- reallocate, move pages, and adjust offset: There is not
enough inactive space in the amap, so the arrays are reallocated, and
the active pages are copied again to the "end" of the amap, and the
slot offset is adjusted to cover the required size. This also
optimizes for hitting case #1 on the next backwards extension.

This provides the missing piece in the "forward extension of
vm_map_entries" logic, so the merge failure counters have been
removed.

Not many applications will make any use of this at this time (except
for jvms and perhaps gcc3), but a "top-down" memory allocator will use
it extensively.

add a new km flag UVM_KMF_CANFAIL, which causes uvm_km_kmemalloc() to
return failure if swap is full and there are no free physical pages.
have malloc() use this flag if M_CANFAIL is passed to it.
use M_CANFAIL to allow amap_extend() to fail when memory is scarce.
this should prevent most of the remaining hangs in low-memory situations.

rearrange a few lines to appease an assertion.

branches: 1.43.2; 1.43.4;
In amap_pp_adjref(), avoid incorrectly merging the first two chunks in
a ppref array when the range being adjusted includes the beginning of
the array.

branches: 1.42.2;
Pool deals fairly well with physical memory shortage, but it doesn't
deal with shortages of the VM maps where the backing pages are mapped
(usually kmem_map). Try to deal with this:

* Group all information about the backend allocator for a pool in a
separate structure. The pool references this structure, rather than
the individual fields.
* Change the pool_init() API accordingly, and adjust all callers.
* Link all pools using the same backend allocator on a list.
* The backend allocator is responsible for waiting for physical memory
to become available, but will still fail if it cannot callocate KVA
space for the pages. If this happens, carefully drain all pools using
the same backend allocator, so that some KVA space can be freed.
* Change pool_reclaim() to indicate if it actually succeeded in freeing
some pages, and use that information to make draining easier and more
efficient.
* Get rid of PR_URGENT. There was only one use of it, and it could be
dealt with by the caller.

From art@openbsd.org.

in amap_pp_adjref(), avoid unnecessary fragmentation of the am_ppref array
by merging the first changed chunk with the last unchanged chunk if possible.

When initially allocating or extending arrays in struct uvm_amap,
adjust allocation size using malloc_roundup(). This eliminates many
unnecessary malloc/memcpy calls.

No need to zero clear after amap->am_bckptr[amap->am_nslot], since we're
clearing corresponding elements in an array amap->am_anon[].

fix bug in amap_wiperange() detected by enami tsugutomo.
loop control was wrong in one case.

add RCSIDs, and in some cases, slightly cleanup #include order

Change some unsigned int variables and parameters to plain ints so
that all usages of those agree on unsigned vs. signed.

branches: 1.35.2;
work around swap-space/extent performance problem which causes
long pauses when processes with lots of swapped-out pages exit.

a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.33.2;
seperate -> separate

branches: 1.32.2;
replace vm_map{,_entry}_t with struct vm_map{,_entry} *.

remove trailing whitespace.

branches: 1.30.2;
clean up DIAGNOSTIC checks, use KASSERT().

Change uvm_analloc() to return a locked anon, update all callers,
and fix an anon locking protocol error in uvm_loanzero().

Sprinkle some assertions:
amap_free(): Assert that the amap is locked.
amap_share_protect(): Assert that the amap is locked.
amap_wipeout(): Assert that the amap is locked.
uvm_anfree(): Assert that the anon has a reference count of 0 and is
not locked.
uvm_anon_lockloanpg(): Assert that the anon is locked.
anon_pagein(): Assert that the anon is locked.
uvmfault_anonget(): Assert that the anon is locked.
uvm_pagealloc_strat(): Assert that the uobj or the anon is locked

And fix the problems these have uncovered:
amap_cow_now(): Lock the new anon after allocating it, and unref and
unlock it (rather than lock!) before freeing it in case
of an error condition. This should fix a problem reported
by Dan Carosone using cdrecord on an i386 MP kernel.
uvm_fault(): Case1B -- Lock the new anon afer allocating it, and unlock
it later when we unlock the old anon.
Case2 -- Lock the new anon after allocating it, and unlock
it later by passing it to uvmfault_unlockall() (we set anon
to NULL if we're not doing a promote fault).

lots of cleanup:
use queue.h macros and KASSERT().
address amap offsets in pages instead of bytes.
make amap_ref() and amap_unref() take an amap, offset and length
instead of a vm_map_entry_t.
improve whitespace and comments.

MALLOC()/FREE() are not to be used for variable size allocations.

Fix a fairly obvious locking error in amap_cow_now() -- the amap was
left locked upon exit from the function (how did this one slip for
so long?)

remove include of <vm/vm.h>

remove/move more mach vm header files:

<vm/pglist.h> -> <uvm/uvm_pglist.h>
<vm/vm_inherit.h> -> <uvm/uvm_inherit.h>
<vm/vm_kern.h> -> into <uvm/uvm_extern.h>
<vm/vm_object.h> -> nothing
<vm/vm_pager.h> -> into <uvm/uvm_pager.h>

also includes a bunch of <vm/vm_page.h> include removals (due to redudancy
with <vm/vm.h>), and a scattering of other similar headers.

branches: 1.22.2; 1.22.12;
eliminate the PMAP_NEW option by making it required for all ports.
ports which previously had no support for PMAP_NEW now implement
the pmap_k* interfaces as wrappers around the non-k versions.

fix allocation handling bugs in amap_alloc1(). if the first or second
sub-structure malloc() failed, it was quite likely that the function
would return success incorrectly. This is this direct cause of the bug
reported in PR#7897. (Thanks to chs for helping to track it down.)

add a `flags' argument to uvm_pagealloc_strat().
define a flag UVM_PGA_USERESERVE to allow non-kernel object
allocations to use pages from the reserve.
use the new flag for allocations in pmap modules.

branches: 1.19.2;
comment cleanup, shift around the inline stuff a bit,
rename VM_AMAP_PPREF (to UVM_AMAP_PPREF).

cleanup/reorg:
- break anon related functions out of uvm_amap.c and put them in their own
file (uvm_anon.c). includes break up uvm_anon_init into an amap and an
an anon init function
- ensure that only functions within the amap module access amap structure
fields (add macros to amap api as needed)

branches: 1.17.2;
be consistent with locking of amaps and anons when freeing them.

shift by PAGE_SHIFT instead of multiplying or dividing by PAGE_SIZE.

fix ppref botch. establish ppref at split time before we add the duplicate
reference.

Allocate vm_anon arrays from kernel_map, not via MALLOC(). Helps relieve
much of UVM's kmem_map usage.

Use the pool allocator (and the "nointr" pool page allocator) for
vm_amap structures.

Merge paddr_t changes into the main branch.

bzero->memset, bcopy->memcpy, bcmp->memcmp

branches: 1.10.2;
add a "<-done!" log

detect ending VA wrap-around in the chunking code of amap_copy.

fixes problem reported by Ken Nakata <kenn@synap.ne.jp> on the mac68k
where the stack amap chunking caused entry->end to wrap around to zero,
thus corrupting the map entry list and causing kmem_map to fill.

Remove inclusions of syscall (and syscall argument) related header files;
we don't need them here.

KNF.

- add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.

KNF

restore rcsids

fix typoes in locking.
use M_UVMAMAP instead of M_TEMP for malloc type.

RCS ID police.

branches: 1.1.1;
Initial revision

Fix various typos in comments.

Go back to freeing struct vm_anon one by one. There may have been an
advantage circa ~2008 but there isn't now.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

branches: 1.39.2;
Back out the amap allocation changes from earlier today - have seen a panic
with them. Retain the lock changes.

- Start trying to reduce the high cache miss rate observed around vm_amap.
On _LP64, pad struct vm_amap to 128 bytes and use the additional space to
hold the arrays for tiny amaps which are common. Carefully size the array
allocations to avoid false sharing, and for smaller amaps try to share
allocated cache lines.

- Eliminate most contention due to amap_list: maintain the list in the pool
cache constructor / destructor like we do for struct file. Cache the
mutexes we allocate here.

- Don't do PR_WAITOK mutex allocations when NOWAIT has been specified.

branches: 1.37.2; 1.37.54;
Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.36.2;
Replace "malloc" in comments, remove unnecessary header inclusions.

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

branches: 1.34.16; 1.34.22; 1.34.24;
"sparce" -> "sparse" + commas after "large", prior to "sparse"

branches: 1.33.26; 1.33.30; 1.33.36; 1.33.38;
Merge unobtrusive locking changes from the vmlocking branch.

branches: 1.32.4; 1.32.12;
Fix lossage from boolean_t -> bool and updated x86 bus_dma.

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.30.10;
make amap use kmem_alloc, rather than malloc.
(ie. make it use kernel_map, rather than kmem_map.)
kmem_map is more restricted than kernel_map,
and there's no point for amap to use it.

branches: 1.29.2; 1.29.10;
- amap_copy: take a "flags" argument instead of booleans.
- add AMAP_COPY_NOMERGE flag, and use it for uvm_map_extract.
PR/32806 from Julio M. Merino Vidal.

remove the following options. no objections on tech-kern@.

UVM_PAGER_INLINE
UVM_AMAP_INLINE
UVM_PAGE_INLINE
UVM_MAP_INLINE

branches: 1.27.2; 1.27.4; 1.27.6;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

branches: 1.25.2;
allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

Initialise (most) pools from a link set instead of explicit calls
to pool_init. Untouched pools are ones that either in arch-specific
code, or aren't initialiased during initial system startup.

Convert struct session, ucred and lockf to pools.

Nuke __P().

branches: 1.22.2;
Add extensible malloc types, adapted from FreeBSD. This turns
malloc types into a structure, a pointer to which is passed around,
instead of an int constant. Allow the limit to be adjusted when the
malloc type is defined, or with a function call, as suggested by
Jonathan Stone.

Properly set page references counts at the start of the newly
allocated ppref data to zero in the case of an amap that has empty
space at the front.

Don't set anything in the ppref array if "len" is zero.

Many thanks to Sami Kantoluoto for providing gdb access to a machine
that would reliably crash with problems related to the above, and to
Stephan Thesing for corroborating that the patch properly addressed
the problem.

Note that the ar_pageoff (and related variables) types must be changed
soon. The use of "int" here is not theoretically sufficient.

Change uvm_km_kmemalloc() to accept flag UVM_KMF_NOWAIT and pass it to
uvm_map(). Change uvm_map() to honnor UVM_KMF_NOWAIT. For this, change
amap_extend() to take a flags parameter instead of just boolean for
direction, and introduce AMAP_EXTEND_FORWARDS and AMAP_EXTEND_NOWAIT flags
(AMAP_EXTEND_BACKWARDS is still defined as 0x0, to keep the code easier to
read).
Add a flag parameter to uvm_mapent_alloc().
This solves a problem a pool_get(PR_NOWAIT) could trigger a pool_get(PR_WAITOK)
in uvm_mapent_alloc().
Thanks to Chuck Silvers, enami tsugutomo, Andrew Brown and Jason R Thorpe
for feedback.

Implement backwards extension of amaps. There are three cases to deal
with:

Case #1 -- adjust offset: The slot offset in the aref can be
decremented to cover the required size addition.

Case #2 -- move pages and adjust offset: The slot offset is not large
enough, but the amap contains enough inactive space *after* the mapped
pages to make up the difference, so active slots are slid to the "end"
of the amap, and the slot offset is, again, adjusted to cover the
required size addition. This optimizes for hitting case #1 again on
the next small extension.

Case #3 -- reallocate, move pages, and adjust offset: There is not
enough inactive space in the amap, so the arrays are reallocated, and
the active pages are copied again to the "end" of the amap, and the
slot offset is adjusted to cover the required size. This also
optimizes for hitting case #1 on the next backwards extension.

This provides the missing piece in the "forward extension of
vm_map_entries" logic, so the merge failure counters have been
removed.

Not many applications will make any use of this at this time (except
for jvms and perhaps gcc3), but a "top-down" memory allocator will use
it extensively.

add a new km flag UVM_KMF_CANFAIL, which causes uvm_km_kmemalloc() to
return failure if swap is full and there are no free physical pages.
have malloc() use this flag if M_CANFAIL is passed to it.
use M_CANFAIL to allow amap_extend() to fail when memory is scarce.
this should prevent most of the remaining hangs in low-memory situations.

branches: 1.17.2; 1.17.10; 1.17.16;
replace vm_map{,_entry}_t with struct vm_map{,_entry} *.

replace {simple_,}lock{_data,}_t with struct {simple,}lock {,*}.

remove trailing whitespace.

branches: 1.14.2;
clean up DIAGNOSTIC checks, use KASSERT().

lots of cleanup:
use queue.h macros and KASSERT().
address amap offsets in pages instead of bytes.
make amap_ref() and amap_unref() take an amap, offset and length
instead of a vm_map_entry_t.
improve whitespace and comments.

branches: 1.12.2;
Don't bother returning the "slot" number from amap_add():
* Nothing currently uses this return value.
* It's arguably an abstraction violation.

Fix amap_unadd()'s API to be consistent w/ amap_add()'s: rather than
take a vm_amap * and a slot number, take a vm_aref * and an offset.

It's now actually possible to use amap_unadd() to remove an anon from
an amap.

Protect prototypes, certain macros, and inlines from userland.

branches: 1.10.4;
comment cleanup, shift around the inline stuff a bit,
rename VM_AMAP_PPREF (to UVM_AMAP_PPREF).

cleanup/reorg:
- break anon related functions out of uvm_amap.c and put them in their own
file (uvm_anon.c). includes break up uvm_anon_init into an amap and an
an anon init function
- ensure that only functions within the amap module access amap structure
fields (add macros to amap api as needed)

branches: 1.8.2;
shift by PAGE_SHIFT instead of multiplying or dividing by PAGE_SIZE.

Merge paddr_t changes into the main branch.

branches: 1.6.2;
KNF.

- add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.

add/cleanup multiple inclusion protection.

restore rcsids

RCS ID police.

branches: 1.1.1;
Initial revision

remove the following options. no objections on tech-kern@.

UVM_PAGER_INLINE
UVM_AMAP_INLINE
UVM_PAGE_INLINE
UVM_MAP_INLINE

branches: 1.24.2; 1.24.4; 1.24.6;
merge ktrace-lwp.

branches: 1.23.2;
Use ANSI function decls.

allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

add some locking assertions.

branches: 1.20.2; 1.20.10; 1.20.12;
Properly set page references counts at the start of the newly
allocated ppref data to zero in the case of an amap that has empty
space at the front.

Don't set anything in the ppref array if "len" is zero.

Many thanks to Sami Kantoluoto for providing gdb access to a machine
that would reliably crash with problems related to the above, and to
Stephan Thesing for corroborating that the patch properly addressed
the problem.

Note that the ar_pageoff (and related variables) types must be changed
soon. The use of "int" here is not theoretically sufficient.

Reorder things so that with multiple inclusion protection that optional
definitions are outside the protection checks.

In amap_ref, only increment the amap's refcnt after we have established
the ppref array. Otherwise, the newly ref'ed pages will be doubly
counted and thus never freed because the pprefcnt can't fall to 0.

branches: 1.17.2; 1.17.14; 1.17.16;
remove trailing whitespace.

Remove a comment which is no longer true. From Artur Grabowski.

branches: 1.15.2;
lots of cleanup:
use queue.h macros and KASSERT().
address amap offsets in pages instead of bytes.
make amap_ref() and amap_unref() take an amap, offset and length
instead of a vm_map_entry_t.
improve whitespace and comments.

branches: 1.14.2; 1.14.12;
eliminate the PMAP_NEW option by making it required for all ports.
ports which previously had no support for PMAP_NEW now implement
the pmap_k* interfaces as wrappers around the non-k versions.

Don't bother returning the "slot" number from amap_add():
* Nothing currently uses this return value.
* It's arguably an abstraction violation.

Fix amap_unadd()'s API to be consistent w/ amap_add()'s: rather than
take a vm_amap * and a slot number, take a vm_aref * and an offset.

It's now actually possible to use amap_unadd() to remove an anon from
an amap.

branches: 1.12.4;
remove now >1 year old pre-release message.

comment cleanup, shift around the inline stuff a bit,
rename VM_AMAP_PPREF (to UVM_AMAP_PPREF).

cleanup/reorg:
- break anon related functions out of uvm_amap.c and put them in their own
file (uvm_anon.c). includes break up uvm_anon_init into an amap and an
an anon init function
- ensure that only functions within the amap module access amap structure
fields (add macros to amap api as needed)

branches: 1.9.2;
shift by PAGE_SHIFT instead of multiplying or dividing by PAGE_SIZE.

Merge paddr_t changes into the main branch.

bzero->memset, bcopy->memcpy, bcmp->memcmp

branches: 1.6.2;
- add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.

add/cleanup multiple inclusion protection.

KNF

restore rcsids

RCS ID police.

branches: 1.1.1;
Initial revision

Handle PG_PAGEOUT in uvm_anon_release() too.

Consistently use UVMHIST(__func__)

Convert UVMHIST_{CALLED,LOG} into UVMHIST_CALLARGS

Trailing whitespace

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Go back to freeing struct vm_anon one by one. There may have been an
advantage circa ~2008 but there isn't now.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

0x%#x --> %#x for non-external codes.
Also, stop mixing up 0x%x and %#x in single files as far as possible.

Use rw_lock_op().

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.70.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

fix the build for when UVMHIST is enabled.

Add missing #include <sys/atomic.h>

Free pages in batch instead of taking uvm_pageqlock for each one.

- Adjust uvmexp.swpgonly with atomics, and make uvm_swap_data_lock static.
- A bit more __cacheline_aligned on mutexes.

branches: 1.64.4; 1.64.8;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

branches: 1.63.22;
Mark a diagnostic-only variable

branches: 1.62.2; 1.62.12; 1.62.16;
uvm_anon_release:
- don't forget to call uvm_anon_dispose.
- simplify code a little.

uvm_anon_freelst:
- clear an_link/an_ref when deferring anon disposal. otherwise others can
see bogus an_ref.
- fix the code to remove anon from the list.

uvm_anon_freelst: do not free PG_RELEASED pages (change uvm_anon_dispose()
to indicate them with a return value).

- Rework uvm_anfree() into uvm_anon_freelst(), which always drops the lock.
- Free anons in uvm_anon_freelst() without lock held.
- Mechanic sync to unused loaning code.

reduce the number of atomic ops in common cases. it's exceptional for
anons to remain longer than amap.

Fix uvmplock regression - a lock against oneself case in amap_swap_off().
Happens since amap is NULL in uvmfault_anonget(), so uvmfault_unlockall()
keeps anon locked, when it should unlock it.

uvm_anon_release: fix a locking error after the rmind-uvmplock merge

Improve comments on uvm_anon.c, tidy up slightly.
No functional changes.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.53.2;
Replace "malloc" in comments, remove unnecessary header inclusions.

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

branches: 1.51.28; 1.51.32; 1.51.38; 1.51.40;
push pmap_clear_reference calls into pdpolicy code, where reference bits
actually matter.

Merge vmlocking2 to head.

Specify PR_LARGECACHE for anon_cache (which is insanely busy).

branches: 1.48.2; 1.48.6;
g/c unused uvm_anon_pool.

Merge from vmlocking:

- pool_cache changes.
- Debugger/procfs locking fixes.
- Other minor changes.

branches: 1.46.2; 1.46.4;
Remove LOCK_ASSERT(!simple_lock_held(&foo));

branches: 1.45.4; 1.45.6; 1.45.8; 1.45.10;
Merge unobtrusive locking changes from the vmlocking branch.

branches: 1.44.8;
Pass an ipl argument to pool_init/POOL_INIT to be used when initializing
the pool's lock.

branches: 1.43.4;
TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.41.4;
remove some __unused from function parameters.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.39.2;
merge yamt-pdpolicy branch.
- separate page replacement policy from the rest of kernel
- implement an alternative replacement policy

branches: 1.38.8; 1.38.20;
merge ktrace-lwp.

wrap swap related code by #ifdef VMSWAP. always #define VMSWAP for now.

revert "defflag VMSWAP" changes for now.
there seems to be far more people who don't want to edit
their kernel config files than i thought.

defflag VMSWAP.

branches: 1.34.2;
Use ANSI function decls.

allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

merge yamt-km branch.
- don't use managed mappings/backing objects for wired memory allocations.
save some resources like pv_entry. also fix (most of) PR/27030.
- simplify kernel memory management API.
- simplify pmap bootstrap of some ports.
- some related cleanups.

branches: 1.31.4; 1.31.6;
uvm_pagefree: when orphaning an A->K loaned page,
- decrement uvmexp.anonpages as it's no longer an anon page.
- null out anon->u.an_page as the anon no longer own the page.
uvm_anfree: add related assertions.

uvm_anfree: remove a comment which is no longer true.

fix a amap_wirerange deadlock problem by re-introducing
PG_RELEASED for anon pages. PR/23171 from Christian Limpach.
for details, see discussion filed in the PR database.

uvm_anon_release: a new function to free anon-owned PG_RELEASED page.
uvm_anfree: we can't wait for the page here because the caller might hold
amap lock. instead, just mark the page as PG_RELEASED.
who unbusy the page should check the PG_RELEASED.
uvm_aio_aiodone: uvm_anon_release() instead of uvm_page_unbusy()
if appropriate.
uvmfault_anonget: check PG_RELEASED.

branches: 1.28.2;
Nuke __P().

fix lock initialization in uvm_anon_add(). from PR 23831.

When retiring a swap device with marked bad blocks on it we should update
the `# swap page in use' and `# swap page only' counters. However, at the
time of swap device removal we can no longer figure out how many of the
bad swap pages are actually also `swap only' pages.

So, on swap I/O errors arrange things to not include the bad swap pages in
the `swpgonly' counter as follows: uvm_swap_markbad() decrements `swpgonly'
by the number of bad pages, and the various VM object deallocation routines
do not decrement `swpgonly' for swap slots marked as SWSLOT_BAD.

uao_pagein_page() & anon_pagein():
* return failure if the page cannot be retrieved.
* wakeup any waiters when releasing a page after successful page in.

Only deactivate pages if their wired count is zero.

Make sure to call uvm_swap_free() and uvm_swap_markbad() with valid (i.e.
positive) slot numbers.

branches: 1.22.6;
add missing anon lock around call to uvm_anon_lockloanpg().

branches: 1.21.4;
add RCSIDs, and in some cases, slightly cleanup #include order

several changes prompted by loaning problems:
- fix the loaned case in uvm_pagefree().
- redo uvmexp.swpgonly accounting to work with page loaning.
add an assertion before each place we adjust uvmexp.swpgonly.
- fix uvm_km_pgremove() to always free any swap space associated with
the range being removed.
- get rid of UVM_LOAN_WIRED flag. instead, we just make sure that
pages loaned to the kernel are never on the page queues.
this allows us to assert that pages are not loaned and wired
at the same time.
- add yet more assertions.

branches: 1.19.2;
add some missing spinlocks.

a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.17.2; 1.17.4;
remove trailing whitespace.

eliminate the VM_PAGER_* error codes in favor of the traditional E* codes.
the mapping is:

VM_PAGER_OK 0
VM_PAGER_BAD <unused>
VM_PAGER_FAIL <unused>
VM_PAGER_PEND 0 (see below)
VM_PAGER_ERROR EIO
VM_PAGER_AGAIN EAGAIN
VM_PAGER_UNLOCK EBUSY
VM_PAGER_REFAULT ERESTART

for async i/o requests, it used to be possible for the request to
be convert to sync, and the pager would return VM_PAGER_OK or VM_PAGER_PEND
to indicate whether the caller should perform post-i/o cleanup.
this is no longer allowed; pagers must now return 0 to indicate that
the async i/o was successfully started, and the caller never needs to
worry about doing the post-i/o cleanup.

branches: 1.15.2;
clean up DIAGNOSTIC checks, use KASSERT().

Page scanner improvements, behavior is actually a bit more like
Mach VM's now. Specific changes:
- Pages now need not have all of their mappings removed before being
put on the inactive list. They only need to have the "referenced"
attribute cleared. This makes putting pages onto the inactive list
much more efficient. In order to eliminate redundant clearings of
"refrenced", callers of uvm_pagedeactivate() must now do this
themselves.
- When checking the "modified" attribute for a page (for clearing
PG_CLEAN), make sure to only do it if PG_CLEAN is currently set on
the page (saves a potentially expensive pmap operation).
- When scanning the inactive list, if a page is referenced, reactivate
it (this part was actually added in uvm_pdaemon.c,v 1.27). This
now works properly now that pages on the inactive list are allowed to
have mappings.
- When scanning the inactive list and considering a page for freeing,
remove all mappings, and then check the "modified" attribute if the
page is marked PG_CLEAN.
- When scanning the active list, if the page was referenced since its
last sweep by the scanner, don't deactivate it. (This part was
actually added in uvm_pdaemon.c,v 1.28.)

These changes greatly improve interactive performance during
moderate to high memory and I/O load.

Change uvm_analloc() to return a locked anon, update all callers,
and fix an anon locking protocol error in uvm_loanzero().

Sprinkle some assertions:
amap_free(): Assert that the amap is locked.
amap_share_protect(): Assert that the amap is locked.
amap_wipeout(): Assert that the amap is locked.
uvm_anfree(): Assert that the anon has a reference count of 0 and is
not locked.
uvm_anon_lockloanpg(): Assert that the anon is locked.
anon_pagein(): Assert that the anon is locked.
uvmfault_anonget(): Assert that the anon is locked.
uvm_pagealloc_strat(): Assert that the uobj or the anon is locked

And fix the problems these have uncovered:
amap_cow_now(): Lock the new anon after allocating it, and unref and
unlock it (rather than lock!) before freeing it in case
of an error condition. This should fix a problem reported
by Dan Carosone using cdrecord on an i386 MP kernel.
uvm_fault(): Case1B -- Lock the new anon afer allocating it, and unlock
it later when we unlock the old anon.
Case2 -- Lock the new anon after allocating it, and unlock
it later by passing it to uvmfault_unlockall() (we set anon
to NULL if we're not doing a promote fault).

when we fail to allocate anons to represent new swap space,
just return an error rather than panicing.

lots of cleanup:
use queue.h macros and KASSERT().
address amap offsets in pages instead of bytes.
make amap_ref() and amap_unref() take an amap, offset and length
instead of a vm_map_entry_t.
improve whitespace and comments.

Do something sane with a DIAGNOSTIC condition in an non-DIAGNOSTIC
kernel.

Correct a comment about locking wrt. uvmfault_anonget().

remove include of <vm/vm.h>

remove/move more mach vm header files:

<vm/pglist.h> -> <uvm/uvm_pglist.h>
<vm/vm_inherit.h> -> <uvm/uvm_inherit.h>
<vm/vm_kern.h> -> into <uvm/uvm_extern.h>
<vm/vm_object.h> -> nothing
<vm/vm_pager.h> -> into <uvm/uvm_pager.h>

also includes a bunch of <vm/vm_page.h> include removals (due to redudancy
with <vm/vm.h>), and a scattering of other similar headers.

branches: 1.5.4;
add support for ``swapctl -d'' (removing swap space).
improve handling of i/o errors in swap space.

reviewed by: Chuck Cranor

branches: 1.4.2;
eliminate the PMAP_NEW option by making it required for all ports.
ports which previously had no support for PMAP_NEW now implement
the pmap_k* interfaces as wrappers around the non-k versions.

In uvm_anon_init() and uvm_anon_add(), initialize the ref count lock.

add uvmexp.swpgonly and use it to detect out-of-swap conditions.

cleanup/reorg:
- break anon related functions out of uvm_amap.c and put them in their own
file (uvm_anon.c). includes break up uvm_anon_init into an amap and an
an anon init function
- ensure that only functions within the amap module access amap structure
fields (add macros to amap api as needed)

Go back to freeing struct vm_anon one by one. There may have been an
advantage circa ~2008 but there isn't now.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

branches: 1.30.54; 1.30.60;
- Rework uvm_anfree() into uvm_anon_freelst(), which always drops the lock.
- Free anons in uvm_anon_freelst() without lock held.
- Mechanic sync to unused loaning code.

Fix uvmplock regression - a lock against oneself case in amap_swap_off().
Happens since amap is NULL in uvmfault_anonget(), so uvmfault_unlockall()
keeps anon locked, when it should unlock it.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.27.2;
udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

branches: 1.26.4; 1.26.6; 1.26.8;
change the order of members of vm_anon for better packing.

branches: 1.25.10; 1.25.24;
Merge vmlocking2 to head.

branches: 1.24.4; 1.24.18; 1.24.24; 1.24.26; 1.24.30;
Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.23.26;
merge ktrace-lwp.

make VMSWAP optional again.

wrap swap related code by #ifdef VMSWAP. always #define VMSWAP for now.

revert "defflag VMSWAP" changes for now.
there seems to be far more people who don't want to edit
their kernel config files than i thought.

defflag VMSWAP.

branches: 1.18.2;
allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

fix a amap_wirerange deadlock problem by re-introducing
PG_RELEASED for anon pages. PR/23171 from Christian Limpach.
for details, see discussion filed in the PR database.

uvm_anon_release: a new function to free anon-owned PG_RELEASED page.
uvm_anfree: we can't wait for the page here because the caller might hold
amap lock. instead, just mark the page as PG_RELEASED.
who unbusy the page should check the PG_RELEASED.
uvm_aio_aiodone: uvm_anon_release() instead of uvm_page_unbusy()
if appropriate.
uvmfault_anonget: check PG_RELEASED.

branches: 1.16.2;
Nuke __P().

branches: 1.15.22;
replace {simple_,}lock{_data,}_t with struct {simple,}lock {,*}.

remove trailing whitespace.

branches: 1.13.2;
when we fail to allocate anons to represent new swap space,
just return an error rather than panicing.

branches: 1.12.4;
add support for ``swapctl -d'' (removing swap space).
improve handling of i/o errors in swap space.

reviewed by: Chuck Cranor

branches: 1.11.2;
Protect prototypes, certain macros, and inlines from userland.

branches: 1.10.4;
add uvmexp.swpgonly and use it to detect out-of-swap conditions.

cleanup/reorg:
- break anon related functions out of uvm_amap.c and put them in their own
file (uvm_anon.c). includes break up uvm_anon_init into an amap and an
an anon init function
- ensure that only functions within the amap module access amap structure
fields (add macros to amap api as needed)

update outdated an_swslot comments

KNF.

Merge with Lite2 + local changes

add/cleanup multiple inclusion protection.

KNF.

restore rcsids

RCS ID police.

branches: 1.1.1;
Initial revision

uvm: Eliminate __HAVE_ATOMIC_AS_MEMBAR conditionals.

Discussed on tech-kern:
https://mail-index.netbsd.org/tech-kern/2023/02/23/msg028729.html

fix various typos in comments, documentation and messages.

sys: Use membar_release/acquire around reference drop.

This just goes through my recent reference count membar audit and
changes membar_exit to membar_release and membar_enter to
membar_acquire -- this should make everything cheaper on most CPUs
without hurting correctness, because membar_acquire is generally
cheaper than membar_enter.

sys: Membar audit around reference count releases.

If two threads are using an object that is freed when the reference
count goes to zero, we need to ensure that all memory operations
related to the object happen before freeing the object.

Using an atomic_dec_uint_nv(&refcnt) == 0 ensures that only one
thread takes responsibility for freeing, but it's not enough to
ensure that the other thread's memory operations happen before the
freeing.

Consider:

Thread A Thread B
obj->foo = 42; obj->baz = 73;
mumble(&obj->bar); grumble(&obj->quux);
/* membar_exit(); */ /* membar_exit(); */
atomic_dec -- not last atomic_dec -- last
/* membar_enter(); */
KASSERT(invariant(obj->foo,
obj->bar));
free_stuff(obj);

The memory barriers ensure that

obj->foo = 42;
mumble(&obj->bar);

in thread A happens before

KASSERT(invariant(obj->foo, obj->bar));
free_stuff(obj);

in thread B. Without them, this ordering is not guaranteed.

So in general it is necessary to do

membar_exit();
if (atomic_dec_uint_nv(&obj->refcnt) != 0)
return;
membar_enter();

to release a reference, for the `last one out hit the lights' style
of reference counting. (This is in contrast to the style where one
thread blocks new references and then waits under a lock for existing
ones to drain with a condvar -- no membar needed thanks to mutex(9).)

I searched for atomic_dec to find all these. Obviously we ought to
have a better abstraction for this because there's so much copypasta.
This is a stop-gap measure to fix actual bugs until we have that. It
would be nice if an abstraction could gracefully handle the different
styles of reference counting in use -- some years ago I drafted an
API for this, but making it cover everything got a little out of hand
(particularly with struct vnode::v_usecount) and I ended up setting
it aside to work on psref/localcount instead for better scalability.

I got bored of adding #ifdef __HAVE_ATOMIC_AS_MEMBAR everywhere, so I
only put it on things that look performance-critical on 5sec review.
We should really adopt membar_enter_preatomic/membar_exit_postatomic
or something (except they are applicable only to atomic r/m/w, not to
atomic_load/store_*, making the naming annoying) and get rid of all
the ifdefs.

Consistently use %#jx instead of 0x%jx or just %jx in UVMHIST_LOG formats

In uvmpd_tryownerlock(), if the initial try-lock of the owner lock fails
then rather than do more try-locks and eventually sleep for a tick,
take a hold on the current owner's lock, drop the page interlock,
and acquire the lock that we took the hold on in a blocking fashion.
After we get the lock, check if the lock that we acquired is still
the lock for the owner of the page that we're interested in.
If the owner hasn't changed then can proceed with this page,
otherwise we will skip this page and move on to a different page.
This dramatically reduces the amount of time that the pagedaemon
sleeps trying to get locks, since even 1 tick is an eternity to sleep
in this context and it was easy to trigger that case in practice,
and with this new method the pagedaemon only very rarely actually blocks
to acquire the lock that it wants since the object locks are adaptive,
and when the pagedaemon does block then the amount of time it spends
sleeping will be generally be much less than 1 tick.

branches: 1.151.2;
in uao_get(), if we unlock the uobj to read a page from swap,
we must clear the cached page array because it is now stale.
also add a missing call to uvm_page_array_fini() if the I/O fails.
fixes PR 55493.

Remove trailing \n from UVMHIST_LOG() format strings.

Consistently use UVMHIST(__func__)

Convert UVMHIST_{CALLED,LOG} into UVMHIST_CALLARGS

Trailing whitespace

uao_get(): in the PGO_SYNCIO case use uvm_page_array and simplify control
flow a little bit.

- Alter the convention for uvm_page_array slightly, so the basic search
parameters can't change part way through a search: move the "uobj" and
"flags" arguments over to uvm_page_array_init() and store those with the
array.

- With that, detect when it's not possible to find any more pages in the
tree with the given search parameters, and avoid repeated tree lookups if
the caller loops over uvm_page_array_fill_and_peek().

PR kern/55300: ubciomove triggers page not dirty assertion

If overwriting an existing page, mark it dirty since there may be no
managed mapping to track the modification.

uao_get(): handle PGO_OVERWRITE.

Suppress GCC warnings and fix a UVMHIST_LOG() statement.

Kernels ALL/amd64 and ALL/i386 and port sparc64 build again.

PR kern/32166: pgo_get protocol is ambiguous
Also problems with tmpfs+nfs noted by hannken@.

Don't pass PGO_ALLPAGES to pgo_get, and ignore PGO_DONTCARE in the
!PGO_LOCKED case. In uao_get() have uvm_pagealloc() take care of page
zeroing and release busy pages on error.

Start trying to reduce cache misses on vm_page during fault processing.

- Make PGO_LOCKED getpages imply PGO_NOBUSY and remove the latter. Mark
pages busy only when there's actually I/O to do.

- When doing COW on a uvm_object, don't mess with neighbouring pages. In
all likelyhood they're already entered.

- Don't mess with neighbouring VAs that have existing mappings as replacing
those mappings with same can be quite costly.

- Don't enqueue pages for neighbour faults unless not enqueued already, and
don't activate centre pages unless uvmpdpol says its useful.

Also:

- Make PGO_LOCKED getpages on UAOs work more like vnodes: do gang lookup in
the radix tree, and don't allocate new pages.

- Fix many assertion failures around faults/loans with tmpfs.

PR kern/55268: tmpfs is slow

uao_get(): in the PGO_LOCKED case, we're okay to allocate a new page as long
as the caller holds a write lock. PGO_NOBUSY doesn't put a stop to that.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Tweak the March 14th change to make page waits interlocked by pg->interlock.
Remove unneeded changes and only deal with the PQ_WANTED flag, to exclude
possible bugs.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

0x%#x --> %#x for non-external codes.
Also, stop mixing up 0x%x and %#x in single files as far as possible.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.133.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

Avoid calling pmap_page_protect() while under uvm_pageqlock.

- Adjust uvmexp.swpgonly with atomics, and make uvm_swap_data_lock static.
- A bit more __cacheline_aligned on mutexes.

Avoid undefined behavior in uao_pagein_page(). Found by kUBSan. OK'd by
riastradh. I think this is a real bug on amd64 at least.

branches: 1.127.2;
allow tmpfs files to be larger than 4GB.

branches: 1.126.2;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

branches: 1.125.2;
add assertions that would have caught the recent audio mmap bugs.

PR kern/51371: fix misleading indentation

branches: 1.123.2;
to garnish, dust with _KERNEL_OPT

branches: 1.122.4;
Allow VM_NFREELIST in uao_set_pgfl, meaning any freelist is OK.

Add uao_set_pgfl to limit a uvm_aobj's pages to a specified freelist.

Brought up on tech-kern:

https://mail-index.netbsd.org/tech-kern/2014/05/20/msg017095.html

branches: 1.120.2;
Mark a diagnostic-only variable

branches: 1.119.2;
#include <sys/atomic.h>

- Manage anonymous UVM object reference count with atomic ops.
- Fix an old bug of possible lock against oneself (uao_detach_locked() is
called from uao_swap_off() with uao_list_lock acquired). Also removes
the try-lock dance in uao_swap_off(), since the lock order changes.

- Describe uvm_aobj and the lock order.
- Remove unnecessary uao_dropswap_range1() wrapper.
- KNF. Sprinkle some __cacheline_aligned.

branches: 1.116.2; 1.116.8; 1.116.12;
Allocate color appropriate pages.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.114.2;
Replace "malloc" in comments, remove unnecessary header inclusions.

Replace uvm_aobj_cache with kmem(9).

udpate license clauses on chuck^2 code to match the new-style BSD licenses.
based on diff that rmind@ sent me (and confirmed with chs@ via email).

no functional change with this commit.

Remove nop code; the code is moved to uao_dropswap_range1() when it is
introduced in rev. 1.75.

branches: 1.110.2; 1.110.4;
Add vm page flag PG_MARKER and use it to tag dummy marker pages
in genfs_do_putpages() and uao_put().
Use 'v_uobj.uo_npages' to check for an empty memq.
Put some assertions where these marker pages may not appear.

Ok: YAMAMOTO Takashi <yamt@netbsd.org>

uvm_fault_{upper,lower}_done: move drop-swap outside the page-queues lock.
Assert for object lock being held (or ref count 0) in uao_set_swslot().

branches: 1.108.2; 1.108.4;
Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Wipe out the last vestiges of POOL_INIT with one swift stroke. In
most cases, use a proper constructor. For proplib, give a local
equivalent of POOL_INIT for the kernel object implementation. This
way the code structure can be preserved, and a local link set is
not hazardous anyway (unless proplib is split to several modules,
but that'll be the day).

tested by booting a kernel in qemu and compile-testing i386/ALL

make some functions static.

branches: 1.105.2;
- g/c stale function prototypes.
- rename UVM_PAGE_HASH_PENALTY to UVM_PAGE_TREE_PENALTY.

branches: 1.104.2; 1.104.10;
- Initialize pool subsystem and kmem(9) earlier, when UVM is up enough.
- Remove uao_hashinit() workaround used for anon-objects.
- Replace malloc with kmem.

OK by <yamt>.

branches: 1.103.2;
Use pool_cache.

branches: 1.102.2;
vm_page: put TAILQ_ENTRY into a union with LIST_ENTRY, so we can use both.

uao_reference, uao_detach: we don't do reference counting on kernel objects,
so don't lock them needlessly.

branches: 1.100.2;
- Convert hashinit() to use kmem_alloc(). The hash tables can be large
and it's better to not have them in kmem_map.
- Convert a couple of minor items along the way to kmem_alloc().
- Fix some memory leaks.

branches: 1.99.2; 1.99.4;
Minor corrections to comments.

uao_put: fix a race with pageout.

branches: 1.97.2; 1.97.6;
push pmap_clear_reference calls into pdpolicy code, where reference bits
actually matter.

Merge vmlocking2 to head.

branches: 1.95.2; 1.95.6;
constify pagerops.

use designated initiaizers for uvm_pagerops.

branches: 1.93.2; 1.93.8; 1.93.10;
In uao_get(), drop object lock only after dropswap to avoid KASSERT panic.

Should fix tmpfs problem reported by riz on current-users. yamt ok.

branches: 1.92.4;
In order to pacify assertions, make uao_list_lock + uvm_swap_data_lock
spinlocks for the time being.

Temporarily work around an assertion from mutex_enter.

Merge unobtrusive locking changes from the vmlocking branch.

branches: 1.89.2;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

branches: 1.88.2;
Pass an ipl argument to pool_init/POOL_INIT to be used when initializing
the pool's lock.