Home | History | Annotate | Download | only in uvm
History log of /src/sys/uvm/uvm_object.h
RevisionDateAuthorComments
 1.40  05-Feb-2024  andvar fix various typos in comments.
 1.39  14-Aug-2020  chs centralize calls from UVM to radixtree into a few functions.
in those functions, assert that the object lock is held in
the correct mode.
 1.38  14-Mar-2020  ad Make uvm_pagemarkdirty() responsible for putting vnodes onto the syncer
work list. Proposed on tech-kern@.
 1.37  23-Feb-2020  ad UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.
 1.36  15-Jan-2020  ad Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.
 1.35  15-Dec-2019  ad branches: 1.35.2;
Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.
 1.34  14-Dec-2019  ad Merge from yamt-pagecache: use radixtree for page lookup.

rbtree page lookup was introduced during the NetBSD 5.0 development cycle to
bypass lock contention problems with the (then) global page hash, and was a
temporary solution to allow us to make progress. radixtree is the intended
replacement.

Ok yamt@.
 1.33  14-Sep-2012  rmind branches: 1.33.38;
- Manage anonymous UVM object reference count with atomic ops.
- Fix an old bug of possible lock against oneself (uao_detach_locked() is
called from uao_swap_off() with uao_list_lock acquired). Also removes
the try-lock dance in uao_swap_off(), since the lock order changes.
 1.32  28-Jan-2012  rmind branches: 1.32.2; 1.32.6;
Describe UVM object and explain lock sharing a little.
 1.31  12-Jun-2011  rmind branches: 1.31.2; 1.31.6;
Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.
 1.30  02-Feb-2011  chuck branches: 1.30.2;
udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.
 1.29  06-Nov-2010  uebayasi branches: 1.29.2; 1.29.4;
Include uvm/uvm_pglist.h for struct pglist.
 1.28  25-Sep-2010  matt Rename rb.h to rbtree.h, as it is more appropriate (c.f. ptree.h). Also
helps find code that hasn't been updated to use the new rbtree API.
 1.27  24-Sep-2010  rmind Fixes/improvements to RB-tree implementation:
1. Fix inverted node order, so that negative value from comparison operator
would represent lower (left) node, and positive - higher (right) node.
2. Add an argument (i.e. "context"), passed to comparison operators.
3. Change rb_tree_insert_node() to return a node - either inserted one or
already existing one.
4. Amend the interface to manipulate the actual object, instead of the
rb_node (in a similar way as Patricia-tree interface does).
5. Update all RB-tree users accordingly.

XXX: Perhaps rename rb.h to rbtree.h, since cleaning-up..

1-3 address the PR/43488 by Jeremy Huddleston.

Passes RB-tree regression tests.
Reviewed by: matt@, christos@
 1.26  04-Jun-2008  ad branches: 1.26.18; 1.26.20;
Replace the global vm_page hash with a per vm_object rbtree.
Proposed on tech-kern@.
 1.25  02-Jun-2008  ad Use atomics to maintain v_usecount.
 1.24  02-Jan-2008  ad branches: 1.24.6; 1.24.8; 1.24.10; 1.24.12;
Merge vmlocking2 to head.
 1.23  01-Dec-2007  yamt branches: 1.23.2; 1.23.6;
constify pagerops.
 1.22  12-Oct-2006  yamt branches: 1.22.8; 1.22.22; 1.22.24; 1.22.30;
move some knowledge about vnode into uvm_vnode.c.
 1.21  11-Dec-2005  christos branches: 1.21.20; 1.21.22;
merge ktrace-lwp.
 1.20  23-Jul-2005  yamt update file timestamps for nfsd loaned-read and mmap.
PR/25279. discussed on tech-kern@.
 1.19  17-Jul-2005  yamt ensure that vnodes with dirty pages are always on syncer's queue.

- genfs_putpages: wait for i/o completion of PG_RELEASED/PG_PAGEOUT pages by
setting "wasclean" false when encountering them.
suggested by Stephan Uphoff in PR/24596 (1).

- genfs_putpages: write protect pages when cleaning out, if
we're going to take the vnode off the syncer's queue.
uvm_fault: don't write-map pages unless its vnode is already on
the syncer's queue.

fix PR/24596 (3) but in the different way from the suggested fix.
(to keep our current behaviour, ie. not to require explicit msync.
discussed on tech-kern@.)

- genfs_putpages: don't mistakenly take a vnode off the queue
by introducing a generation number in genfs_node.
genfs_getpages: increment the generation number.
suggested by Stephan Uphoff in PR/24596 (2).

- add some assertions.
 1.18  06-Jun-2005  yamt branches: 1.18.2;
introduce a macro to initialize uvm_object and use it.
 1.17  29-Nov-2003  yamt branches: 1.17.14;
mincore: don't treat an aobj as a device mapping.
 1.16  20-Jun-2002  chs branches: 1.16.6;
count aobj pages (most notably kernel stack pages) as anon pages
for memory usage-balancing purposes.
 1.15  15-May-2002  matt branches: 1.15.2; 1.15.4;
When core dumping a process, don't dump maps backed up by the device pager.
(move the pagerops externs to uvm_object.h and out the C files).
 1.14  30-Oct-2001  thorpej - Add a new vnode flag VEXECMAP, which indicates that a vnode has
executable mappings. Stop overloading VTEXT for this purpose (VTEXT
also has another meaning).
- Rename vn_marktext() to vn_markexec(), and use it when executable
mappings of a vnode are established.
- In places where we want to set VTEXT, set it in v_flag directly, rather
than making a function call to do this (it no longer makes sense to
use a function call, since we no longer overload VTEXT with VEXECMAP's
meaning).

VEXECMAP suggested by Chuq Silvers.
 1.13  15-Sep-2001  chs branches: 1.13.2;
a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.
 1.12  26-May-2001  chs branches: 1.12.2; 1.12.4;
replace {simple_,}lock{_data,}_t with struct {simple,}lock {,*}.
 1.11  09-Mar-2001  chs add UBC memory-usage balancing. we track the number of pages in use for
each of the basic types (anonymous data, executable image, cached files)
and prevent the pagedaemon from reusing a given page if that would reduce
the count of that type of page below a sysctl-setable minimum threshold.
the thresholds are controlled via three new sysctl tunables:
vm.anonmin, vm.vnodemin, and vm.vtextmin. these tunables are the
percentages of pageable memory reserved for each usage, and we do not allow
the sum of the minimums to be more than 95% so that there's always some
memory that can be reused.
 1.10  28-Jan-2001  thorpej branches: 1.10.2;
Put the extern decl of uvm_vnodeops in uvm_object.h
 1.9  28-Jan-2001  thorpej Define a UVM_OBJ_IS_VNODE() macro to test if an object is a vnode.
 1.8  25-May-1999  thorpej branches: 1.8.2;
Define a new kernel object type, "intrsafe", which are used for objects
which can be used in an interrupt context. Use pmap_kenter*() and
pmap_kremove() only for mappings owned by these objects.

Fixes some locking protocol issues related to MP support, and eliminates
all of the pmap_enter vs. pmap_kremove inconsistencies.
 1.7  25-May-1999  thorpej Macro'ize the test for "object is a kernel object".
 1.6  25-Mar-1999  mrg branches: 1.6.4;
remove now >1 year old pre-release message.
 1.5  09-Mar-1998  mrg KNF.
 1.4  10-Feb-1998  perry add/cleanup multiple inclusion protection.
 1.3  07-Feb-1998  mrg restore rcsids
 1.2  06-Feb-1998  thorpej RCS ID police.
 1.1  05-Feb-1998  mrg branches: 1.1.1;
Initial revision
 1.1.1.1  05-Feb-1998  mrg initial import of the new virtual memory system, UVM, into -current.

UVM was written by chuck cranor <chuck@maria.wustl.edu>, with some
minor portions derived from the old Mach code. i provided some help
getting swap and paging working, and other bug fixes/ideas. chuck
silvers <chuq@chuq.com> also provided some other fixes.

this is the UVM kernel code portion.


this will be KNF'd shortly. :-)
 1.6.4.1  21-Jun-1999  thorpej Sync w/ -current.
 1.8.2.2  12-Mar-2001  bouyer Sync with HEAD.
 1.8.2.1  11-Feb-2001  bouyer Sync with HEAD.
 1.10.2.6  01-Aug-2002  nathanw Catch up to -current.
 1.10.2.5  20-Jun-2002  nathanw Catch up to -current.
 1.10.2.4  14-Nov-2001  nathanw Catch up to -current.
 1.10.2.3  21-Sep-2001  nathanw Catch up to -current.
 1.10.2.2  21-Jun-2001  nathanw Catch up to -current.
 1.10.2.1  09-Apr-2001  nathanw Catch up with -current.
 1.12.4.1  01-Oct-2001  fvdl Catch up with -current.
 1.12.2.3  06-Sep-2002  jdolecek sync kqueue branch with HEAD
 1.12.2.2  23-Jun-2002  jdolecek catch up with -current on kqueue branch
 1.12.2.1  10-Jan-2002  thorpej Sync kqueue branch with -current.
 1.13.2.1  12-Nov-2001  thorpej Sync the thorpej-mips-cache branch with -current.
 1.15.4.1  21-Jun-2002  lukem Pull up revision 1.16 (requested by chs in ticket #329):
count aobj pages (most notably kernel stack pages) as anon pages
for memory usage-balancing purposes.
 1.15.2.1  15-Jul-2002  gehenna catch up with -current.
 1.16.6.4  10-Nov-2005  skrll Sync with HEAD. Here we go again...
 1.16.6.3  21-Sep-2004  skrll Fix the sync with head I botched.
 1.16.6.2  18-Sep-2004  skrll Sync with HEAD.
 1.16.6.1  03-Aug-2004  skrll Sync with HEAD
 1.17.14.1  24-Aug-2005  riz Pull up following revision(s) (requested by yamt in ticket #688):
sys/miscfs/genfs/genfs_vnops.c: revision 1.98 via patch
sys/ufs/ffs/ffs_vfsops.c: revision 1.165
sys/ufs/lfs/lfs_extern.h: revision 1.69
sys/fs/filecorefs/filecore_vfsops.c: revision 1.20
sys/nfs/nfs_node.c: revision 1.80
sys/fs/smbfs/smbfs_node.c: revision 1.24
sys/fs/cd9660/cd9660_vfsops.c: revision 1.24
sys/fs/msdosfs/msdosfs_denode.c: revision 1.8
sys/miscfs/genfs/genfs_node.h: revision 1.6
sys/ufs/lfs/lfs_vfsops.c: revision 1.183
sys/ufs/ext2fs/ext2fs_vfsops.c: revision 1.86
sys/fs/adosfs/advfsops.c: revision 1.23
sys/fs/ntfs/ntfs_vfsops.c: revision 1.31
- constify genfs_ops.
- use member designators.

sys/miscfs/genfs/genfs_vnops.c: revision 1.99 via patch
genfs_getpages: don't forget to put the vnode onto the syncer's work que
ue
even in the case of PGO_LOCKED.

sys/uvm/uvm_bio.c: revision 1.40
sys/uvm/uvm_pager.h: revision 1.29
sys/miscfs/genfs/genfs_vnops.c: revision 1.100 via patch
sys/ufs/ufs/ufs_inode.c: revision 1.50
- introduce PGO_NOBLOCKALLOC and use it for ubc mapping
to prevent unnecessary block allocations in the case that
page size > block size.
- ufs_balloc_range: use VM_PROT_WRITE+PGO_NOBLOCKALLOC rather than
VM_PROT_READ.

sys/uvm/uvm_fault.c: revision 1.96
sys/miscfs/genfs/genfs_vnops.c: revision 1.101 via patch
sys/uvm/uvm_object.h: revision 1.19
sys/miscfs/genfs/genfs_node.h: revision 1.7
ensure that vnodes with dirty pages are always on syncer's queue.
- genfs_putpages: wait for i/o completion of PG_RELEASED/PG_PAGEOUT pages by
setting "wasclean" false when encountering them.
suggested by Stephan Uphoff in PR/24596 (1).
- genfs_putpages: write protect pages when cleaning out, if
we're going to take the vnode off the syncer's queue.
uvm_fault: don't write-map pages unless its vnode is already on
the syncer's queue.
fix PR/24596 (3) but in the different way from the suggested fix.
(to keep our current behaviour, ie. not to require explicit msync.
discussed on tech-kern@.)
- genfs_putpages: don't mistakenly take a vnode off the queue
by introducing a generation number in genfs_node.
genfs_getpages: increment the generation number.
suggested by Stephan Uphoff in PR/24596 (2).
- add some assertions.

sys/miscfs/genfs/genfs_vnops.c: revision 1.102 via patch
genfs_putpages: don't bother to clean the vnode unless VONWORKLST.

sys/ufs/ffs/ffs_vnops.c: revision 1.71
ffs_full_fsync: because VBLK/VCHR can be mmap'ed,
do VOP_PUTPAGES for them as well.

sys/uvm/uvm_fault.c: revision 1.97
uvm_fault: check a correct object in the case of layered filesystems.
fix PR/30811 from Jukka Salmi.

sys/uvm/uvm_object.h: revision 1.20
sys/ufs/ffs/ffs_vfsops.c: revision 1.167
sys/uvm/uvm_bio.c: revision 1.41
sys/ufs/ufs/ufs_vnops.c: revision 1.129
sys/uvm/uvm_mmap.c: revision 1.92
sys/uvm/uvm_fault.c: revision 1.98
sys/kern/vfs_subr.c: revision 1.252
sys/fs/msdosfs/denode.h: revision 1.5
sys/miscfs/genfs/genfs_vnops.c: revision 1.103 via patch
sys/fs/msdosfs/msdosfs_denode.c: revision 1.9
sys/sys/vnode.h: revision 1.141
sys/ufs/ufs/ufs_inode.c: revision 1.51
sys/ufs/ufs/ufs_extern.h: revision 1.45 via patch
sys/miscfs/genfs/genfs_node.h: revision 1.8
sys/ufs/lfs/lfs_vfsops.c: revision 1.184
sys/uvm/uvm_pager.h: revision 1.30
sys/ufs/ext2fs/ext2fs_vfsops.c: revision 1.87
update file timestamps for nfsd loaned-read and mmap.
PR/25279. discussed on tech-kern@.

sys/miscfs/genfs/genfs_vnops.c: revision 1.104 via patch
don't write-protect wired pages. pointed by Chuck Silvers.
for now, leave a vnode on the syncer's queue, as suggested by him.

sys/ufs/ffs/ffs_vnops.c: revision 1.72
revert VCHR part of ffs_vnops.c 1.71.
as VCHR uses the device pager, no point to call VOP_PUTPAGES here.
pointed by Chuck Silvers.
 1.18.2.4  21-Jan-2008  yamt sync with head
 1.18.2.3  07-Dec-2007  yamt sync with head
 1.18.2.2  30-Dec-2006  yamt sync with head.
 1.18.2.1  21-Jun-2006  yamt sync with head.
 1.21.22.1  22-Oct-2006  yamt sync with head
 1.21.20.1  18-Nov-2006  ad Sync with head.
 1.22.30.2  18-Feb-2008  mjf Sync with HEAD.
 1.22.30.1  08-Dec-2007  mjf Sync with HEAD.
 1.22.24.1  09-Jan-2008  matt sync with HEAD
 1.22.22.1  03-Dec-2007  joerg Sync with HEAD.
 1.22.8.2  21-Aug-2007  yamt destroy vmobjlock.
 1.22.8.1  13-Mar-2007  ad Pull in the initial set of changes for the vmlocking branch.
 1.23.6.1  02-Jan-2008  bouyer Sync with HEAD
 1.23.2.1  04-Dec-2007  ad Pull the vmlocking changes into a new branch.
 1.24.12.1  23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.24.10.2  09-Oct-2010  yamt sync with head
 1.24.10.1  04-May-2009  yamt sync with head.
 1.24.8.2  17-Jun-2008  yamt sync with head.
 1.24.8.1  04-Jun-2008  yamt sync with head
 1.24.6.1  05-Jun-2008  mjf Sync with HEAD.

Also fix build.
 1.26.20.3  05-Mar-2011  rmind sync with head
 1.26.20.2  26-Apr-2010  rmind Add ubc_purge() and purge/deassociate any related UBC entries during
object (usually, vnode) destruction. Since locking (and thus object)
is required to enter/remove mappings - object is not allowed anymore
to disappear with any UBC entries left.

From original patch by ad@ with some modifications.
 1.26.20.1  16-Mar-2010  rmind Change struct uvm_object::vmobjlock to be dynamically allocated with
mutex_obj_alloc(). It allows us to share the locks among UVM objects.
 1.26.18.2  22-Oct-2010  uebayasi Sync with HEAD (-D20101022).
 1.26.18.1  28-Apr-2010  uebayasi Don't expose uvm_page.h internal for usual uvm(9) users.
 1.29.4.1  08-Feb-2011  bouyer Sync with HEAD
 1.29.2.1  06-Jun-2011  jruoho Sync with HEAD.
 1.30.2.1  23-Jun-2011  cherry Catchup with rmind-uvmplock merge.
 1.31.6.1  18-Feb-2012  mrg merge to -current.
 1.31.2.5  30-Oct-2012  yamt sync with head
 1.31.2.4  01-Aug-2012  yamt - fix integrity sync.
putpages for integrity sync (fsync, msync with MS_SYNC, etc) should not
skip pages being written back by other threads.

- adapt to radix tree tag api changes.
 1.31.2.3  17-Apr-2012  yamt sync with head
 1.31.2.2  06-Nov-2011  yamt remove pg->listq and uobj->memq
 1.31.2.1  02-Nov-2011  yamt page cache related changes

- maintain object pages in radix tree rather than rb tree.
- reduce unnecessary page scan in putpages. esp. when an object has a ton of
pages cached but only a few of them are dirty.
- reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.
- fix nfs commit range tracking.
- fix nfs write clustering. XXX hack
 1.32.6.1  20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.32.2.1  22-Nov-2012  riz Pull up following revision(s) (requested by rmind in ticket #694):
sys/uvm/uvm_aobj.h: revision 1.22
sys/uvm/uvm_aobj.c: revision 1.117
sys/uvm/uvm_aobj.c: revision 1.118
sys/uvm/uvm_aobj.c: revision 1.119
sys/uvm/uvm_object.h: revision 1.33
- Describe uvm_aobj and the lock order.
- Remove unnecessary uao_dropswap_range1() wrapper.
- KNF. Sprinkle some __cacheline_aligned.
- Manage anonymous UVM object reference count with atomic ops.
- Fix an old bug of possible lock against oneself (uao_detach_locked() is
called from uao_swap_off() with uao_list_lock acquired). Also removes
the try-lock dance in uao_swap_off(), since the lock order changes.
 1.33.38.1  08-Apr-2020  martin Merge changes from current as of 20200406
 1.35.2.2  29-Feb-2020  ad Sync with head.
 1.35.2.1  17-Jan-2020  ad Sync with head.

RSS XML Feed