Home | History | Annotate | Download | only in include
History log of /src/sys/arch/i386/include/pmap.h
RevisionDateAuthorComments
 1.129  20-Aug-2022  riastradh x86: Move definition of struct pmap to pmap_private.h.

This makes pmap_resident_count and pmap_wired_count out-of-line
functions instead of inline. No functional change intended
otherwise.
 1.128  20-Aug-2022  riastradh x86: Split most of pmap.h into pmap_private.h or vmparam.h.

This way pmap.h only contains the MD definition of the MI pmap(9)
API, which loads of things in the kernel rely on, so changing x86
pmap internals no longer requires recompiling the entire kernel every
time.

Callers needing these internals must now use machine/pmap_private.h.
Note: This is not x86/pmap_private.h because it contains three parts:

1. CPU-specific (different for i386/amd64) definitions used by...

2. common definitions, including Xenisms like xpmap_ptetomach,
further used by...

3. more CPU-specific inlines for pmap_pte_* operations

So {amd64,i386}/pmap_private.h defines 1, includes x86/pmap_private.h
for 2, and then defines 3. Maybe we should split that out into a new
pmap_pte.h to reduce this trouble.

No functional change intended, other than that some .c files must
include machine/pmap_private.h when previously uvm/uvm_pmap.h
polluted the namespace with pmap internals.

Note: This migrates part of i386/pmap.h into i386/vmparam.h --
specifically the parts that are needed for several constants defined
in vmparam.h:

VM_MAXUSER_ADDRESS
VM_MAX_ADDRESS
VM_MAX_KERNEL_ADDRESS
VM_MIN_KERNEL_ADDRESS

Since i386 needs PDP_SIZE in vmparam.h, I added it there on amd64
too, just to keep things parallel.
 1.127  20-Aug-2022  riastradh x86: Move struct vm_page_md to common x86/pmap.h.
 1.126  19-Jul-2020  maxv don't include opt_user_ldt.h when it is not needed
 1.125  15-May-2020  ad Revert previous after thinking about it. It was wrong, don't need to use
an atomic to clear a PTE or set initial version unless the circumstances
call for it.
 1.124  17-Mar-2020  ad Always set PTEs using atomics. There are too many assumptions to go wrong.
 1.123  09-Mar-2019  maxv New software PTE bits.
 1.122  11-Feb-2019  cherry We reorganise definitions for XEN source support as follows:

XEN - common sources required for baseline XEN support.
XENPV - sources required for support of XEN in PV mode.
XENPVHVM - sources required for support for XEN in HVM mode.
XENPVH - sources required for support for XEN in PVH mode.
 1.121  19-Nov-2018  maxv Introduce pl_pi, will be used soon.
 1.120  19-Nov-2018  maxv Rename 'mask' -> 'frame', we will use the real 'mask' soon.
 1.119  25-Jul-2018  maxv Remove NPTECL, unused.
 1.118  19-Sep-2016  maya branches: 1.118.14; 1.118.16;
move function prototype to x86, so it is available to amd64 too
 1.117  21-Apr-2014  christos branches: 1.117.4; 1.117.8;
forward decl for struct pcb.
 1.116  18-Mar-2013  para branches: 1.116.6; 1.116.10;
i386 pmap.h got a forward declarition for vm_map from systm.h
insert a forward declaration
 1.115  19-Feb-2012  cherry branches: 1.115.2;
Removing remaining references to the alternate PTE space. Modify documentation appropriately
 1.114  19-Jan-2012  bouyer pmap_pte_set() is not supposed to be atomic, so only raise IPL, no need to
take pte_lock
 1.113  09-Jan-2012  cherry Make cross-cpu pte access MP safe.
XXX: review cases of use of pmap_set_pte() vs direct use of xpq_queue_pte_update()
 1.112  06-Nov-2011  cherry branches: 1.112.4;
[merging from cherry-xenmp] make pmap_kernel() shadow PMD per-cpu and MP aware.
 1.111  06-Nov-2011  cherry [merging from cherry-xenmp] Make the xen MMU op queue locking api private. Implement per-cpu queues.
 1.110  13-Aug-2011  cherry branches: 1.110.2;
Add locking around ops to the hypervisor MMU "queue".
 1.109  01-Feb-2011  chuck branches: 1.109.2;
udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.
 1.108  14-Nov-2010  uebayasi branches: 1.108.2; 1.108.4;
Move struct vm_page_md definition from vmparam.h to pmap.h, because
it's used only by pmap. vmparam.h has definitions for wider
audience.

All GENERIC kernels build tested, except ia64.

powerpc/include/booke/vmparam.h has one too, but it has no pmap.h,
so it's left as is.
 1.107  24-Jul-2010  jym Welcome PAE inside i386 current.

This patch is inspired by work previously done by Jeremy Morse, ported by me
to -current, merged with the work previously done for port-xen, together with
additionals fixes and improvements.

PAE option is disabled by default in GENERIC (but will be enabled in ALL in
the next few days).

In quick, PAE switches the CPU to a mode where physical addresses become
36 bits (64 GiB). Virtual address space remains at 32 bits (4 GiB). To cope
with the increased size of the physical address, they are manipulated as
64 bits variables by kernel and MMU.

When supported by the CPU, it also allows the use of the NX/XD bit that
provides no-execution right enforcement on a per physical page basis.

Notes:

- reworked locore.S

- introduce cpu_load_pmap(), used to switch pmap for the curcpu. Due to the
different handling of pmap mappings with PAE vs !PAE, Xen vs native, details
are hidden within this function. This helps calling it from assembly,
as some features, like BIOS calls, switch to pmap_kernel before mapping
trampoline code in low memory.

- some changes in bioscall and kvm86_call, to reflect the above.

- the L3 is "pinned" per-CPU, and is only manipulated by a
reduced set of functions within pmap. To track the L3, I added two
elements to struct cpu_info, namely ci_l3_pdirpa (PA of the L3), and
ci_l3_pdir (the L3 VA). Rest of the code considers that it runs "just
like" a normal i386, except that the L2 is 4 pages long (PTP_LEVELS is
still 2).

- similar to the ci_pae_l3_pdir{,pa} variables, amd64's xen_current_user_pgd
becomes an element of cpu_info (slowly paving the way for MP world).

- bootinfo_source struct declaration is modified, to cope with paddr_t size
change with PAE (it is not correct to assume that bs_addr is a paddr_t when
compiled with PAE - it should remain 32 bits). bs_addrs is now a
void * array (in bootloader's code under i386/stand/, the bs_addrs
is a physaddr_t, which is an unsigned long).

- fixes in multiboot code (same reason as bootinfo): paddr_t size
change. I used Elf32_* types, use RELOC() where necessary, and move the
memcpy() functions out of the if/else if (I do not expect sym and str tables
to overlap with ELF).

- 64 bits atomic functions for pmap

- all pmap_pdirpa access are now done through the pmap_pdirpa macro. It
hides the L3/L2 stuff from PAE, as well as the pm_pdirpa change in
struct pmap (it now becomes a PDP_SIZE array, with or without PAE).

- manipulation of recursive mappings ( PDIR_SLOT_{,A}PTEs ) is done via
loops on PDP_SIZE.

See also http://mail-index.netbsd.org/port-i386/2010/07/17/msg002062.html

No objection raised on port-i386@ and port-xen@R for about a week.

XXX kvm(3) will be fixed in another patch to properly handle both PAE and !PAE
kernel dumps (VA => PA macros are slightly different, and need proper 64 bits
PA support in kvm_i386).

XXX Mixing PAE and !PAE modules may lead to unwanted/unexpected results. This
cannot be solved easily, and needs lots of thinking before being declared
safe (paddr_t/bus_addr_t size handling, PD/PT macros abstractions).
 1.106  15-Jul-2010  jym Purely cosmetic: move L2_SLOT_KERNBASE out of the PAE #ifdef/#endif block.
 1.105  01-Mar-2010  jym branches: 1.105.2;
Use PDP_SIZE for NTOPLEVEL_PDES (number of top level PDEs) instead of
#ifdef'ing PAE.
 1.104  09-Feb-2010  jym Fix typos in comments.
 1.103  26-Oct-2008  mrg branches: 1.103.8; 1.103.14;
- use _I386_FOO_H for multi-include protection
- use <i386/foo.h> in a couple of places
 1.102  23-Jan-2008  bouyer branches: 1.102.6; 1.102.10; 1.102.16;
Merge the bouyer-xeni386 branch. This brings in PAE support to NetBSD xeni386
(domU only). PAE support is enabled by 'options PAE', see the new XEN3PAE_DOMU
and INSTALL_XEN3PAE_DOMU kernel config files.

See the comments in arch/i386/include/{pte.h,pmap.h} to see how it works.
In short, we still handle it as a 2-level MMU, with the second level page
directory being 4 pages in size. pmap switching is done by switching the
L2 pages in the L3 entries, instead of loading %cr3. This is almost required
by Xen, which handle the last L2 page (the one mapping 0xc0000000 - 0xffffffff)
in a very special way. But this approach should also work for native PAE
support if ever supported (in fact, the pmap should almost suport native
PAE, what's missing is bootstrap code in locore.S).
 1.101  20-Jan-2008  bouyer Make first argument of Xen's pmap_pte_cas() volatile, fix a warning
building pmap.c.
 1.100  13-Jan-2008  yamt add pmap_pte_cas.
 1.99  12-Jan-2008  bouyer Xen doesn't need NKPTP_MIN/NKPTP_MAX any more.
 1.98  11-Jan-2008  bouyer Merge the bouyer-xeni386 branch to head, at tag bouyer-xeni386-merge1 (the
branch is still active and will see i386PAE support developement).
Sumary of changes:
- switch xeni386 to the x86/x86/pmap.c, and the xen/x86/x86_xpmap.c
pmap bootstrap.
- merge back most of xen/i386/ to i386/i386
- change the build to reduce diffs between i386 and amd64 in file locations
- remove include files that were identical to the i386/amd64 counterparts,
the build will find them via the xen-ma/machine link.
 1.97  28-Nov-2007  ad branches: 1.97.6;
Remove remaining CPUCLASS_386 tests.
 1.96  28-Nov-2007  ad Use the new atomic ops.
 1.95  22-Nov-2007  bouyer Pull up the bouyer-xenamd64 branch to HEAD. This brings in amd64 support
to NetBSD/Xen, both Dom0 and DomU.
 1.94  18-Oct-2007  yamt branches: 1.94.2;
merge yamt-x86pmap branch.

- reduce differences between amd64 and i386. notably, share pmap.c
between them. it makes several i386 pmap improvements available to
amd64, including tlb shootdown reduction and bug fixes from Stephan Uphoff.
- implement deferred pmap switching for amd64.
- remove LARGEPAGES option. always use large pages if available.
also, make it work on amd64.
 1.93  17-Oct-2007  garbled Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree. Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches. The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.
 1.92  17-Oct-2007  ad Match amd64:

-#if defined(_KERNEL) || defined(_LKM)
+#if defined(_KERNEL)
 1.91  26-Sep-2007  ad branches: 1.91.2;
Only include machine/cpufunc.h if _KERNEL.
 1.90  29-Aug-2007  ad branches: 1.90.2;
Merge most x86 changes from the vmlocking branch, except the threaded soft
interrupt stuff. This is mostly comprised of changes to the pmap modules to
work on multiprocessor systems without kernel_lock, and changes to speed up
tlb shootdowns.
 1.89  21-Feb-2007  thorpej branches: 1.89.4; 1.89.12; 1.89.16; 1.89.20; 1.89.22;
Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.
 1.88  16-Nov-2006  christos branches: 1.88.4;
__unused removal on arguments; approved by core.
 1.87  12-Oct-2006  christos - sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386
 1.86  16-Feb-2006  perry branches: 1.86.14; 1.86.16;
Change "inline" back to "__inline" in .h files -- C99 is still too
new, and some apps compile things in C89 mode. C89 keywords stay.

As per core@.
 1.85  26-Dec-2005  perry branches: 1.85.2; 1.85.4; 1.85.6;
u_intN_t -> uintN_t
 1.84  24-Dec-2005  perry Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.
 1.83  08-Aug-2005  junyoung - Fix typo in comment.
- Remove trailing spaces.
 1.82  26-Aug-2004  junyoung branches: 1.82.12;
As suggested on tech-kern@ days ago:
* Get rid of PTmap, PTD, PTDpde, APTmap, APTD, and APTDpde from locore.S.
* Rename PTDpaddr to PDPpaddr, ptdpaddr in struct cpu_kcore_hdr to pdppaddr for consistency.
 1.81  22-Aug-2004  junyoung Fix various typos in comments including ones that weren't noticed
for years.
 1.80  05-Aug-2004  mycroft Set NKPTP default to 0 (it's calculated in locore). Also reduce NKPTP_MIN to
2 -- though, with our current rounding of the data section, we will never take
advantage of that.
 1.79  20-Feb-2004  yamt defer pmap switching until it's really needed
to avoid frequent loading of cr3 register, which involves tlb flush.

with some fixes/improvements from Stephan Uphoff and Bang Jun-Young.
 1.78  27-Oct-2003  junyoung Nuke __P().
 1.77  23-Oct-2003  chs rename pv_next to pv_node. use SPLAY_RIGHT() instead of expanding it inline.
 1.76  23-Oct-2003  provos changed linked list in pmap_remove_pv to a splay tree; approved: fvdl@
 1.75  24-Aug-2003  chs add support for non-executable mappings (where the hardware allows this)
and make the stack and heap non-executable by default. the changes
fall into two basic catagories:

- pmap and trap-handler changes. these are all MD:
= alpha: we already track per-page execute permission with the (software)
PG_EXEC bit, so just have the trap handler pay attention to it.
= i386: use a new GDT segment for %cs for processes that have no
executable mappings above a certain threshold (currently the
bottom of the stack). track per-page execute permission with
the last unused PTE bit.
= powerpc/ibm4xx: just use the hardware exec bit.
= powerpc/oea: we already track per-page exec bits, but the hardware only
implements non-exec mappings at the segment level. so track the
number of executable mappings in each segment and turn on the no-exec
segment bit iff the count is 0. adjust the trap handler to deal.
= sparc (sun4m): fix our use of the hardware protection bits.
fix the trap handler to recognize text faults.
= sparc64: split the existing unified TSB into data and instruction TSBs,
and only load TTEs into the appropriate TSB(s) for the permissions.
fix the trap handler to check for execute permission.
= not yet implemented: amd64, hppa, sh5

- changes in all the emulations that put a signal trampoline on the stack.
instead, we now put the trampoline into a uvm_aobj and map that into
the process separately.

originally from openbsd, adapted for netbsd by me.
 1.74  22-Jul-2003  yamt move pv_head and page attributes into md part of vm_page structure
instead of having separate arrays for them.
(i.e. use __HAVE_VM_PAGE_MD instead of __HAVE_PMAP_PHYSSEG.)
 1.73  09-Apr-2003  thorpej branches: 1.73.2;
Add the ability for pool caches to cache the physical address of
objects. Clients of the pool_cache API must consistently use
the "paddr" variants or not, otherwise behavior is undefined.

Enable this on Alpha, ARM, MIPS, and x86. Other platforms must
define POOL_VTOPHYS() in the appropriate manner in order to enable
the feature.

Part 1 of a series of simple patches contributed by Wasabi Systems
to improve network performance.
 1.72  09-Apr-2003  thorpej Remove something accidentally included in last commit.
 1.71  02-Apr-2003  thorpej Use PAGE_SIZE rather than NBPG.
 1.70  03-Mar-2003  fvdl define pmap_cpu_has_pg_n and pmap_cpu_has_invlpg to make code sharing
easier.
 1.69  26-Feb-2003  fvdl Redirect some includes to x86/foo.h
 1.68  29-Jan-2003  drochner give some VM which was reserved for the UAREA but not used anymore back
to the user and adjust some comments (which were not accurate anyway
since NOREDZONE)
binary compatibility note: changing VM_MAXUSER_ADDRESS might influence
some sanity check in kvm_proc, where arguments on the stack are dealt
with, but it was a variable anyway and noone did care...
 1.67  17-Jan-2003  thorpej Merge the nathanw_sa branch.
 1.66  02-Nov-2002  perry /*ARGSUSED*/ one function -- eliminates 414 lint complaints
 1.65  01-Oct-2002  fvdl Merge Bill Sommerfeld's i386 MP branch. This code has some known
caveats, but works quite well in a lot of MP cases, and all
UP cases that I have tested. Parts of this will hopefully be
reworked in the not-too-distant future.
 1.64  22-Sep-2002  chs it really helps to get the stub right before cutting + pasting it 27 times.
alas, I did not. doh.
 1.63  22-Sep-2002  chs add pmap_remove_all() hook (empty on most platforms so far).
 1.62  23-Dec-2001  thorpej Use ANSI declarations for static inline functions, and give them
the __unused__ attribute.
 1.61  07-Nov-2001  yamt correct comment.
 1.60  10-Sep-2001  chris branches: 1.60.2;
Update pmap_update to now take the updated pmap as an argument.
This will allow improvements to the pmaps so that they can more easily defer expensive operations, eg tlb/cache flush, til the last possible moment.

Currently this is a no-op on most platforms, so they should see no difference.

Reviewed by Jason.
 1.59  12-Aug-2001  chs branches: 1.59.2;
update some comments.
 1.58  30-May-2001  mrg branches: 1.58.2;
use _KERNEL_OPT
 1.57  26-May-2001  chs replace {simple_,}lock{_data,}_t with struct {simple,}lock {,*}.
 1.56  29-Apr-2001  thorpej The idle loop page zero'er no longer needs to do uncached access
now that we have page coloring.
 1.55  22-Apr-2001  thorpej Undo a misguided previous change to the pmap_update() API.
 1.54  22-Apr-2001  thorpej Give pmap_update() an argument (a pmap_t) so that it knows which
pmap it should be updating.
 1.53  04-Jan-2001  thorpej branches: 1.53.2;
Just garbage-collect the #if 0'd pmap_transfer() stuff.
 1.52  01-Jan-2001  thorpej Don't equate pmap_update() with tlbflush(), but instead call
tlbflush() when that's what we mean (currently everywhere, except
in the one place in MI code where it is called).

The whole pmap_update() thing needs to be reexamined, but this helps
to clarify things a little with the currently-defined semantics of
that function.
 1.51  06-Dec-2000  chs in kvtopte(), return the mapping of the pde in the recursive page-table
mapping space rather than the mapping in the kernel address space.
 1.50  28-Sep-2000  is We do wired page accounting. Tell UVM about it.
 1.49  21-Sep-2000  thorpej Make PMAP_PAGEIDLEZERO() return a boolean value. FALSE indidcates
that the page being zero'd was not completed and that page zeroing
should be aborted. This may be used by machine-dependent code doing
slow page access to reduce the latency of running a process that has
become runnable while in the middle of doing a slow page zero.
 1.48  07-Sep-2000  thorpej Add optional support for mapping the kernel with large (4MB) pages.
Right now, only does kernel text, and wastes a little memory, but
there is some noticeable performance improvement even with just text.
 1.47  06-Sep-2000  thorpej vtopte() and kvtopte() now check for kernel vs. non-kernel addresses,
respectively.
 1.46  06-Sep-2000  thorpej Remove some vto*() macros that won't stand a chance of working
with PSE or PAE, and reimplement vtphys() in terms of pmap_extract().
 1.45  15-Jun-2000  mycroft branches: 1.45.2;
Disable pmap_copy() for now.
 1.44  24-Apr-2000  thorpej branches: 1.44.2;
Glue in uvm_pageidlezero(). While here, improve some panic messages
in pmap.c, as requested by Jonathan Stone.
 1.43  11-Feb-2000  thorpej branches: 1.43.2;
Move KERNBASE to 0xc0000000, giving the kernel a more-or-less full 1G
of virtual address space, leaving userland with 3G, and update comments
to match the new reality.

We knew we were going to have to bite this bullet eventually, and there
are a couple of outstanding PRs related to this issue (9389 and 9313).
Complete solution to those PRs is going to involve some sort of run-time
decision on how large kmem_map should be, as well as changing some data
structure allocation strategies in UVM. However, this change will at
least allow the PR submitter to simply throw resources at the problem.
 1.42  28-Jul-1999  thorpej branches: 1.42.2;
Don't inline pmap_kenter_pa(). It doesn't buy us much to do so, and
it's nice to have it show up in stack traces.
 1.41  18-Jul-1999  chs KNF.
 1.40  17-Jun-1999  thorpej PMAP_NEW is no longer optional on the i386; the old pmap's page table
allocation strategy no longer works at all. Move pmap.new.* to pmap.*.

To read the revision history of PMAP_NEW up until this merge, use cvs
rlog of the old pmap.new.* files.
 1.39  12-May-1999  thorpej Move the user-set LDT out of the PCB and into the pmap. Applications
which set the LDT and share VM space (e.g. new versions of WINE) expect
the LDT to be logically coupled to the address space. Use the new pmap_fork()
interface to copy non-shared user-set LDTs when the address space is forked.
 1.38  15-Aug-1998  mycroft branches: 1.38.6; 1.38.8;
Assign my copyrights to TNF.
 1.37  13-Aug-1998  thorpej vm_offset_t -> {vaddr_t,paddr_t}, vm_size_t -> vsize_t
 1.36  18-Feb-1998  thorpej Fixup an inconsistency in the declaration of PTDpaddr, pointed out by
Chris Demetriou.
 1.35  18-Feb-1998  cgd Move pmap_map() function definition to MD headers, as appropriate. It's
an internal function, and the VM system shouldn't try to prototype it.
(Note that some ports _don't_ prototype it.)
 1.34  10-Feb-1998  mrg - add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.
 1.33  06-Feb-1998  mrg add the i386 MD portions for UVM.
 1.32  13-Jan-1998  mrg this is the i386 MD portion of the MACHINE_NEW_NONCONTIG code, as
written by chuck cranor. thanks to mycroft for helping me find the
one little line of code i accidentally deleted while merging it.

this is not enabled by default. `options MACHINE_NEW_NONCONTIG'
will use this code. eventually, this should go into <machine/vmparam.h>
insteaed of MACHINE_NONCONTIG.
 1.31  06-Jan-1998  thorpej Garbage-collect pm_pdchanged; it's not used by anything.
 1.30  03-Jan-1998  thorpej Now that all ports have pmap_activate(), and it has an identical interface,
prototype it in <vm/pmap.h>
 1.29  07-Dec-1997  fvdl Define constants for the start and end VA of user page tables.
 1.28  04-Nov-1997  thorpej Make pmap_{,de}activate() take a pointer to a proc.
 1.27  20-Oct-1997  fvdl branches: 1.27.2;
Change various constants that depend on the kernel VM base address. Lower
it to 0xf0000000 to give it more breathing space.
 1.26  12-Jun-1997  thorpej Nuke __VM_PMAP_HACK.
 1.25  16-May-1997  gwr Add #define __VM_PMAP_HACK as a temporary measure.
 1.24  18-Nov-1996  fvdl Changes for NKPDE calculation from cgd.

- New variables: biosextmem, biosbasemem, nkpde
- Above can be set by using the options BIOSEXTMEM, BIOSBASEMEM,
and NKPDE respectively (EXTMEM_SIZE is now called BIOSEXTMEM).
When preset this way, they won't be filled in / calculated.
- Readable by sysctl using machdep.nkpde, machdep.biosbasemem
and machdep.biosextmem.
- nkpde is calculated as:

min(NKPDE_MAX, NKPDE_BASE + (biosextmem >> 10) * NKPDE_SCALE)

Where NKPDE_MAX is 31, NKPDE_BASE is 4, NKPDE_SCALE = 1.
 1.23  03-May-1996  christos Add missing prototypes.
 1.22  12-Feb-1996  christos Move pmap_page_index declaration to <vm/pmap.h>
 1.21  11-Oct-1995  mycroft Various changes from John Kohl and me:
Map kernel stacks only at unique addresses.
Use one TSS per process.
Add sysarch calls for modifying IOPL and the I/O permission bitmap.
Add a compacting GDT entry allocator, for TSS and LDT selectors.
Enable modifying %fs and %gs with PT_SETREGS.
Sanitize various bits of code.
 1.20  26-Jun-1995  cgd IdlePTD has been gone for a while. PTDpaddr now exists.
 1.19  11-May-1995  jtc KERNEL -> _KERNEL
 1.18  10-Apr-1995  mycroft Bring back pmap_kernel(), for now always inlined as a pointer to
kernel_pmap_store.
 1.17  28-Mar-1995  jtc KERNEL -> _KERNEL
 1.16  31-Oct-1994  andrew Increase NKPDE to cope with 64MB RAM.
 1.15  27-Oct-1994  cgd new RCS ID format.
 1.14  09-Oct-1994  mycroft #include cpufunc.h from pmap.h.
 1.13  09-Oct-1994  mycroft Define pmap_update() using tlbflush().
 1.12  09-Sep-1994  mycroft Make kernel_pmap a macro that points directly to kernel_pmap_store.
 1.11  07-Sep-1994  mycroft Various changes:
* Inline some functions.
* Eliminate redundant calls to pmap_page_index().
* Reduce the number of TLB flushes in accordance with the i386 SSWG.
* Inline pmap_remove() in pmap_remove_all(). (Most if it is unnecessary.)
* Allocate pv_entry's in pages, and keep a list of free entries in each page,
and a list of pages with free entries. (Trying to reduce malloc() overhead,
and improve locality of reference.)
* Remove a couple of macros that aren't really useful.
* Remove pmap_kernel() completely.
 1.10  15-Aug-1994  mycroft Eliminate struct pte and struct pde.
 1.9  02-Jun-1994  mycroft branches: 1.9.2;
Increase default NKPDE.
 1.8  23-May-1994  cgd cast to quiet warning
 1.7  05-Jan-1994  mycroft branches: 1.7.2;
Move kernel to 0xf8000000.
 1.6  14-Dec-1993  mycroft Fold in most of the magnum vm changes.
 1.5  30-Aug-1993  brezak branches: 1.5.2;
Add prototype for pmap_bootstrap().
 1.4  27-Aug-1993  brezak i386 Machine dependant changes to handle non-contiguous memory.
 1.3  29-Jul-1993  jtc Change "#endif FOO" to "#endif /* FOO */", to keep gcc -Wall, lint, etc.
from complaining about a trivial issue.
 1.2  22-May-1993  cgd add rcsids to everything and clean up headers
 1.1  21-Mar-1993  cgd branches: 1.1.1;
Initial revision
 1.1.1.1  21-Mar-1993  cgd initial import of 386bsd-0.1 sources
 1.5.2.4  13-Dec-1993  mycroft Stylistic changes.
 1.5.2.3  26-Oct-1993  mycroft Nuke I386_PAGE_SIZE and I386_PDR_SIZE. Make the kernel page directory 7
pages.
 1.5.2.2  15-Oct-1993  mycroft Major cleanup of include files and constants.
 1.5.2.1  10-Oct-1993  mycroft load_cr3 --> lcr3
 1.7.2.2  05-Jan-1994  mycroft Move kernel to 0xf8000000.
 1.7.2.1  05-Jan-1994  mycroft file pmap.h was added on branch magnum on 1994-01-05 16:02:39 +0000
 1.9.2.2  11-Oct-1994  mycroft Update from trunk.
 1.9.2.1  15-Aug-1994  mycroft update from trunk
 1.27.2.1  09-Dec-1997  thorpej Pull up rev 1.29 (fvdl)
 1.38.8.2  02-Aug-1999  thorpej Update from trunk.
 1.38.8.1  21-Jun-1999  thorpej Sync w/ -current.
 1.38.6.1  19-Feb-2000  he Apply patch (requested by thorpej):
Move KERNBASE to 0xc0000000, giving the kernel a more-or-less
full 1GB of virtual address space, leaving userland with 3GB,
and update comments to match the new reality.
 1.42.2.3  05-Jan-2001  bouyer Sync with HEAD
 1.42.2.2  08-Dec-2000  bouyer Sync with HEAD.
 1.42.2.1  20-Nov-2000  bouyer Update thorpej_scsipi to -current as of a month ago
A i386 GENERIC kernel compiles without the siop, ahc and bha drivers
(will be updated later). i386 IDE/ATAPI and ncr work, as well as
sparc/esp_sbus. alpha should work as well (untested yet).
siop, ahc and bha will be updated once I've updated the branch to current
-current, as well as machine-dependant code.
 1.43.2.16  29-Dec-2001  sommerfeld Yet another mergeup.

New work:
- Frank van der Linden's tlb shootdown fix.
Mainline functionality merged:
- IrDA
- Kernel RCSID's
- Transmeta CPU support
- ACPI
- XMM register access through procfs
 1.43.2.15  22-Sep-2001  sommerfeld Merge branch with current, post ubcperf, MTRR, etc., etc.,
with the recent fifo_putpages fix to the trunk, GENERIC.MPDEBUG config
boots multiuser on my BP6.
 1.43.2.14  03-Sep-2001  sommerfeld Merge with -current once more.
 1.43.2.13  18-Jun-2001  sommerfeld checkpoint merge-in-progress from mainline.
 1.43.2.12  26-May-2001  sommerfeld Merge up with -current
 1.43.2.11  30-Apr-2001  sommerfeld Merge sommerfeld_i386mp_1 branch with -current as of today.
GENERIC.MP kernel boots and runs multiuser on my ABIT BP6
 1.43.2.10  07-Jan-2001  sommerfeld Snapshot of merge-in-progress with -current.

[Not expected to build]. Catch up with the last N months worth of
changes to -current.
 1.43.2.9  03-Jan-2001  thorpej Clean up the TLB invalidation logic. Remove the last vestiges of
the old "remove record" stuff, and just use the shootdown path, even
for uniprocessor kernels. Change the shootdown path to defer actually
peforming work (or sending the IPI to signal other processors to begin
work) until explicitly triggered by a call to pmap_tlb_shootnow().

Also remove all of the PTP and PV entry stealing code -- it's complicated,
not often called, and not really useful anymore now that UVM itself can
handle pmap_enter() failures.
 1.43.2.8  02-Jan-2001  thorpej Mirror a change made on the trunk:

Don't equate pmap_update() with tlbflush(), but instead call
tlbflush() when that's what we mean (currently everywhere, except
in the one place in MI code where it is called).

The whole pmap_update() thing needs to be reexamined, but this helps
to clarify things a little with the currently-defined semantics of
that function.
 1.43.2.7  31-Dec-2000  thorpej - Keep track of which CPUs are using a given pmap, and use this
information to flush the APTE space of other CPUs as necessary.
- Use atomic operations in the appropriate places to avoid race
conditions between the kernel and the CPU microcode which could
lose modified/referenced information.

Reviewed by Bill Sommerfeld.
 1.43.2.6  23-Sep-2000  sommerfeld Pending more complete pmap merge, deal with PMAP_PAGEIDLEZERO change
 1.43.2.5  06-Sep-2000  sommerfeld If MULTIPROCESSOR, allocate per-cpu VA space and special PTE for page
zero, page copy, and ptp mapping. Eliminate the locks which were used
to protect those special PTE/VA pairs since they're no longer
necessary.
 1.43.2.4  18-Aug-2000  sommerfeld Do tlb shootdown work at splipi().
Pass cpu_info to tlb shootdown ipi entrypoint.
 1.43.2.3  25-Jun-2000  sommerfeld Merge up to just-post-1.5 -current
 1.43.2.2  03-May-2000  sommerfeld Merge with -current
 1.43.2.1  20-Feb-2000  sommerfeld MP: preliminary TLB shootdown support.

Note that the pmap previously had its own idea of a deferred
invalidation queue; use the TLB shootdown queue mechanism for this as
well.
 1.44.2.1  22-Jun-2000  minoura Sync w/ netbsd-1-5-base.
 1.45.2.1  28-Sep-2000  is Pullup from -current, requested by is, approved by thorpej:

sys/arch/arm32/include/pmap.h 1.18
sys/arch/atari/include/pmap.h 1.21
sys/arch/i386/include/pmap.h 1.50
sys/arch/pc532/include/pmap.h 1.29
sys/arch/sh3/include/pmap.h 1.6

These are architectures that do implemented wired page accounting;
advertize it to UVM, so that mlock() from non-root processes works.
 1.53.2.8  11-Nov-2002  nathanw Catch up to -current
 1.53.2.7  18-Oct-2002  nathanw Catch up to -current.
 1.53.2.6  08-Jan-2002  nathanw Catch up to -current.
 1.53.2.5  14-Nov-2001  nathanw Catch up to -current.
 1.53.2.4  21-Sep-2001  nathanw Catch up to -current.
 1.53.2.3  24-Aug-2001  nathanw Catch up with -current.
 1.53.2.2  21-Jun-2001  nathanw Catch up to -current.
 1.53.2.1  05-Mar-2001  nathanw Initial commit of scheduler activations and lightweight process support.
 1.58.2.4  10-Oct-2002  jdolecek sync kqueue with -current; this includes merge of gehenna-devsw branch,
merge of i386 MP branch, and part of autoconf rototil work
 1.58.2.3  10-Jan-2002  thorpej Sync kqueue branch with -current.
 1.58.2.2  13-Sep-2001  thorpej Update the kqueue branch to HEAD.
 1.58.2.1  25-Aug-2001  thorpej Merge Aug 24 -current into the kqueue branch.
 1.59.2.1  01-Oct-2001  fvdl Catch up with -current.
 1.60.2.1  12-Nov-2001  thorpej Sync the thorpej-mips-cache branch with -current.
 1.73.2.7  10-Nov-2005  skrll Sync with HEAD. Here we go again...
 1.73.2.6  21-Sep-2004  skrll Fix the sync with head I botched.
 1.73.2.5  18-Sep-2004  skrll Sync with HEAD.
 1.73.2.4  03-Sep-2004  skrll Sync with HEAD
 1.73.2.3  25-Aug-2004  skrll Sync with HEAD.
 1.73.2.2  12-Aug-2004  skrll Sync with HEAD.
 1.73.2.1  03-Aug-2004  skrll Sync with HEAD
 1.82.12.7  04-Feb-2008  yamt sync with head.
 1.82.12.6  21-Jan-2008  yamt sync with head
 1.82.12.5  07-Dec-2007  yamt sync with head
 1.82.12.4  27-Oct-2007  yamt sync with head.
 1.82.12.3  03-Sep-2007  yamt sync with head.
 1.82.12.2  26-Feb-2007  yamt sync with head.
 1.82.12.1  21-Jun-2006  yamt sync with head.
 1.85.6.1  22-Apr-2006  simonb Sync with head.
 1.85.4.1  09-Sep-2006  rpaulo sync with head
 1.85.2.1  18-Feb-2006  yamt sync with head.
 1.86.16.2  10-Dec-2006  yamt sync with head.
 1.86.16.1  22-Oct-2006  yamt sync with head
 1.86.14.1  18-Nov-2006  ad Sync with head.
 1.88.4.1  27-Feb-2007  yamt - sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.
 1.89.22.3  23-Mar-2008  matt sync with HEAD
 1.89.22.2  09-Jan-2008  matt sync with HEAD
 1.89.22.1  06-Nov-2007  matt sync with HEAD
 1.89.20.6  03-Dec-2007  joerg Sync with HEAD.
 1.89.20.5  27-Nov-2007  joerg Sync with HEAD. amd64 Xen support needs testing.
 1.89.20.4  26-Oct-2007  joerg Sync with HEAD.

Follow the merge of pmap.c on i386 and amd64 and move
pmap_init_tmp_pgtbl into arch/x86/x86/pmap.c. Modify the ACPI wakeup
code to restore CR4 before jumping back into kernel space as the large
page option might cover that.
 1.89.20.3  02-Oct-2007  joerg Sync with HEAD.
 1.89.20.2  10-Sep-2007  joerg Introduce pmap_init_tmp_pgtbl to build a temporary copy of the kernel
side page mapping and an identity mapping low page for use in real mode.
Switch MP bootstrap and i386 ACPI wakeup code to use it.
 1.89.20.1  03-Sep-2007  jmcneill Sync with HEAD.
 1.89.16.1  03-Sep-2007  skrll Sync with HEAD.
 1.89.12.2  17-Oct-2007  garbled Sync with HEAD
 1.89.12.1  03-Oct-2007  garbled Sync with HEAD
 1.89.4.8  03-Dec-2007  ad Sync with HEAD.
 1.89.4.7  23-Oct-2007  ad Sync with head.
 1.89.4.6  09-Oct-2007  ad Sync with head.
 1.89.4.5  23-Aug-2007  ad Add pmap_pte_set, pmap_pte_setbits, pmap_pte_clearbits where missing.
 1.89.4.4  21-Aug-2007  ad - Add ci_cpumask and use in preference to (1 << ci_cpuid). Some processors
are bad at shifting by non-constant amounts.
- pmap_tlb_shootdown() must be called with preemption disabled.
- Defer waiting for tlb shootdown to complete after kenter_pa()/kremove().
Spin in pmap_update() and in pmap_detactivate(), in case the LWP has been
preempted
- When preparing to send a multicast shootdown IPI, avoid scanning the list
of CPUs if the pmap's bitmask indicates that it's only active locally.
 1.89.4.3  18-Aug-2007  yamt - fix TLB problems in recursive PTE mappings, which causes various crashes.
as they are used by the kernel itself, they need to be synced regardless
of lazy tlb flushing mode.

- pmap_tlb_shootdown: fix the case of sva != -1 && eva == 0.

ok'ed by Andrew Doran.
 1.89.4.2  29-Jul-2007  ad - When zeroing/copying pages, use SSE2 movtni to avoid polluting the cache.
- By default, align assembly routines on 32-byte starting boundaries.
- There are now 8 interrupt priority levels, half of which are softints.
Update intrdefs.h to match.
- Always clear/set spinlock words - removes lots of ifdefs.
- Remove the horrible ci_self150 hack that I introduced.
- Overhaul how TLB shootdown is performed. Inspired by a similar change in
OpenBSD but implemented quite differently. This should be a lot faster
but I have not benchmarked it yet.
 1.89.4.1  28-Apr-2007  ad - Make the PV cache per-CPU and eliminate the global pvalloc_lock. Cuts
contention on the freelist locks by a factor of ~50 on a dual core box.
- Mirror a patch for amd64 posted by oster@, and make the TLB shootdown
freelist per-CPU. Increase the number of TLB shootdown jobs from 16
to 32 per CPU.
 1.90.2.13  18-Oct-2007  yamt sync with head.
 1.90.2.12  14-Oct-2007  yamt forward decl of trapframe.
 1.90.2.11  08-Oct-2007  yamt merge some parts of x86 pmap.h.
 1.90.2.10  07-Oct-2007  yamt tweak assertions to reduce diffs between i386 and amd64.
 1.90.2.9  07-Oct-2007  yamt remove unnecessary casts.
 1.90.2.8  07-Oct-2007  yamt sync comments and whitespaces.
 1.90.2.7  07-Oct-2007  yamt g/c nkpde.
 1.90.2.6  07-Oct-2007  yamt whitespace.
 1.90.2.5  07-Oct-2007  yamt remove unused definitions.
 1.90.2.4  06-Oct-2007  yamt sync with head.
 1.90.2.3  04-Oct-2007  yamt remove LARGEPAGES option. always use large pages if available.
 1.90.2.2  24-Sep-2007  yamt remove hardcoded numbers of PTPs used for kernel image and bootstrap tables.
 1.90.2.1  23-Sep-2007  yamt - sync with amd64.
- pmap_alloc_level: remove fast call special case.
- tweak some printf to work for both of LP64 and LP32.
- remove machdep.nkpde sysctl.
- remove automatic adjustment of nkpde for now. will revisit later.
 1.91.2.3  26-Oct-2007  bouyer Make amd64, i386 and xen kernels build and work again.
 1.91.2.2  25-Oct-2007  bouyer Finish sync with HEAD. Especially use the new x86 pmap for xenamd64.
For this:
- rename pmap_pte_set() to pmap_pte_testset()
- make pmap_pte_set() a function or macro for non-atomic PTE write
- define and use pmap_pa2pte()/pmap_pte2pa() to read/write PTE entries
- define pmap_pte_flush() which is a nop in x86 case, and flush the
MMUops queue in the Xen case
 1.91.2.1  25-Oct-2007  bouyer Sync with HEAD.
 1.94.2.2  18-Feb-2008  mjf Sync with HEAD.
 1.94.2.1  08-Dec-2007  mjf Sync with HEAD.
 1.97.6.9  20-Jan-2008  bouyer Sync with HEAD: make first argument of pmap_pte_cas() volatile.
 1.97.6.8  19-Jan-2008  bouyer Make things build again after sync with HEAD
 1.97.6.7  19-Jan-2008  bouyer Sync with HEAD
 1.97.6.6  18-Jan-2008  bouyer Fix APDP handling. A XEN i386PAE kernel now boots multiuser
 1.97.6.5  17-Jan-2008  bouyer - Fix L2_SLOT_APTE value (not sure how I got this value but it was definitively
wrong)
- Use global variable for the PAE L3 page adresses, so that pmap.c can get it
from the bootstrap code
- Extent the size of our virtual PDP from 3 to 4 pages, so that pmap->pm_pdir[]
is contigous for the whole VA range. The last page is a shadow of
the kernel's real PDP (L3[3]).
- make pm_pdirpa an array of 4 paddr_t if using PAE. introduce a
pmap_pdirpa macro to get the physical address of a given PD entry.
- fix pmap_map_pte

The kernel now boots single-user. fsck will cause a kernel fault in
pmap_pdes_invalid() on exit.
 1.97.6.4  15-Jan-2008  bouyer Snapshot of work in progress: an Xen i386PAE kernel boots and start init
on a amd64 dom0, but panics when init forks.
This code needs a lot of cleanup, and the pmap handling is minimal to
allow init to start. It's a proof of concept of how PAE on Xen can work.

For PAE guest, the Xen MMU handling differs in some significant way
from the i386 or amd64 Xen.
The L3 page has only 4 entries, the last one mapping 0xc0000000->0xffffffff
(which happens to be our kenrel VM range, that's cool). The L2 page
pointed to by this last entry is handled specially by Xen because it
contains some Xen private mapping, including a recursive mapping. So this
page can only be pointed to by exactly one L3 entry, and nothing else
(it can't be part of a recursive mapping for example). In addition, it
would waste too much VA space to do recursive mapping at the L3 level.

We do pmap switching at the L# level, instead of doing it though %cr3.
%cr3 is static, as is L3[3] which contains only kenrel mappings.
pmap_load() does pmap switching though the first 3 entries for L3.

PTE mapping is done though 4 contigous L2 entries; the last one pointing
to a shadow of L3[3]. This way we can consider we have a 2-level VM system,
but with the L2 being 4 pages in size instead of one. The plx_i()
macros can be used with it to access the PTE without changes.

This can be reused as is for native PAE support (without the L3[3] shadow
which wouldn't be needed here)
 1.97.6.3  13-Jan-2008  bouyer Update system macros for PAE. Because the L3 page directory has only 4 entries,
and because of the way Xen handles it, it's better to have it has static as
possible. So we claim to have only 2 levels, with level 2 being a
"super" PD of 4 pages. pmap switches will be handled by changing the
L3 entries instead of %cr3. Only the first 3 entries will have to be changed,
the last one mapping KVA space only.
 1.97.6.2  13-Jan-2008  bouyer Work in progress on xeni386 PAE support:
Make xeni386 build with a 64bit paddr_t. For this vaddr_t vs paddr_t vs
pointers usages had to be clarified.
If 'options PAE' is present in a Xen3 kernel, switch paddr_t, pd_entry_t
and pt_entry_t to 64bits, and add the PAE entry in the __xen_guest ELF section.
 1.97.6.1  11-Dec-2007  bouyer Switch i386 to x86/x86/pmap.c
 1.102.16.1  13-Dec-2008  haad Update haad-dm branch to haad-dm-base2.
 1.102.10.3  11-Aug-2010  yamt sync with head.
 1.102.10.2  11-Mar-2010  yamt sync with head
 1.102.10.1  04-May-2009  yamt sync with head.
 1.102.6.1  17-Jan-2009  mjf Sync with HEAD.
 1.103.14.3  16-Nov-2010  uebayasi Sync with HEAD.
 1.103.14.2  17-Aug-2010  uebayasi Sync with HEAD.
 1.103.14.1  30-Apr-2010  uebayasi Sync with HEAD.
 1.103.8.4  27-Aug-2011  jym Sync with HEAD. Most notably: uvm/pmap work done by rmind@, and MP Xen
work of cherry@.

No regression observed on suspend/restore.
 1.103.8.3  28-Mar-2011  jym Sync with HEAD. TODO before merge:
- shortcut for suspend code in sysmon, when powerd(8) is not running.
Borrow ``xs_watch'' thread context?
- bug hunting in xbd + xennet resume. Rings are currently thrashed upon
resume, so current implementation force flush them on suspend. It's not
really needed.
 1.103.8.2  10-Jan-2011  jym Sync with HEAD
 1.103.8.1  24-Oct-2010  jym Sync with HEAD
 1.105.2.1  05-Mar-2011  rmind sync with head
 1.108.4.1  08-Feb-2011  bouyer Sync with HEAD
 1.108.2.1  06-Jun-2011  jruoho Sync with HEAD.
 1.109.2.3  20-Sep-2011  cherry Remove the "xpq lock", since we have per-cpu mmu queues now. This may need further testing. Also add some preliminary locking around queue-ops in the network backend driver
 1.109.2.2  20-Aug-2011  cherry PAE MP support (preliminary), amd64 per-cpu L4 model redesigned, i386 pmap_pa_start/end fixup
 1.109.2.1  31-Jul-2011  cherry grow MP support for i386. boots to single user
 1.110.2.3  22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.110.2.2  17-Apr-2012  yamt sync with head
 1.110.2.1  10-Nov-2011  yamt sync with head
 1.112.4.2  24-Feb-2012  mrg sync to -current.
 1.112.4.1  18-Feb-2012  mrg merge to -current.
 1.115.2.3  03-Dec-2017  jdolecek update from HEAD
 1.115.2.2  20-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.115.2.1  23-Jun-2013  tls resync from head
 1.116.10.1  10-Aug-2014  tls Rebase.
 1.116.6.1  18-May-2014  rmind sync with head
 1.117.8.1  04-Nov-2016  pgoyette Sync with HEAD
 1.117.4.1  05-Oct-2016  skrll Sync with HEAD
 1.118.16.2  08-Apr-2020  martin Merge changes from current as of 20200406
 1.118.16.1  10-Jun-2019  christos Sync with HEAD
 1.118.14.2  26-Nov-2018  pgoyette Sync with HEAD, resolve a couple of conflicts
 1.118.14.1  28-Jul-2018  pgoyette Sync with HEAD

RSS XML Feed