x86: Move definition of struct pmap to pmap_private.h.

This makes pmap_resident_count and pmap_wired_count out-of-line
functions instead of inline. No functional change intended
otherwise.

x86: Split most of pmap.h into pmap_private.h or vmparam.h.

This way pmap.h only contains the MD definition of the MI pmap(9)
API, which loads of things in the kernel rely on, so changing x86
pmap internals no longer requires recompiling the entire kernel every
time.

Callers needing these internals must now use machine/pmap_private.h.
Note: This is not x86/pmap_private.h because it contains three parts:

1. CPU-specific (different for i386/amd64) definitions used by...

2. common definitions, including Xenisms like xpmap_ptetomach,
further used by...

3. more CPU-specific inlines for pmap_pte_* operations

So {amd64,i386}/pmap_private.h defines 1, includes x86/pmap_private.h
for 2, and then defines 3. Maybe we should split that out into a new
pmap_pte.h to reduce this trouble.

No functional change intended, other than that some .c files must
include machine/pmap_private.h when previously uvm/uvm_pmap.h
polluted the namespace with pmap internals.

Note: This migrates part of i386/pmap.h into i386/vmparam.h --
specifically the parts that are needed for several constants defined
in vmparam.h:

VM_MAXUSER_ADDRESS
VM_MAX_ADDRESS
VM_MAX_KERNEL_ADDRESS
VM_MIN_KERNEL_ADDRESS

Since i386 needs PDP_SIZE in vmparam.h, I added it there on amd64
too, just to keep things parallel.

x86: Move struct vm_page_md to common x86/pmap.h.

Revert previous after thinking about it. It was wrong, don't need to use
an atomic to clear a PTE or set initial version unless the circumstances
call for it.

Always set PTEs using atomics. There are too many assumptions to go wrong.

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
- "shad", to track uninitialized memory with a bit granularity (1:1).
Each bit set to 1 in the shad corresponds to one uninitialized bit of
real kernel memory.
- "orig", to track the origin of the memory with a 4-byte granularity
(1:1). Each uint32_t cell in the orig indicates the origin of the
associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
- a code designating the type of memory (Stack, Pool, etc), and
- a compressed pointer, which points either (1) to a string containing
the name of the variable associated with the cell, or (2) to an area
in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

Fix KUBSAN: the kernel size now exceeds the mapping limit, so bump the
limit.

Add support for USER_LDT in SVS. This allows us to have both enabled at
the same time.

We allocate an LDT for each CPU in the GDT and map an area for it, in
addition to the default LDT already present. In context switches between
different processes, we choose between the default or the per-cpu LDT
selector: if the user set specific LDT entries, we memcpy them to the
per-cpu LDT and load the per-cpu selector.

Tested by Naveen Narayanan (with Wine on amd64).

Add PCID support in SVS. This avoids TLB flushes during kernel<->user
transitions, which greatly reduces the performance penalty introduced by
SVS.

We use two ASIDs, 0 (kern) and 1 (user), and use invpcid to flush pages
in both ASIDs.

The read-only machdep.svs.pcid={0,1} sysctl is added, and indicates whether
SVS+PCID is in use.

New software PTE bits.

We reorganise definitions for XEN source support as follows:

XEN - common sources required for baseline XEN support.
XENPV - sources required for support of XEN in PV mode.
XENPVHVM - sources required for support for XEN in HVM mode.
XENPVH - sources required for support for XEN in PVH mode.

Introduce pl_pi, will be used soon.

Rename 'mask' -> 'frame', we will use the real 'mask' soon.

Remove the constants of the DMAP, they are unused, and move NL4_SLOT_DIRECT
into amd64/.

Add support for kASan on amd64. Written by me, with some parts inspired
from Siddharth Muralee's initial work. This feature can detect several
kinds of memory bugs, and it's an excellent feature.

It can be enabled by uncommenting these three lines in GENERIC:

#makeoptions KASAN=1 # Kernel Address Sanitizer
#options KASAN
#no options SVS

The kernel is compiled without SVS, without DMAP and without PCPU area.
A shadow area is created at boot time, and it can cover the upper 128TB
of the address space. This area is populated gradually as we allocate
memory. With this design the memory consumption is kept at its lowest
level.

The compiler calls the __asan_* functions each time a memory access is
done. We verify whether this access is legal by looking at the shadow
area.

We declare our own special memcpy/memset/etc functions, because the
compiler's builtins don't add the __asan_* instrumentation.

Initially all the mappings are marked as valid. During dynamic
allocations, we add a redzone, which we mark as invalid. Any access on
it will trigger a kASan error message. Additionally, the compiler adds
a redzone on global variables, and we mark these redzones as invalid too.
The illegal-access detection works with a 1-byte granularity.

For now, we cover three areas:

- global variables
- kmem_alloc-ated areas
- malloc-ated areas

More will come, but that's a good start.

Remove big outdated comment, remove unused macros, remove XXX that has
nothing to do here, style.

More ASLR: randomize the location of the PTE area. The PTE slot is not
created in locore anymore, but a little later; by using the already
entered L4 page, rather than the recursive slot itself (which doesn't
exist yet).

In the prekern we still map the slot - the prekern behaves as an external
locore -, because we need it as part of the randomization/relocation
work. The kernel then removes this slot, and regenerates a randomized
one.

Tested on GENERIC and GENERIC_KASLR, Xen doesn't have it and dom0 still
boots fine.

Move the PTE area from slot 255 to slot 509. I've never understood why we
put it on 255; the "kernel" half of the VM space begins on slot 256, so
if anything, the PTE area should have been above it, not below.

Virtually extend the user slots in slotspace, because we don't want
(randomized) kernel mappings to land on slot 255.

The prekern is updated accordingly.

Tested on GENERIC, GENERIC_KASLR and XEN3_DOM0.

Introduce PDIR_SLOT_USERLIM, which indicates the limit of the user slots.
Use it instead of PDIR_SLOT_PTE when we just want to iterate over the
user slots. Also use it in SVS, I had hardcoded 255 because there was no
proper define (which there now is).

Randomize the main memory on Xen, same as native. Tested on amd64-dom0.

More ASLR: randomize the kernel main memory. VM_MIN_KERNEL_ADDRESS becomes
variable, and its location is chosen at boot time. There is room for
improvement, since for now we ask for an alignment of NBPD_L4.

This is enabled by default in GENERIC, but not in Xen. Tested extensively
on GENERIC and GENERIC_KASLR, XEN3_DOM0 still boots fine.

Remove KERN_BASE, unused. It has always been wrong anyway, the value
should have been passed into VA_SIGN_NEG().

Remove NPTECL, unused.

branches: 1.46.2;
add experimental new function uvm_direct_process(), to allow of read/writes
of contents of uvm pages without mapping them into kernel, using
direct map or moral equivalent; pmaps supporting the interface need
to provide pmap_direct_process() and define PMAP_DIRECT

implement the new interface for amd64; I hear alpha and mips might be relatively
easy to add too, but I lack the knowledge

part of resolution for PR kern/53124

branches: 1.45.2;
Remove svs_pgg_update(). Instead of manually changing PG_G on each page,
we can disable the global-paging mechanism in %cr4 with CR4_PGE. Do that.

In addition, install CR4_PGE when SVS is disabled manually (via the
sysctl).

Now, doing "sysctl -w machdep.svs_enabled=0" restores the performance
completely, exactly as if SVS hadn't been enabled in the first place.

Improve the SVS initialization.

Declare x86_patch_window_open() and x86_patch_window_close(), and globalify
x86_hotpatch().

Introduce svs_enable() in x86/svs.c, that does the SVS hotpatching.

Change svs_init() to take a bool. This function gets called twice; early
when the system just booted (and nothing is initialized), lately when at
least pmap_kernel has been initialized.

Add svs_enabled, which defaults to 'true' when SVS is compiled (no dynamic
detection yet).

Increase the size of the initial mapping of the kernel. KASLR kernels are
bigger than their GENERIC counterparts, and the limit will soon be hit on
them.

Add a new option, SVS (for Separate Virtual Space), that unmaps kernel
pages when running in userland. For now, only the PTE area is unmapped.

Sent on tech-kern@.

Increase the kernel heap size from 512GB to 32TB, in such a way that it
is able to map the maximum amount of ram supported twice (16TB x 2).

branches: 1.39.8;
Remove useless values, and explain where some others come from

Remove pmap_prealloc_lowmem_ptps on amd64. This function creates levels in
the page tree so that the first 2MB of virtual memory can be kentered in
L1.

Strictly speaking, the kernel should never kenter a virtual page below
VM_MIN_KERNEL_ADDRESS, because then it wouldn't be available in userland.
It used to need the first 2MB in order to map the CPU trampoline and the
initial VAs used by the bootstrap code. Now, the CPU trampoline VA is
allocated with uvm_km_alloc and the VAs used by the bootstrap code are
allocated with pmap_bootstrap_valloc, and in either case the resulting VA
is above VM_MIN_KERNEL_ADDRESS.

The low levels in the page tree are therefore unused. By removing this
function, we are making sure no one will be tempted to map an area below
VM_MIN_KERNEL_ADDRESS in kernel mode, and particularly, we are making sure
NULL cannot be kentered.

In short, there is no way to map NULL in kernel mode anymore.

branches: 1.37.2;
Explain where this value comes from.

KNF so it appears aligned on NXR, and fix a comment.

Bump amd64 module map size to 32 MB.

For lack of anything better to do, after no progress in discussion on
the matter:

https://mail-index.netbsd.org/port-amd64/2014/08/22/msg002108.html

Needed in order to load the (solaris module needed by) dtrace module.

branches: 1.34.2; 1.34.14; 1.34.16;
Extend the xpmap API, as described in [1]. This change is mechanical and
avoids exposing the MD phys_to_machine/machine_to_phys tables directly.
Added:

- xpmap_ptom handles PFN (pseudo physical) to MFN (machine frame number)
translations, and is under control of the domain.
- xpmap_mtop is its counterpart (MFN to PFN), and is under control of
hypervisor.

xpmap_ptom_map() map a pseudo-phys address to a machine address
xpmap_ptom_unmap() unmap a pseudo-phys address (invalidation)
xpmap_ptom_isvalid() check for pseudo-phys address validity

The parameters are physical/machine addresses, like bus_dma/bus_space(9).
As x86 MFNs are tracked by u_long (Xen's choice) while machine addresses
can be 64 bits entities (PAE), use ptoa() to avoid truncation when bit
shifting by PAGE_SHIFT.

I kept the same namespace (xpmap_) to avoid code churn.

[1] http://mail-index.netbsd.org/port-xen/2009/05/09/msg004951.html

XXX will document ptoa/atop/trunc_page separately.

allow more space for modules.

Removing remaining references to the alternate PTE space. Modify documentation appropriately

branches: 1.31.2;
pmap_pte_set() is not supposed to be atomic, so only raise IPL, no need to
take pte_lock

for xen on amd64 PDP_BASE points to the per-cpu ci->ci_kpm_pdir copy of *pmap_kernel()*'s L4 pdir, which is an alias for ci->ci_kpm_pdir. This is unlike PAE, where PDP_BASE points to the per-pmap pm_pdir consisting of 4 pages, the last of which is the "shadow". This "shadow" is not used directly in an active pmap, since it duplicates the kernel space and, for PAE, xen dissallows multiple cpus pointing to the same L3[3] page. Therefore, we use a per-cpu copy of the pmap_kernel() pdir's L3[3] page, ci->ci_pae_l3_pdir[3], while L3[0-2] point to the original pmap's pm_pdir[0 - 2]. Thus the "shadow" pdir only exists on i386 PAE. Note that on PAE, the recursive PDIR_SLOT_PTE is not per-cpu, and therefore cannot be made to point to per-cpu pdirs via (L4_BASE + PDIR_SLOT_PTE), unlike xen x86_64 where this is exactly the case.

Make cross-cpu pte access MP safe.
XXX: review cases of use of pmap_set_pte() vs direct use of xpq_queue_pte_update()

branches: 1.28.4;
[merging from cherry-xenmp] make pmap_kernel() shadow PMD per-cpu and MP aware.

[merging from cherry-xenmp] Make the xen MMU op queue locking api private. Implement per-cpu queues.

branches: 1.26.2;
Implement sparse dumps for amd64 (copied from i386). Disabled for now via
sysctl.
XXX: most of the code can be merged.

Add locking around ops to the hypervisor MMU "queue".

branches: 1.24.2;
udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

branches: 1.23.2; 1.23.4;
Move struct vm_page_md definition from vmparam.h to pmap.h, because
it's used only by pmap. vmparam.h has definitions for wider
audience.

All GENERIC kernels build tested, except ia64.

powerpc/include/booke/vmparam.h has one too, but it has no pmap.h,
so it's left as is.

branches: 1.22.8; 1.22.14; 1.22.16;
put the contents of these header files around #ifdef __x86_64__, and
#include the <i386/foo.h> in the #else clause, making these files
largely bit-size independant.

branches: 1.21.6; 1.21.10; 1.21.16;
Merge the bouyer-xeni386 branch. This brings in PAE support to NetBSD xeni386
(domU only). PAE support is enabled by 'options PAE', see the new XEN3PAE_DOMU
and INSTALL_XEN3PAE_DOMU kernel config files.

See the comments in arch/i386/include/{pte.h,pmap.h} to see how it works.
In short, we still handle it as a 2-level MMU, with the second level page
directory being 4 pages in size. pmap switching is done by switching the
L2 pages in the L3 entries, instead of loading %cr3. This is almost required
by Xen, which handle the last L2 page (the one mapping 0xc0000000 - 0xffffffff)
in a very special way. But this approach should also work for native PAE
support if ever supported (in fact, the pmap should almost suport native
PAE, what's missing is bootstrap code in locore.S).

Make first argument of Xen's pmap_pte_cas() volatile, fix a warning
building pmap.c.

add pmap_pte_cas.

Bump NKL2_KIMG_ENTRIES to allow for 20MB of kernel.

Well past time for an in-kernel linker...

branches: 1.17.6;
Remove remaining CPUCLASS_386 tests.

Use the new atomic ops.

Pull up the bouyer-xenamd64 branch to HEAD. This brings in amd64 support
to NetBSD/Xen, both Dom0 and DomU.

branches: 1.14.2;
merge yamt-x86pmap branch.

- reduce differences between amd64 and i386. notably, share pmap.c
between them. it makes several i386 pmap improvements available to
amd64, including tlb shootdown reduction and bug fixes from Stephan Uphoff.
- implement deferred pmap switching for amd64.
- remove LARGEPAGES option. always use large pages if available.
also, make it work on amd64.

Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree. Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches. The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

branches: 1.12.2;
Only include machine/cpufunc.h if _KERNEL.

branches: 1.11.2;
Merge most x86 changes from the vmlocking branch, except the threaded soft
interrupt stuff. This is mostly comprised of changes to the pmap modules to
work on multiprocessor systems without kernel_lock, and changes to speed up
tlb shootdowns.

branches: 1.10.4; 1.10.10; 1.10.12; 1.10.16; 1.10.20; 1.10.22;
Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.9.20;
Change "inline" back to "__inline" in .h files -- C99 is still too
new, and some apps compile things in C89 mode. C89 keywords stay.

As per core@.

branches: 1.8.2; 1.8.4; 1.8.6;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

branches: 1.6.2;
Remove bogus external declaration for pdes, it appears not to be needed.

kvtopte: use a correct base addr for LARGEPAGES.

correct VAs in a comment.

Add a prototype for pmap_changeprot_local, a function that changes
protection for a page and doesn't care about TLB shootdowns.

Use the SPLAY_* macros. Copied from the i386 pmap, okay'ed by fvdl@

branches: 1.1.2;
Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.