Build and install the prekern by default. I didn't build a full
distribution to test this change, but it seems fine enough.

Fix tags target.

branches: 1.6.2;
Fix 'tags' target.

branches: 1.5.8; 1.5.14; 1.5.16;
- install i386 includes on amd64
- sort

Use ${TOOL_SED} instead if plain sed in Makefiles.

branches: 1.3.6; 1.3.10; 1.3.16;
Install xenio.h and xenio3.h in /usr/include/xen, for the benefit
of xentools3.

XXX ignore those in Makefile.ioctl-c, they don't compile properly outside
of the Xen context and the ioctls from xenio.h conflicts with
soundcard.h

branches: 1.2.18; 1.2.80; 1.2.86;
x86_64 -> amd64

Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.

branches: 1.1.2;
file Makefile.wakecode was initially added on branch jmcneill-pm.

KNF, and reduce the diff between amd64 and i386.

branches: 1.12.6; 1.12.10;
LLVM doesn't support data32/addr32, but is smart enough to figure the
necessary prefixes out.

branches: 1.11.2; 1.11.4;
Set the CR0_AM bit so processes can enable alignment check errors under
x86 through PSL_AC bit.

ATF test incoming shortly.

branches: 1.10.12; 1.10.16; 1.10.18;
Add definition for WAKEUP_vesa_modenum

Ensure that the VBE mode is only restored when machdep.acpi_vbios_reset=1

Pass the VBE mode number from the bootloader to the kernel, and then
make the ACPI wakecode aware of it. Restore the desired VBE mode on resume
when acpi_vbios_reset=1, so suspend/resume with genfb console will work.

branches: 1.7.12;
Random underdocumented CPU facts:
AMD K8 and newer will trap when enabling the NX support outside PAE
paging mode. The AMD64 wake code was restoring the MSR EFER to switch to
Long Mode and naturally didn't have paging enabled at that point.
While this works fine with Intel CPUs, it resulted in an immediate
reboot with AMD processors.

Fixed by a joint brain storming session of jmcneill@, cegger@ and
myself, based on some input from the hardware developers. This fixes
PR 38587.

Remove unnecessary rdmsr when modifying MSR_EFER.

Share cpu.h between the x86 ports.

branches: 1.4.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.3.2; 1.3.4; 1.3.10; 1.3.12; 1.3.14;
Add new IPI for saving CPU state explicitly, share high-level part of
ACPI wakeup code and teach it how to start the APs again. As a side
effect the CPU_START interface allows choosing between different
bootstrap codes more easily now.

branches: 1.2.2;
Merge jmcneill-pm branch.

branches: 1.1.2; 1.1.8; 1.1.10; 1.1.12;
file acpi_wakecode.S was initially added on branch jmcneill-pm.

branches: 1.1.2;
file acpi_wakecode.h was initially added on branch jmcneill-pm.

Add new IPI for saving CPU state explicitly, share high-level part of
ACPI wakeup code and teach it how to start the APs again. As a side
effect the CPU_START interface allows choosing between different
bootstrap codes more easily now.

Move mapping of the real mode location for the ACPI wakeup code into a
separate function called from acpi_md_callback.

Steal some rope from innocent users and disallow ACPI suspend if more
than one CPU was found.

branches: 1.4.2;
Merge jmcneill-pm branch.

branches: 1.3.48; 1.3.50; 1.3.56; 1.3.60; 1.3.62;
merge ktrace-lwp.

branches: 1.2.16;
__KERNEL_RCSID()

branches: 1.1.2;
Empty sleep stub function.

Drop NENTRY() from the x86 kernels, use ENTRY(). With PMCs (and other hardware
tracing facilities) we have a much better ways of monitoring the CPU activity
than GPROF, without software modification.

Also I think GPROF has never worked, because the 'start' functions of both
i386 and amd64 use ENTRY(), and it would have caused a function call while the
kernel was not yet relocated.

branches: 1.9.2; 1.9.4;
Use NENTRY -> END.

Improve our segregs model. Pass 3/3.

Treat %gs the same way we treat %ds/%es/%fs: restore it in INTRFASTEXIT
on 32bit LWPs.

On Xen however, its behavior does not change, because we need to do an
hypercall before INTR_RESTORE_GPRS, and that's too complicated for now.

As a side effect, this change fixes a bug in the ACPI wakeup code; %fs/%gs
were not restored on 32bit LWPs, and chances are they would segfault
shortly afterwards.

Support for USER_LDT on amd64 is almost complete now.

Improve our segregs model. Pass 1/3.

Right now, we are saving and restoring %ds/%es each time we enter/leave the
kernel. However, we let %fs/%gs live in the kernel space, and we rely on
the fact that when switching to an LWP, %fs/%gs are set right away (via
cpu_switchto or setregs).

It has two drawbacks: we are taking care of %ds/%es while they are
deprecated (useless) on 64bit LWPs, and we are restricting %fs/%gs while
they still have a meaning on 32bit LWPs.

Therefore, handle 32bit and 64bit LWPs differently:
* 64bit LWPs use fixed segregs, which are not taken care of.
* 32bit LWPs have dynamic segregs, always saved/restored.

For now, only %ds and %es are changed; %fs and %gs will be in the next
passes.

The trapframe is constructed as usual. In INTRFASTEXIT, we restore %ds/%es
depending on the %cs value. If %cs contains one of the two standard 64bit
selectors, don't do anything. Otherwise, restore everything.

When doing a context switch, just restore %ds/%es to their default values.
On a 32bit LWP they will be overwritten by INTRFASTEXIT; on a 64bit LWP
they won't be updated.

In the ACPI wakeup code, restore %ds/%es to the default 64bit user value.

Make sure %edx is clear.

KNF, and reduce the diff between amd64 and i386.

branches: 1.4.42; 1.4.62; 1.4.66;
Share cpu.h between the x86 ports.

branches: 1.3.2; 1.3.4; 1.3.10; 1.3.12; 1.3.14; 1.3.16;
Add new IPI for saving CPU state explicitly, share high-level part of
ACPI wakeup code and teach it how to start the APs again. As a side
effect the CPU_START interface allows choosing between different
bootstrap codes more easily now.

branches: 1.2.2;
Merge jmcneill-pm branch.

branches: 1.1.2; 1.1.8; 1.1.10; 1.1.12;
file acpi_wakeup_low.S was initially added on branch jmcneill-pm.

branches: 1.1.2;
file genwakecode.sh was initially added on branch jmcneill-pm.

Revert an inadvertent bit of v 1.12

G/C amd64_mainbus_print(); nothing uses it.

Use {,e}isabus_attach().

Rename pvbus component to pv to allow conditional NPV define

pvbus is always defined in files.pv, which prevents the use of
#if NPVBUS > 0. Renaming it to pv aligns with naming conventions used
for other bus components and allows NPV to be conditionally defined.

Allow NVMM to use pv(4) bus

Trivial bus implementation inspired by OpenBSD's pv(4) to attach devices
that don't need nor rely on a PCI or ISA bus.

branches: 1.7.12;
Merge thorpej-cfargs2.

branches: 1.6.8;
Merge thorpej-cfargs branch:

Simplify and make extensible the config_search() / config_found() /
config_attach() interfaces: rather than having different variants for
which arguments you want pass along, just have a single call that
takes a variadic list of tag-value arguments.

Adjust all call sites:
- Simplify wherever possible; don't pass along arguments that aren't
actually needed.
- Don't be explicit about what interface attribute is attaching if
the device only has one. (More simplification.)
- Add a config_probe() function to be used in indirect configuiration
situations, making is visibly easier to see when indirect config is
in play, and allowing for future change in semantics. (As of now,
this is just a wrapper around config_match(), but that is an
implementation detail.)

Remove unnecessary or redundant interface attributes where they're not
needed.

There are currently 5 "cfargs" defined:
- CFARG_SUBMATCH (submatch function for direct config)
- CFARG_SEARCH (search function for indirect config)
- CFARG_IATTR (interface attribte)
- CFARG_LOCATORS (locators array)
- CFARG_DEVHANDLE (devhandle_t - wraps OFW, ACPI, etc. handles)

...and a sentinel value CFARG_EOL.

Add some extra sanity checking to ensure that interface attributes
aren't ambiguous.

Use CFARG_DEVHANDLE in MI FDT, OFW, and ACPI code, and macppc and shark
ports to associate those device handles with device_t instance. This
will trickle trough to more places over time (need back-end for pre-OFW
Sun OBP; any others?).

branches: 1.5.4; 1.5.16;
Fix NLAPIC, NISA and NIOAPIC related conditional compile errors.

This will allow us to now compile an amd64 kernel without PCI.

No functional changes.

branches: 1.4.2;
Style, once again.

This change modifies the mainbus(4) entry point for all x86 sub-archs
in the following way:

i) It provides a unified entry point in
x86/x86/mainbus.c:mainbus_attach()
ii) It carves out the preliminary bus attachment sequence that is
common to all sub-archs into
x86/x86/mainbus.c: x86_cpubus_attach()
iii) It consolidates the remaining pathways as internal callee
functions so that these may be called piecemeal if required. A
special usecase of this is XEN PVHVM which may need to call the
native configure path, the xen configure path, or both.
iv) It moves the driver private data structures from
i386/i386_mainbus.c to an x86/ level one. This allows for other
sub-arch's to do similar, if needed. (They do not at the moment).
v) For dom0 kernels, it enables 'acpi0 at mainbus?' and
'acpi0 at hypervisorbus'. This serves two purposes:
a) To demonstrate the possibility of dynamic configuration tree
traversal ordering changes.
b) To allow for the common acpi_check(self, "acpibus") call in
x86/mainbus.c to not barf when it is called from the dom0 attach
path. We allow for the acpi0 device to be a child of mainbus with
the changes to amd64/conf/XEN3_DOM0 and i386/conf/XEN3PAE_DOM0
without actually probing further in the code. This path will later
be pursued in a PVHVM boot codepath.

There should be no operative changes with this change. If there are,
please complain loudly.

Don't forget pedigree. Re-introduce old RCS Id tags from the originals

Move mainbus(4) driver files in various x86 sub-archs to name prefixed
versions. This allows us to further modularise them by unifying common
bus probe code in x86/x86/mainbus.c to be introduced next.

This commit has no functional changes. It is done for ease of
visibility of newer diffs in the queue.

amd64_trap.S: Annotate trap vectors with their mnemonics.

Derived from Intel and AMD manuals.

Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 3
(3A, 3B, 3C, & 3D: System Programming Guide, Order Number:
325384-077US, April 2022.
https://cdrdv2.intel.com/v1/dl/getContent/671447

AMD64 Technology: AMD64 Architecture Programmers' Manual, Volume 2:
System Programming, Publication No. 24953, Revision 3.40, January 2023.
https://www.amd.com/system/files/TechDocs/24593.pdf

No functional change intended.

XXX Should apply the same treatment to i386_trap.S.

branches: 1.54.4;
NetBSD/x86: Raise the number of interrupt sources per CPU from 32 to 56.

There has been no objection for three years.
https://mail-index.netbsd.org/port-amd64/2019/09/22/msg003012.html
Implemented by nonaka@n.o, updated by me.

Nix trailing whitespace.

On amd64, Xen PV calls syscalls and traps with events enabled.
Disable events on entry to be safe.
It should have been mostly safe for most cases, but for FPU traps
we need to reload the FPU state if we got interrupted at trap entry.

Hopefully fixes:
panic: kernel diagnostic assertion "curlwp->l_md.md_flags & MDL_FPU_IN_CPU" failed: file "/home/source/ab/HEAD/src/sys/arch/x86/x86/fpu.c", line 524

when running tests.

Panic instead of printf, same as syscall.

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
- "shad", to track uninitialized memory with a bit granularity (1:1).
Each bit set to 1 in the shad corresponds to one uninitialized bit of
real kernel memory.
- "orig", to track the origin of the memory with a 4-byte granularity
(1:1). Each uint32_t cell in the orig indicates the origin of the
associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
- a code designating the type of memory (Stack, Pool, etc), and
- a compressed pointer, which points either (1) to a string containing
the name of the variable associated with the cell, or (2) to an area
in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

Rewrite the FPU code on x86. This greatly simplifies the logic and removes
the dependency on IPL_HIGH. NVMM is updated accordingly. Posted on
port-amd64 a week ago.

Bump the kernel version to 9.99.16.

Two changes in the CPU mitigations:

* Micro-optimize: put every mitigation in the same branch. This removes
two branches in each exc/int return path, and removes all branches in
the syscall return path.

* Modify the SpectreV2 mitigation to be compatible with SpectreV4. I
recently realized that both couldn't be enabled at the same time on
Intel. This is because initially, when there was just SpectreV2, we
could reset the whole IA32_SPEC_CTRL MSR. But then Intel added another
bit in it for SpectreV4, so it isn't right to reset it entirely
anymore. SSBD needs to stay.

Mitigation for INTEL-SA-00233: Microarchitectural Data Sampling (MDS).

It requires a microcode update, now available on the Intel website. The
microcode modifies the behavior of the VERW instruction, and makes it flush
internal CPU buffers. We hotpatch the return-to-userland path to add VERW.

Two sysctls are added:

machdep.mds.mitigated = {0/1} user-settable
machdep.mds.method = {string} constructed by the kernel

The kernel will automatically enable the mitigation if the updated
microcode is present. If the new microcode is not present, the user can
load it via cpuctl, and set machdep.mds.mitigated=1.

We reorganise definitions for XEN source support as follows:

XEN - common sources required for baseline XEN support.
XENPV - sources required for support of XEN in PV mode.
XENPVHVM - sources required for support for XEN in HVM mode.
XENPVH - sources required for support for XEN in PVH mode.

Eliminate the only ASM reference to VM_MIN_KERNEL_ADDRESS. Rename the
value to VM_SPACE_SEP_HIGH32, it is now the highest 32bits of the first
va of the higher half of the address space (right after the canonical
hole).

Drop NENTRY() from the x86 kernels, use ENTRY(). With PMCs (and other hardware
tracing facilities) we have a much better ways of monitoring the CPU activity
than GPROF, without software modification.

Also I think GPROF has never worked, because the 'start' functions of both
i386 and amd64 use ENTRY(), and it would have caused a function call while the
kernel was not yet relocated.

Handle NMIs correctly when SVS is enabled. We store the kernel's CR3 at the
top of the NMI stack, and we unconditionally switch to it, because we don't
know with which page tables we received the NMI. Hotpatch the whole thing as
usual.

This restores the ability to use PMCs on Intel CPUs.

branches: 1.42.2;
When the previous contrext is in kernel mode we are not guaranteed to have
a 16-byte-aligned stack pointer, so align it. That's what the CPU would do
on exception entry.

Mitigation for the SS bug, CVE-2018-8897. We disabled dbregs a month ago
in -current and -8 so we are not particularly affected anymore.

The #DB handler runs on ist3, if we decide to process the exception we
copy the iret frame on the correct non-ist stack and continue as usual.

Add the IBRS mitigation for SpectreV2 on amd64.

Different operations are performed during context transitions:

user->kernel: IBRS <- 1
kernel->user: IBRS <- 0

And during context switches:

user->user: IBPB <- 0
kernel->user: IBPB <- 0
[user->kernel:IBPB <- 0 this one may not be needed]

We use two macros, IBRS_ENTER and IBRS_LEAVE, to set the IBRS bit. The
thing is hotpatched for better performance, like SVS.

The idea is that IBRS is a "privileged" bit, which is set to 1 in kernel
mode and 0 in user mode. To protect the branch predictor between user
processes (which are of the same privilege), we use the IBPB barrier.

The Intel manual also talks about (MWAIT/HLT)+HyperThreading, and says
that when using either of the two instructions IBRS must be disabled for
better performance on the core. I'm not totally sure about this part, so
I'm not adding it now.

IBRS is available only when the Intel microcode update is applied. The
mitigation must be enabled manually with machdep.spectreV2.mitigated.

Tested by msaitoh a week ago (but I adapted a few things since). Probably
more changes to come.

(Re)Fix handling of segment register faults. My previous attempt did fix
faults occuring when reloading %es/%ds/%fs/%gs, but it did not fix faults
occuring when executing 'iretq', because before iretq we needed to do +16
in %rsp, and the resulting stack layout was not the one kernuser_reenter()
expected (tf_trapno and tf_err were not there).

So now: pop tf_trapno and tf_err right away in intrfastexit(), and update
the layout in kernuser_reenter() accordingly. The resulting code is
actually simpler.

Tested by "hardcoding" an iretq fault; the process correctly receives a
SIGSEGV.

(Note that segment register faults do not happen in the wild, you really
need to try hard to trigger one.)

Remove the sysretq fault handler. It is broken with SVS, and not really
needed anyway. Initially I had added it so that if such a fault was
received the kernel would panic "cleanly" instead of crashing in a
potentially undefined way.

I'll re-add this handler later.

branches: 1.37.2;
Fix handling of segment register faults when running with SVS. The behavior
is changed also in the non-SVS case.

I've put a documentation in amd64_trap.S. Basically, the problem with SVS
is that if iret faults, we already have a full trapframe pushed on the
stack and the CPU will push another frame on this stack (nested), but it
hits the redzone below the stack since it is still running with the user
page table loaded.

To fix that, we pop a good part of the trapframe earlier in intrfastexit.
If iret faults, the current %rsp has enough room for an iret frame, and
the CPU can push that without problem. We then switch back to the outer
iret frame (the frame the CPU was trying to pop by executing iret, but that
it didn't pop for real because iret faulted), call INTRENTRY, and handle
the trap as if it had been received from userland directly.

Ah. Don't use NENTRY() to declare check_swapgs, use LABEL() instead. NENTRY
puts the code in the .text section, so the effect of TEXT_USER_BEGIN was
overwritten, and check_swapgs was not put in the .text.user section.

As a result kernels running SVS would crash when jumping here - because we
execute this place with the user page table loaded, and in this page table
only .text.user is mapped.

While here, rename check_swapgs -> kernuser_reenter, because we do more
things than just SWAPGS.

Replace %rax -> %rdi, so that check_swapgs clobbers only one register.

There are two places where we reload %gs:

* In setusergs. Here we can't fault. So we don't need to handle this
case.

* In intrfastexit for 32bit processes. This case needs to be handled,
and we already have a label.

So use the label instead of disassembling %rip.

Revert all my latest changes, and restore this file back to how it was
in rev1.24. I wanted to replace the functions dynamically for SVS, but
that was a dumb idea, we'll just hotpatch instead.

Pass the name of the function as argument in SWAPGS_HANDLER.

Declare check_swapgs in an ASM macro. No real functional change.

Use ASM macros for the rest of the entry points. No real functional
change. Now the format of the entry points is:

.macro TRAP_ENTRY_POINT_xx arg1,arg2,arg3
...the asm code...
.endm

TEXT_USER_BEGIN
TRAP_ENTRY_POINT_xx arg1,arg2,arg3
TEXT_USER_END

Declare and use TRAP_ENTRY_POINT_DNA. This time we don't give an is_ztrap
argument, because the macro is tied to the entry point, and it would be
wrong to suggest the paramater is controllable. No real functional change.

Now that [Z]TRAP and [Z]TRAP_NJ are identical, put back the

INTRENTRY
jmp .Lalltraps_noentry

instructions for Xen, and remove [Z]TRAP_NJ.

Declare and use TRAP_ENTRY_POINT_SPUR. No real functional change.

Declare and use TRAP_ENTRY_POINT_FPU. No real functional change.

Start using ASM macros to define the trap entry points. No real functional
change.

Don't restore segment registers when leaving NMIs. In nmitrap (and the
functions it later calls), we are not allowing the trap frame to change;
so the segregs don't change since we are running with interrupts disabled
and there is no rescheduling in this case.

Unmap the kernel from userland in SVS, and leave only the needed
trampolines. As explained below, SVS should now completely mitigate
Meltdown on GENERIC kernels, even though it needs some more tweaking
for GENERIC_KASLR.

Until now the kernel entry points looked like:

FUNC(intr)
pushq $ERR
pushq $TRAPNO
INTRENTRY
... handle interrupt ...
INTRFASTEXIT
END(intr)

With this change they are split and become:

FUNC(handle)
... handle interrupt ...
INTRFASTEXIT
END(handle)

TEXT_USER_BEGIN
FUNC(intr)
pushq $ERR
pushq $TRAPNO
INTRENTRY
jmp handle
END(intr)
TEXT_USER_END

A new section is introduced, .text.user, that contains minimal kernel
entry/exit points. In order to choose what to put in this section, two
macros are introduced, TEXT_USER_BEGIN and TEXT_USER_END.

The section is mapped in userland with normal 4K pages.

In GENERIC, the section is 4K-page-aligned and embedded in .text, which
is mapped with large pages. That is to say, when an interrupt comes in,
the CPU has the user page tables loaded and executes the 'intr' functions
on 4K pages; after calling SVS_ENTER (in INTRENTRY) these 4K pages become
2MB large pages, and remain so when executing in kernel mode.

In GENERIC_KASLR, the section is 4K-page-aligned and independent from the
other kernel texts. The prekern just picks it up and maps it at a random
address.

In GENERIC, SVS should now completely mitigate Meltdown: what we put in
.text.user is not secret.

In GENERIC_KASLR, SVS would have to be improved a bit more: the
'jmp handle' instruction is actually secret, since it leaks the address
of the section we are jumping into. By exploiting Meltdown on Intel, this
theoretically allows a local user to reconstruct the address of the first
text section. But given that our KASLR produces several texts, and that
each section is not correlated with the others, the level of protection
KASLR provides is still good.

Compile amd64_trap.S as a file instead of including it.

Eliminate a '.text'.

Don't declare exceptions[] with IDTVEC, it's an array, not a function.
Rename it to x86_exceptions[], and move it to .rodata.

Fix the double-fault handler. We're executing on ist1 and must not jump
out of it, so don't enable interrupts. And use the SVS_*_ALTSTACK macros.

While here, fix the NMI handler too: it should use SVS_LEAVE_ALTSTACK.

Unmap the kernel heap from the user page tables (SVS).

This implementation is optimized and organized in such a way that we
don't need to copy the kernel stack to a safe place during user<->kernel
transitions. We create two VAs that point to the same physical page; one
will be mapped in userland and is offset in order to contain only the
trapframe, the other is mapped in the kernel and maps the entire stack.

Sent on tech-kern@ a week ago.

Add a new option, SVS (for Separate Virtual Space), that unmaps kernel
pages when running in userland. For now, only the PTE area is unmapped.

Sent on tech-kern@.

Implement a real hotpatch feature.

Define a HOTPATCH() macro, that puts a label and additional information
in the new .rodata.hotpatch kernel section. In patch.c, scan the section
and patch what needs to be. Now it is possible to hotpatch the content of
a macro.

SMAP is switched to use this new system; this saves a call+ret in each
kernel entry/exit point.

Many other operating systems do the same.

Mmh, I made a mistake in r1.10 - I forgot to update this function call.

Use decimal numbering - hex is just misleading -, use ZTRAP_NJ for NMIs,
and declare intrspurious independently.

Hide a bunch of raw symbols.

Have the cpu clear PSL_D automatically when entering the kernel via a
syscall. Then, don't clear PSL_D and PSL_AC in the syscall entry point,
they are now both cleared by the cpu (faster). However they still need to
be manually cleared in the interrupt/trap entry points.

Declare INTRFASTEXIT as a function, so that there is only one iretq in the
kernel. Then, check %rip against the address of this iretq instead of
disassembling (%rip) - which could fault again, or point at some random
address which happens to contain the iretq opcode. The same is true for gs
below, but I'll fix that in another commit.

Remove useless debug code, and split trap() into smaller functions, easier
to understand. NMIs take another, faster path now. No functional change
beyond that.

Add a layer of mitigation against the intel sysret vuln: restore %gs when
sysretq faults. Right now we try to make sure that %rip is canonical by
performing sanity checks in several places, but I've already found missing
checks two times already, and there may be others.

By performing an additional swapgs here, we are turning ring0 exploits to
simple DoSes - which are still security bugs, but of a lower impact.

Reorder for clarity, and style.

Revert my previous change. I hadn't checked carefully enough: the
symbols are used in src/external. There is a number of things that seem
wrong to me here, but I'm not changing them for now.

Remove unused and broken code. On amd64 we won't want int3 from kernel
mode to be valid.

branches: 1.5.6;
Unconditionnally save the segment registers - because we could have a
kernel %gs and a userland %es/%ds -, and explain why T_NMI is a special
case.

Note that checking %gs directly is not a good idea: recent CPUs have the
FSGSBASE instruction set, which allows userland to directly modify %gs
without going through the kernel. If we ever enable this set, we will have
to change this function, since we won't be able to test %gs against
VM_MIN_KERNEL_ADDRESS anymore.

branches: 1.4.2;
Explain a little.

branches: 1.3.2;
KNF a bit, so I don't get scared each time I open a file

branches: 1.2.4; 1.2.8; 1.2.10;
Change the argument to fpudna() to be the trapframe.
Move the checks for fpu traps in kernel into x86/fpu.c.
Remove the code from amd64/trap.c related to fpu traps (they've not gone
there for ages - expect to panic in kernel mode).
In fpudna():
- Don't actually enable hardware interrupts unless we need to
allow in IPIs.
- There is no point in enabling them when they are blocked in software
(by splhigh()).
- Keep the splhigh() to avoid a load of the KASSERTS() firing.

branches: 1.1.2;
Split these to improve diffability.

Merge amd64func.S into cpufunc.S, and clean up.

branches: 1.7.34;
Use END() to set ELF symbol size.

branches: 1.6.32; 1.6.42;
Improve x86 tsc handling:

- Ditch the cross-CPU calibration stuff. It didn't work properly, and it's
near impossible to synchronize the CPUs in a running system, because bus
traffic will interfere with any calibration attempt, messing up the
timings.

- Only enable the TSC on CPUs where we are sure it does not drift. If we are
On a known good CPU, give the TSC high timecounter quality, making it the
default.

- When booting CPUs, detect TSC skew and account for it. Most Intel MP
systems have synchronized counters, but that need not be true if the
system has a complicated bus structure. As far as I know, AMD systems
do not have synchronized TSCs and so we need to handle skew.

- While an AP is waiting to be set running, try and make the TSC drift by
entering a reduced power state. If we detect drift, ensure that the TSC
does not get a high timecounter quality. This should not happen and is
only for safety.

- Make cpu_counter() stuff LKM safe.

branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.4.2; 1.4.4; 1.4.6;
Disable the fast tsc_get_timecount() for now. The divide can overflow
causing an exception.

branches: 1.3.2; 1.3.4;
Shave more cycles off tsc_get_timecount(). From dsl@

branches: 1.2.2;
tsc_get_timecount: remove unneeded instructions. Pointed out by dsl@.

Implement an optimized, preemption-safe asm version of tsc_get_timecount().
The C version needs work to be preemption safe. Cuts the clock cycles
for microtime() from 950 down to 300 on a Pentium D.

Move the MI parts of KASAN into kern/subr_asan.c. This file includes
machine/asan.h, which contains the MD functions. We use an include rather
than a plain C file, because we want GCC to optimize/inline some functions
into one single block.

The amd64 MD parts of KASAN are moved accordingly.

The naming convention we use is:

kasan_*
a generic kasan object, declared in subr_asan.c
kasan_md_*
an MD kasan object, declared in machine/asan.h, and used
in subr_asan.c
__md_*
an MD object, declared in machine/asan.h, and not used
outside

Overall this makes it easier to add KASAN support on more architectures.

Discussed with several people.

Rename kasan_shadow_fill, remove one check in it, and inline it. Remove
the use-after-scope code for now, because our GCC does not support that
and when it does we will want to test the feature for real rather than
letting a potentially broken code compile.

Remove functions that aren't supposed to be used.

Don't go beyond start().

branches: 1.7.2;
Add kasan interceptors for strcpy/strcmp/strlen.

Improve the detection on global variables, no need to round up.

Unwind the stack on error, to get the full path that led to the illegal
access. Example of output:

kASan: Unauthorized Access In 0xffffffff80e6219c: Addr 0xffffbb007a39fd03 [1 byte, read]
#0 0xffffffff80e6219c in ras_purgeall <netbsd>
#1 0xffffffff80e62330 in sys_rasctl <netbsd>
#2 0xffffffff80265008 in syscall <netbsd>

(I manually added a one-byte stack read overflow in rasctl to demonstrate.)

Explicitly unpoison the stack when entering a softint.

Softints are the only place where we "discard" a part of the stack: we may
have left the thread without allowing the asan instrumentation to clear
the poison, and in this case, we can get false positives when we hit a
poisoned area of the stack while executing another handler within the same
softint thread.

(I was actually getting a rare false positive in ip6intr.)

Add support for monitoring the stack with kASan. This allows us to detect
illegal memory accesses occuring there.

The compiler inlines a piece of code in each function that adds redzones
around the local variables and poisons them. The illegal accesses are then
detected using the usual kASan machinery.

The stack size is doubled, from 4 pages to 8 pages.

Several boot functions are marked with the __noasan flag, to prevent the
compiler from adding redzones in them (because we haven't yet initialized
kASan). The kasan_early_init function is called early at boot time to
quickly create the shadow for the current stack; after this is done, we
don't need __noasan anymore in the boot path.

We pass -fasan-shadow-offset=0xDFFF900000000000, because the compiler
wants to do
shad = shadow-offset + (addr >> 3)
and we do, in kasan_addr_to_shad
shad = KASAN_SHADOW_START + ((addr - CANONICAL_BASE) >> 3)
hence
shad = KASAN_SHADOW_START + (addr >> 3) - (CANONICAL_BASE >> 3)
= [KASAN_SHADOW_START - (CANONICAL_BASE >> 3)] + (addr >> 3)
implies
shadow-offset = KASAN_SHADOW_START - (CANONICAL_BASE >> 3)
= 0xFFFF800000000000 - (0xFFFF800000000000 >> 3)
= 0xDFFF900000000000

In UVM, we add a kasan_free (that is not preceded by a kasan_alloc). We
don't add poisoned redzones ourselves, but all the functions we execute
do, so we need to manually clear the poison before freeing the stack.

With the help of Kamil for the makefile stuff.

Fix the computation in kasan_shadow_map, we may need one more page because
of the rounddown.

Add support for kASan on amd64. Written by me, with some parts inspired
from Siddharth Muralee's initial work. This feature can detect several
kinds of memory bugs, and it's an excellent feature.

It can be enabled by uncommenting these three lines in GENERIC:

#makeoptions KASAN=1 # Kernel Address Sanitizer
#options KASAN
#no options SVS

The kernel is compiled without SVS, without DMAP and without PCPU area.
A shadow area is created at boot time, and it can cover the upper 128TB
of the address space. This area is populated gradually as we allocate
memory. With this design the memory consumption is kept at its lowest
level.

The compiler calls the __asan_* functions each time a memory access is
done. We verify whether this access is legal by looking at the shadow
area.

We declare our own special memcpy/memset/etc functions, because the
compiler's builtins don't add the __asan_* instrumentation.

Initially all the mappings are marked as valid. During dynamic
allocations, we add a redzone, which we mark as invalid. Any access on
it will trigger a kASan error message. Additionally, the compiler adds
a redzone on global variables, and we mark these redzones as invalid too.
The illegal-access detection works with a 1-byte granularity.

For now, we cover three areas:

- global variables
- kmem_alloc-ated areas
- malloc-ated areas

More will come, but that's a good start.

Introduce Xen PVH support in GENERIC.
This is compiled in with
options XENPVHVM
x86 changes:
- add Xen section and xen pvh entry points to locore.S. Set vm_guest
to VM_GUEST_XENPVH in this entry point.
Most of the boot procedure (especially page table setup and switch to
paged mode) is shared with native.
- change some x86_delay() to delay_func(), which points to x86_delay() for
native/HVM, and xen_delay() for PVH

Xen changes:
- remove Xen bits from init_x86_64_ksyms() and init386_ksyms()
and move to xen_init_ksyms(), used for both PV and PVH
- set ISA no-legacy-devices property for PVH
- factor out code from Xen's cpu_bootconf() to xen_bootconf()
in xen_machdep.c
- set up a specific pvh_consinit() which starts with printk()
(which uses a simple hypercall that is available early) and switch to
xencons when we can use pmap_kenter_pa().

Redo the page allocator to perform better, especially on multi-core and
multi-socket systems. Proposed on tech-kern. While here:

- add rudimentary NUMA support - needs more work.
- remove now unused "listq" from vm_page.

branches: 1.28.4;
Move initialization code out of efi_probe into efi_init

and call it from cpu_configure

Call _proc0_tss_ldt_init only once, and rename them.

x86: hypervisor detection from FreeBSD for x2APIC support.

KNF a bit, so I don't get scared each time I open a file

branches: 1.24.14;
add include of opt_intrdebug.h.

branches: 1.23.4; 1.23.10; 1.23.14;
Include <sys/device.h> for config_rootfound, instead of relying on it
being included by accident. From Jarle Greipsland in PR 43449.

branches: 1.22.4; 1.22.6; 1.22.8;
ansify function definitions

branches: 1.21.6; 1.21.12;
Share cpu.h between the x86 ports.

Improve x86 tsc handling:

- Ditch the cross-CPU calibration stuff. It didn't work properly, and it's
near impossible to synchronize the CPUs in a running system, because bus
traffic will interfere with any calibration attempt, messing up the
timings.

- Only enable the TSC on CPUs where we are sure it does not drift. If we are
On a known good CPU, give the TSC high timecounter quality, making it the
default.

- When booting CPUs, detect TSC skew and account for it. Most Intel MP
systems have synchronized counters, but that need not be true if the
system has a complicated bus structure. As far as I know, AMD systems
do not have synchronized TSCs and so we need to handle skew.

- While an AP is waiting to be set running, try and make the TSC drift by
entering a reduced power state. If we detect drift, ensure that the TSC
does not get a high timecounter quality. This should not happen and is
only for safety.

- Make cpu_counter() stuff LKM safe.

branches: 1.19.6; 1.19.8; 1.19.10; 1.19.12;
Initialise the Local Vector Table of the primary LAPIC directly after
enabling it. Explicitly initialise LINT0 as ExtInt and LINT1 as NMI,
the platform default. Mask the NMIs on the application processors and
mask the ExtInt if a IOAPIC was found.

With this patch, "disable ioapic" is supposed to work and it will allow
enabling the local APIC on all systems that have one to gain e.g. the
better clock interrupt.

Merge jmcneill-pm branch.

branches: 1.17.2; 1.17.4; 1.17.6;
Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree. Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches. The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

branches: 1.16.8; 1.16.10;
merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.15.12; 1.15.16; 1.15.18; 1.15.24;
convert to timecounters (from branch simonb-timecounters)

branches: 1.14.2; 1.14.8;
Revert yesterday's change that attempted to fix the detection of the
boot device when using a Multiboot boot loader. It couldn't work because
these boot loaders do not pass a checksum of the disk so matchbiosdisk()
cannot really find any matches. I should have gone to sleep before
commiting...

Found by xtraeme@.

branches: 1.13.2;
When booting an i386 kernel with Multiboot, properly detect the boot device
by looking it up in the x86_alldisks table (instead of trying to match it
to 'wd*' manually).

In order to do this, move the cpu_rootconf function from x86 common code
to amd64 and i386 specific one. This way, i386 can do an extra step (call
the appropriate Multiboot code) in the appropriate place (after
x86_matchbiosdisks and before findroot()).

branches: 1.12.4;
merge ktrace-lwp.

branches: 1.11.12;
Centralize the declaration of booted_device and booted_partition.

Move boot device detection code from i386 and amd64 ports to x86_autoconf.c.
Rename i386_alldisks and x86_64_alldisks to x86_alldisks, adjust other
references to compensate.

match_harddisk(): Open RAW_PART, not the booted partition; we're
only performing the ioctl to fetch the disklabel.

There's no need to pass a proc value when using UIO_SYSSPACE with
vn_rdwr(9) and uiomove(9).

OK'd by Jason Thorpe

Replace the traditional buffer memory management -- based on fixed per buffer
virtual memory reservation and a private pool of memory pages -- by a scheme
based on memory pools.

This allows better utilization of memory because buffers can now be allocated
with a granularity finer than the system's native page size (useful for
filesystems with e.g. 1k or 2k fragment sizes). It also avoids fragmentation
of virtual to physical memory mappings (due to the former fixed virtual
address reservation) resulting in better utilization of MMU resources on some
platforms. Finally, the scheme is more flexible by allowing run-time decisions
on the amount of memory to be used for buffers.

On the other hand, the effectiveness of the LRU queue for buffer recycling
may be somewhat reduced compared to the traditional method since, due to the
nature of the pool based memory allocation, the actual least recently used
buffer may release its memory to a pool different from the one needed by a
newly allocated buffer. However, this effect will kick in only if the
system is under memory pressure.

Overhaul MBR handling (part 1):

<sys/bootblock.h>:
* Added definitions for the Master Boot Record (MBR) used by
a variety of systems (primarily i386), including the format
of the BIOS Parameter Block (BPB).
This information was cribbed from a variety of sources
including <sys/disklabel_mbr.h> which this is a superset of.

As part of this, some data structure elements and #defines
were renamed to be more "namespace friendly" and consistent
with other bootblocks and MBR documentation.
Update all uses of the old names to the new names.

<sys/disklabel_mbr.h>:
* Deprecated in favor of <sys/bootblock.h> (the latter is more
"host tool" friendly).

amd64 & i386:
* Renamed /usr/mdec/bootxx_dosfs to /usr/mdec/bootxx_msdos, to
be consistent with the naming convention of the msdosfs tools.

* Removed /usr/mdec/bootxx_ufs, as it's equivalent to bootxx_ffsv1
and it's confusing to have two functionally equivalent bootblocks,
especially given that "ufs" has multiple meanings (it could be
a synonym for "ffs", or the group of ffs/lfs/ext2fs file systems).

* Rework pbr.S (the first sector of bootxx_*):
+ Ensure that BPB (bytes 11..89) and the partition table
(bytes 446..509) do not contain code.
+ Add support for booting from FAT partitions if BOOT_FROM_FAT
is defined. (Only set for bootxx_msdos).
+ Remove "dummy" partition 3; if people want to installboot(8)
these to the start of the disk they can use fdisk(8) to
create a real MBR partition table...
+ Compile with TERSE_ERROR so it fits because of the above.
Whilst this is less user friendly, I feel it's important
to have a valid partition table and BPB in the MBR/PBR.

* Renamed /usr/mdec/biosboot to /usr/mdec/boot, to be consistent
with other platforms.

* Enable SUPPORT_DOSFS in /usr/mdec/boot (stage2), so that
we can boot off FAT partitions.

* Crank version of /usr/mdec/boot to 3.1, and fix some of the other
entries in the version file.

installboot(8) (i386):
* Read the existing MBR of the filesystem and retain the BIOS
Parameter Block (BPB) in bytes 11..89 and the MBR partition
table in bytes 446..509. (Previously installboot(8) would
trash those two sections of the MBR.)

mbrlabel(8):
* Use sys/lib/libkern/xlat_mbr_fstype.c instead of homegrown code
to map the MBR partition type to the NetBSD disklabel type.


Test built "make release" for i386, and new bootblocks verified to work
(even off FAT!).

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

__KERNEL_RCSID()

branches: 1.3.2;
Make sure to include opt_foo.h if a defflag option FOO is used.

don't #include <sys/dkstat.h> where it is (appearently) unused

Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.

Merge into x86/.

Reduce the diff between i386 and amd64 (bios32_service not implemented
there).

search SMBIOS from UEFI configuration table when boot with UEFI.

branches: 1.21.2; 1.21.4;
KNF a bit, so I don't get scared each time I open a file

branches: 1.20.2; 1.20.16;
Avoid empty loop bodies.

branches: 1.19.12; 1.19.16;
Add a flags argument to pmap_kenter_pa(9).
Patch showed on tech-kern@ http://mail-index.netbsd.org/tech-kern/2009/11/04/msg006434.html
No objections.

KNF, same object code generated

bcopy -> memcpy

ansify function definitions

Make SMBIOS rev. message use aprint_debug (matches i386)

branches: 1.14.2;
remove unused malloc.h

branches: 1.13.8;
Remove clause 3 and 4 from TNF licenses

branches: 1.12.2; 1.12.4;
use POSIX integer types

branches: 1.11.10;
Merge jmcneill-pm branch.

branches: 1.10.2; 1.10.4;
aprintify

Pull up the bouyer-xenamd64 branch to HEAD. This brings in amd64 support
to NetBSD/Xen, both Dom0 and DomU.

branches: 1.8.2;
Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree. Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches. The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

branches: 1.7.4;
Merge most x86 changes from the vmlocking branch, except the threaded soft
interrupt stuff. This is mostly comprised of changes to the pmap modules to
work on multiprocessor systems without kernel_lock, and changes to speed up
tlb shootdowns.

branches: 1.6.4; 1.6.8; 1.6.10;
Add (commented-out) support for IPMI on amd64--pretty much copied straight
from i386.

branches: 1.5.2; 1.5.4; 1.5.10;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.4.24; 1.4.26; 1.4.30;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

branches: 1.2.16;
__KERNEL_RCSID()

branches: 1.1.2;
Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.

Remove bpf_jit which was ported from FreeBSD recently.

It will soon be replaced with the new bpfjit kernel module.

branches: 1.1.2;
Add BPF JIT compiler, currently supporting amd64 and i386. Code obtained
from FreeBSD. Also, make few BPF fixes and simplifications while here.
Note that bpf_jit_enable is false for now.

OK dyoung@, some feedback from matt@

Remove bpf_jit which was ported from FreeBSD recently.

It will soon be replaced with the new bpfjit kernel module.

branches: 1.1.2;
Add BPF JIT compiler, currently supporting amd64 and i386. Code obtained
from FreeBSD. Also, make few BPF fixes and simplifications while here.
Note that bpf_jit_enable is false for now.

OK dyoung@, some feedback from matt@

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
- "shad", to track uninitialized memory with a bit granularity (1:1).
Each bit set to 1 in the shad corresponds to one uninitialized bit of
real kernel memory.
- "orig", to track the origin of the memory with a 4-byte granularity
(1:1). Each uint32_t cell in the orig indicates the origin of the
associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
- a code designating the type of memory (Stack, Pool, etc), and
- a compressed pointer, which points either (1) to a string containing
the name of the variable associated with the cell, or (2) to an area
in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

branches: 1.11.34;
Use END() to set ELF symbol size.

branches: 1.10.4; 1.10.14;
The %dx argument of in/out is not a memory reference, so don't use it as
such.

branches: 1.9.2;
As Andrew Doran points out, _ALIGN_TEXT is unused, and to test that
X86_BUS_SPACE_IO equals 0 is no longer necessary. Get rid of the
_ALIGN_TEXT definition, and do not "assert" that X86_BUS_SPACE_IO == 0.

On x86, change the bus_space_tag_t to a pointer to a struct
bus_space_tag. For now, bus_space_tag's only member is
bst_type, the type of space, which is either X86_BUS_SPACE_IO
or X86_BUS_SPACE_MEM. In the future, new bus_space_tag members
will refer to override-functions installed by a new function,
bus_space_tag_create(9).

Add pointers to constant struct bus_space_tag, x86_bus_space_io and
x86_bus_space_mem. Use them to replace most uses of X86_BUS_SPACE_IO
and X86_BUS_SPACE_MEM.

Add an x86-specific bus_space_is_equal(9) implementation that compares
the two tags' bst_type.

branches: 1.7.12; 1.7.18; 1.7.20;
enable profiling of assembly functions.

branches: 1.6.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.5.2;
Add 64-bit bus_space methods.

branches: 1.4.14; 1.4.16;
Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.

branches: 1.3.2; 1.3.4; 1.3.6; 1.3.8; 1.3.10; 1.3.12; 1.3.14;
Fix inverted tests. VT switching works again.

Shuffle registers a bit to make bus_space_write_region_N work again.

x86 changes for pcc and LKMs.

- Replace most inline assembly with proper functions. As a side effect
this reduces the size of amd64 GENERIC by about 120kB, and i386 by a
smaller amount. Nearly all of the inlines did something slow, or something
that does not need to be fast.
- Make curcpu() and curlwp functions proper, unless __GNUC__ && _KERNEL.
In that case make them inlines. Makes curlwp LKM and preemption safe.
- Make bus_space and bus_dma more LKM friendly.
- Share a few more files between the ports.
- Other minor changes.

Save maxv@ some work and put back the compat_13_sigreturn changes that allow
amd64 to run ancient i386 binaries.

Drop COMPAT_13 on amd64, already not enabled. Reduces the number of
critical places.

branches: 1.1.2;
Move the empty sigreturn functions to their own files to allow the compat
module to load by itself.

branches: 1.5.2; 1.5.18;
Move the empty sigreturn functions to their own files to allow the compat
module to load by itself.

remove sigcontext-style signal delivery code

Don't use err and trapno from a copied in mcontext.

Do checks on passed in context values (sigreturn/setcontext) explicitly;
can't rely on catching the trap on iret for a few reasons.

SIGINFO support.
Todo: 32bit compat support (COMPAT_NETBSD32 will not compile right now,
as it won't on other platforms).

s/acually/actually/ in comment.

P.S. the comment itself is hardly readable though.

branches: 1.35.6;
Typo in comment

Make copystr() a MI C function, part of libkern and shared on all
architectures.

Notes:

- On alpha and ia64 the function is kept but gets renamed locally to avoid
symbol collision. This is because on these two arches, I am not sure
whether the ASM callers do not rely on fixed registers, so I prefer to
keep the ASM body for now.
- On Vax, only the symbol is removed, because the body is used from other
functions.
- On RISC-V, this change fixes a bug: copystr() was just a wrapper around
strlcpy(), but strlcpy() makes the operation less safe (strlen on the
source beyond its size).
- The kASan, kCSan and kMSan wrappers are removed, because now that
copystr() is in C, the compiler transformations are applied to it,
without the need for manual wrappers.

Could test on amd64 only, but should be fine.

Hum. Fix a potentially catastrophic bug: kcopy() sets DF=1 if the areas
overlap, but doesn't clear it if the copy faults. If this happens, we
return to the caller with DF=1, and each future memory copy will be
backwards.

I wonder if there really are places where kcopy() is called with
overlapping areas.

Fix a think-o / paste-o in the max user address check, pointed out
by maxv@.

Overhaul the API used to fetch and store individual memory cells in
userspace. The old fetch(9) and store(9) APIs (fubyte(), fuword(),
subyte(), suword(), etc.) are retired and replaced with new ufetch(9)
and ustore(9) APIs that can return proper error codes, etc. and are
implemented consistently across all platforms. The interrupt-safe
variants are no longer supported (and several of the existing attempts
at fuswintr(), etc. were buggy and not actually interrupt-safe).

Also augmement the ucas(9) API, making it consistently available on
all plaforms, supporting uniprocessor and multiprocessor systems, even
those that do not have CAS or LL/SC primitives.

Welcome to NetBSD 8.99.37.

Drop NENTRY() from the x86 kernels, use ENTRY(). With PMCs (and other hardware
tracing facilities) we have a much better ways of monitoring the CPU activity
than GPROF, without software modification.

Also I think GPROF has never worked, because the 'start' functions of both
i386 and amd64 use ENTRY(), and it would have caused a function call while the
kernel was not yet relocated.

branches: 1.29.2; 1.29.4;
Implement a real hotpatch feature.

Define a HOTPATCH() macro, that puts a label and additional information
in the new .rodata.hotpatch kernel section. In patch.c, scan the section
and patch what needs to be. Now it is possible to hotpatch the content of
a macro.

SMAP is switched to use this new system; this saves a call+ret in each
kernel entry/exit point.

Many other operating systems do the same.

Don't fall through functions, explicitly jump instead. While here don't
call smap_enable twice (harmless), and add END() markers.

Always use END() markers when declaring functions in assembly, so that ld
can compute the size of the functions. A few remain.

While here, fix a bug in the INTRSTUB macro: we are falling through
resume_, but it is aligned, so it looks like we're executing the inter-
function padding - which probably happens to contain NOPs, but that's
still bad.

fix comment, rdx, not edx

Add support for SMAP on amd64.

PSL_AC is cleared from %rflags in each kernel entry point. In the copy
sections, a copy window is opened and the kernel can touch userland
pages. This window is closed when the kernel is done, either at the end
of the copy sections or in the fault-recover functions.

This implementation is not optimized yet, due to the fact that INTRENTRY
is a macro, and we can't hotpatch macros.

Sent on tech-kern@ a month or two ago, tested on a Kabylake.

Move incq outside of the copy section. No functional change, reduces
my smap diff.

Split comment, otherwise it is misleading. kcopy operates on kernel
memory, and must *not* be used with userland pages.

style, reduces an incoming diff

Fix a bug in ucas_32 and ucas_64. There is a branch where they don't
initialize %rax.

branches: 1.20.10;
KNF, and use C-style comments. Also, remove fusword/susword.

Remove the amd64 implementation of fuword and suword. They are not used in
the MI+amd64 code - Christos replaced them yesterday by copy*.

They are both buggy:
- suword does not properly check the userspace limit: 64 bits are copied,
but the max address checked is VM_MAXUSER_ADDRESS-4, which means that 4
bytes may overflow. Reported by Ed Schouten.
- fuword is supposed to be symmetrical with suword. But it uses 32bit
registers, so it stores 32bit values! Spotted by Chuck (chs@).

branches: 1.18.14; 1.18.18; 1.18.20; 1.18.28; 1.18.34; 1.18.36; 1.18.38; 1.18.40;
fix more bugs in ucas_*():
move the "old" value to %*ax so that cmpxchg* can work.
remove the PCB_ONFAULT stuff, onfault_handler() handles this already.
return the error from fault handler in ucas_fault rather than forcing EFAULT.

Fix ucas_32/ucas_64 on amd64.

branches: 1.16.2; 1.16.4;
- Use uvm_lwp_setuarea() instead of directly setting address to lwp_t::l_addr.
- Replace most remaining uses of l_addr with uvm_lwp_getuarea() or lwp_getpcb().
- Amend assembly in ports where it accesses PCB via struct user.
- Rename L_ADDR to L_PCB in few places. Reduce sys/user.h inclusions.

tlbstate is 'int', so use 'cmpl' not 'cmpq'.
Fixes gprof on amd64 PR/40960.

Change amd64 fault handler to check instruction pointer for copyin/out et al
functions, like in i386. Avoids setting pcb_onfault, saves few instructions.

Add ucas (CAS for user-space address) support for i386 and amd64.
API provides ucas_int() and ucas_ptr() for now.

Reviewed by <ad>.

branches: 1.12.2; 1.12.4; 1.12.8;
Annotate some of the assembler.
Remove 2 redundant register-register moves.

branches: 1.11.4;
fillw is dead.

branches: 1.10.2;
Make kernel pagefault handling preeemption safe.

branches: 1.9.6; 1.9.8; 1.9.10;
Reorder code so that we don't default-mispredict the 'deferred switch'
branch on (almost) every copy-in/out.

Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.

branches: 1.7.2;
merge yamt-x86pmap branch.

- reduce differences between amd64 and i386. notably, share pmap.c
between them. it makes several i386 pmap improvements available to
amd64, including tlb shootdown reduction and bug fixes from Stephan Uphoff.
- implement deferred pmap switching for amd64.
- remove LARGEPAGES option. always use large pages if available.
also, make it work on amd64.

Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree. Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches. The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

branches: 1.5.8; 1.5.10; 1.5.12; 1.5.14;
merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.4.2; 1.4.4; 1.4.10;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.3.26;
merge ktrace-lwp.

branches: 1.2.4;
Check the passed in address as well as determining the maximum length
using VM_MAXUSER_ADDRESS in copyinstr and copyoutstr.

Problem originally fixed in OpenBSD/i386.
This fix suggested by Charles Hannum (mycroft at netbsd dot org).

branches: 1.1.2; 1.1.4; 1.1.8; 1.1.10; 1.1.12;
Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.

Merge/move core_machdep.c into x86, no difference between i386 and amd64.

cpu_coredump: pass UIO_SYSSPACE to coredump_write(), not UIO_USERSPACE.

branches: 1.2.2; 1.2.4; 1.2.6;
remove unused malloc.h

branches: 1.1.4;
Make the emulations, exec formats, coredump, NFS, and the NFS server
into modules. By and large this commit:

- shuffles header files and ifdefs
- splits code out where necessary to be modular
- adds module glue for each of the components
- adds/replaces hooks for things that can be installed at runtime

Merge most x86 changes from the vmlocking branch, except the threaded soft
interrupt stuff. This is mostly comprised of changes to the pmap modules to
work on multiprocessor systems without kernel_lock, and changes to speed up
tlb shootdowns.

branches: 1.17.4; 1.17.8; 1.17.10;
Ansify, de-__P(), unregister.

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.15.2; 1.15.4; 1.15.10;
clean up how cpus and ioapics are attached at the mainbus:
Seperate "cpubus" and "ioapicbus" -- while they share a common "address
space" (the apic id), the kernel doesn't use this fact. There are different
data passed to cpus and apics, which caused some ugly polymorphism. This
also saves the special "submatch" functions needed to distingush cpus
and ioapics for autoconf. (And it makes that "apid" locators wired
in the kernel configuration are honored now; this allows one to dumb down
an mp box to singleprocessor by userconfig.)
Print "apid" locators in the buses "print" function "as everyone does",
so the per-port cpu drivers don't need to do it.
Being here, constify "struct cpu_functions" and g/c the unused MP_PICMODE
flag.

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.13.2;
Merge newlock2 to head.

branches: 1.12.4;
revert previous (executable mapping of data needed for secondary CPU
spinup), this is not needed anymore

Map kernel stack and "struct cpuinfo" of secondary processors with
execute permission. (I don't understand yet why the latter is needed.)
This fixes crashes of MP kernels reported by Kurt Schreiner and in
the context of PR port-amd64/33927 (which is about a different problem
otherwise).
MP kernels reported to boot again by Dave Huang and Kurt Schreiner.

branches: 1.10.2;
convert to timecounters (from branch simonb-timecounters)

branches: 1.9.4; 1.9.6; 1.9.8; 1.9.14;
merge ktrace-lwp.

branches: 1.8.2;
merge yamt-km branch.
- don't use managed mappings/backing objects for wired memory allocations.
save some resources like pv_entry. also fix (most of) PR/27030.
- simplify kernel memory management API.
- simplify pmap bootstrap of some ports.
- some related cleanups.

branches: 1.7.4; 1.7.6;
Set the double fault stack pointer at the end of the stack.

Make the IDT read-only. It's only mapped R/W when really needed. Could
catch disasters such as overwriting it. Should probably be made
conditional on DIAGNOSTIC later, but there's no time-critical code
involved here.

Move the double fault stack away from the IPI stack. It's now shared
between CPUs, but that's not a big deal; double faults are fatal
and can't be recovered from.

Compile TSC support code when __x86_64__ is defined.

branches: 1.4.2;
use no-execute page permissions if supported

Uppercase CPU, plural is CPUs.

__KERNEL_RCSID()

branches: 1.1.2;
Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.

Since cpu_in_cksum.S can be built outside of the kernel, add an ugly #ifdef
_KERNEL for kMSan.

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
- "shad", to track uninitialized memory with a bit granularity (1:1).
Each bit set to 1 in the shad corresponds to one uninitialized bit of
real kernel memory.
- "orig", to track the origin of the memory with a 4-byte granularity
(1:1). Each uint32_t cell in the orig indicates the origin of the
associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
- a code designating the type of memory (Stack, Pool, etc), and
- a compressed pointer, which points either (1) to a string containing
the name of the variable associated with the cell, or (2) to an area
in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

branches: 1.3.18;
handle PIC compilation (if we are building a PIE system; this is used by tests)

branches: 1.2.10;
Use END() to set ELF symbol size. Set local string type/size too.

branches: 1.1.2; 1.1.4; 1.1.12; 1.1.50; 1.1.60;
Refactor in_cksum/in4_cksum/in6_cksum implementations:
- All three functions are included in the kernel by default.
They call a backend function cpu_in_cksum after possibly
computing the checksum of the pseudo header.
- cpu_in_cksum is the core to implement the one-complement sum.
The default implementation is moderate fast on most platforms
and provides a 32bit accumulator with 16bit addends for L32 platforms
and a 64bit accumulator with 32bit addends for L64 platforms.
It handles edge cases like very large mbuf chains (could happen with
native IPv6 in the future) and provides a good base for new native
implementations.
- Modify i386 and amd64 assembly to use the new interface.

This disables the MD implementations on !x86 until the conversion is
done. For Alpha, the portable version is faster.

paravirt_membar_sync(9): New memory barrier.

For use in paravirtualized drivers which require store-before-load
ordering -- irrespective of whether the kernel is built for a single
processor, or whether the (virtual) machine is booted with a single
processor.

This is even required on architectures that don't even have a
store-before-load ordering barrier, like m68k; adding, e.g., a virtio
bus is _as if_ the architecture has been extended with relaxed memory
ordering when talking with that new bus. Such architectures need
some way to request the hypervisor enforce that ordering -- on m68k,
that's done by issuing a CASL instruction, which qemu maps to an
atomic r/m/w with sequential consistency ordering in the host.

PR kern/59618: occasional virtio block device lock ups/hangs

branches: 1.69.2;
x86: Expose dtrace_smap_enable/disable symbols even under XENPV.

They're no-ops in that case, but this enables the dtrace modules to
load and work.

PR port-xen/58373: Missing KDTRACE_HOOKS in Xen kernels

xen: Don't hotpatch away LOCK prefix in xen_mb, even on UP boots.

Both xen_mb and membar_sync are designed to provide store-before-load
ordering, but xen_mb has to provide it in synchronizing guest with
hypervisor, while membar_sync only has to provide it in synchronizing
one (guest) CPU with another (guest) CPU.

It is safe to hotpatch away the LOCK prefix in membar_sync on a
uniprocessor boot because membar_sync is only designed to coordinate
between normal memory on multiple CPUs, and is never necessary when
there's only one CPU involved.

But xen_mb is used to coordinate between the guest and the `device'
implemented by a hypervisor, which might be running on another
_physical_ CPU even if the NetBSD guest only sees one `CPU', i.e.,
one _virtual_ CPU. So even on `uniprocessor' boots, xen_mb must
still issue an instruction with store-before-load ordering on
multiprocessor systems, such as a LOCK ADD (or MFENCE, but MFENCE is
costlier for no benefit here).

No need to change xen_wmb (release ordering, load/store-before-store)
or xen_rmb (acquire ordering, load-before-load/store) because every
x86 store is a store-release and every x86 load is a load-acquire,
even on multiprocessor systems, so there's no hotpatching involved
anyway.

PR kern/57199

branches: 1.67.6;
dtrace: add support for SMAP

Add support in dtrace for SMAP, so that actions like copyinstr() work.
It would be better if dtrace could use the SMAP_* hotpatch macros directly,
but the hotpatching code does not currently operate on kernel modules,
so we'll use some tiny functions in the base kernel for now.

Eliminate l->l_ncsw and l->l_nivcsw. From memory think they were added
before we had per-LWP struct rusage; the same is now tracked there.

branches: 1.65.18;
Introduce smap_enable()/smap_disable() functions, to be used from
C code.

branches: 1.64.2;
Revert most of ad's movs/stos change. Instead do a lot simpler: declare
svs_quad_copy() used by SVS only, with no need for instrumentation, because
SVS is disabled when sanitizers are on.

remove unused x86_stos

Nix trailing whitespace.

Serialize rdtsc using with lfence, mfence or cpuid to read TSC more precisely.

x86/x86/tsc.c rev. 1.67 reduced cache problem and got big improvement, but it
still has room. I measured the effect of lfence, mfence, cpuid and rdtscp.
The impact to TSC skew and/or drift is:

AMD: mfence > rdtscp > cpuid > lfence-serialize > lfence = nomodify
Intel: lfence > rdtscp > cpuid > nomodify

So, mfence is the best on AMD and lfence is the best on Intel. If it has no
SSE2, we can use cpuid.

NOTE:
- An AMD's document says DE_CFG_LFENCE_SERIALIZE bit can be used for
serializing, but it's not so good.
- On Intel i386(not amd64), it seems the improvement is very little.
- rdtscp instruct can be used as serializing instruction + rdtsc, but
it's not good as [lm]fence. Both Intel and AMD's document say that
the latency of rdtscp is bigger than rdtsc, so I suspect the difference
of the result comes from it.

Print a rate limited warning if the TSC timecounter goes backwards from the
viewpoint of any single LWP.

Reported-by: syzbot+6dd5a230d19f0cbc7814@syzkaller.appspotmail.com

Instrument STOS/MOVS for KMSAN to unbreak it.

- mismatched END pointed out by maxv@
- ditch the frame, tracer should be able to deal without it in leaf on x86_64

- Add a couple of wrapper functions around STOS and MOVS and use them to zero
and copy PTEs in preference to memset()/memcpy().

- Remove related SSE / pageidlezero stuff.

this is kmsan

Deal with KMSAN fussiness. Pointed out by msaitoh@.

xen needs the TSC funcs too

Make cpu_counter(), cpu_counter32() and tsc_get_timecount() into a single
preemption-safe routine.

cpu_counter: only need to clear %eax (zero extends).

Move x86_hotpatch() in !XENPV section. Fixes XEN3* builds.

Modify the hotpatch mechanism, in order to make it much less ROP-friendly.

Currently x86_patch_window_open is a big problem, because it is a perfect
function to inject/modify executable code with ROP.

- Remove x86_patch_window_open(), along with its x86_patch_window_close()
counterpart.
- Introduce a read-only link-set of hotpatch descriptor structures,
which reference a maximum of two read-only hotpatch sources.
- Modify x86_hotpatch() to open a window and call the new
x86_hotpatch_apply() function in a hard-coded manner.
- Modify x86_hotpatch() to take a name and a selector, and have
x86_hotpatch_apply() resolve the descriptor from the name and the
source from the selector, before hotpatching.
- Move the error handling in a separate x86_hotpatch_cleanup() function,
that gets called after we closed the window.

The resulting implementation is a bit complex and non-obvious. But it
gains the following properties: the code executed in the hotpatch window
is strictly hard-coded (no callback and no possibility to execute your own
code in the window) and the pointers this code accesses are strictly
read-only (no possibility to forge pointers to hotpatch an area that was
not designated as hotpatchable at compile-time, and no possibility to
choose what bytes to write other than the maximum of two read-only
templates that were designated as valid for the given destination at
compile-time).

With current CPUs this slightly improves a situation that is already
pretty bad by definition on x86. Assuming CET however, this change closes
a big hole and is kinda great.

The only ~problem there is, is that dtrace-fbt tries to hotpatch random
places with random bytes, and there is just no way to make it safe.
However dtrace is only in a module, that is rarely used and never compiled
into the kernel, so it's not a big problem; add a shitty & vulnerable
independent hotpatch window in it, and leave big XXXs. It looks like fbt
is going to collapse soon anyway.

mi_userret(): take care of calling preempt(), set spc_curpriority directly,
and remove MD code that does the same.

Remove the ins* and outs* functions. Not sanitizer-friendly, and unused
anyway.

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
- "shad", to track uninitialized memory with a bit granularity (1:1).
Each bit set to 1 in the shad corresponds to one uninitialized bit of
real kernel memory.
- "orig", to track the origin of the memory with a 4-byte granularity
(1:1). Each uint32_t cell in the orig indicates the origin of the
associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
- a code designating the type of memory (Stack, Pool, etc), and
- a compressed pointer, which points either (1) to a string containing
the name of the variable associated with the cell, or (2) to an area
in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

More inlined ASM.

Merge amd64func.S into cpufunc.S, and clean up.

Convert rdmsr_locked and wrmsr_locked to inlines.

branches: 1.43.2;
More inlines, prerequisites for future changes. Also, remove fngetsw(),
which was a duplicate of fnstsw().

Inline x86_cpuid2(), prerequisite for future changes. Also, add "memory"
on certain other inlines, to make sure GCC does not reorder.

Add PCID support in SVS. This avoids TLB flushes during kernel<->user
transitions, which greatly reduces the performance penalty introduced by
SVS.

We use two ASIDs, 0 (kern) and 1 (user), and use invpcid to flush pages
in both ASIDs.

The read-only machdep.svs.pcid={0,1} sysctl is added, and indicates whether
SVS+PCID is in use.

Misc changes in the x86 FPU code. Reduces a future diff. No real functional
change.

More inlined ASM. While here switch to proper types.

Start converting the x86 CPU functions to inlined ASM. Matters for NVMM,
where some are invoked millions of times.

Remove unused functions and reorder a little.

We reorganise definitions for XEN source support as follows:

XEN - common sources required for baseline XEN support.
XENPV - sources required for support of XEN in PV mode.
XENPVHVM - sources required for support for XEN in HVM mode.
XENPVH - sources required for support for XEN in PVH mode.

Rollback http://mail-index.netbsd.org/source-changes/2018/12/22/msg101629.html

This change breaks module loading due to weak alias being unsupported
in the kernel module linker.

Requested by maxv@ and others as it affects their work.

No immediate decision on a replacement method is available, but other options
suggested include pre-processing, conditional compilation (#ifdef etc) and other
source level methods to avoid linktime decision making.

Introduce a weak alias method of exporting different implementations
of the same API.

For eg: the amd64 native implementation of invlpg() now becomes
amd64_invlpg() with a weak symbol export of invlpg(), while the XEN
implementation becomes xen_invlpg(), also weakly exported as invlpg()

Note that linking in both together without having an override function
named invlpg() would be a mistake, as we have limited control over
which of the two options would emerge as the finally exported invlpg()
resulting in a potential situation where the wrong function is finally
exported. This change avoids this situation.

We should however include an override function invlpg() in that case,
such that it is able to then pass on the call to the appropriate
backing function (amd64_invlpg() in the case of native, and
xen_invlpg() in the case of under XEN virtualisation) at runtime.

This change does not introduce such a function and therefore does not
alter builds to include native as well as XEN implementations in the
same binary. This will be done later, with the introduction of XEN
PVHVM mode, where precisely such a runtime switch is required.

There are no operational changes introduced by this change.

More ASLR. Randomize the location of the direct map at boot time on amd64.
This doesn't need "options KASLR" and works on GENERIC. Will soon be
enabled by default.

The location of the areas is abstracted in a slotspace structure. Ideally
we should always use this structure when touching the L4 slots, instead of
the current cocktail of global variables and constants.

machdep initializes the structure with the default values, and we then
randomize its dmap entry. Ideally machdep should randomize everything at
once, but in the case of the direct map its size is determined a little
later in the boot procedure, so we're forced to randomize its location
later too.

Drop NENTRY() from the x86 kernels, use ENTRY(). With PMCs (and other hardware
tracing facilities) we have a much better ways of monitoring the CPU activity
than GPROF, without software modification.

Also I think GPROF has never worked, because the 'start' functions of both
i386 and amd64 use ENTRY(), and it would have caused a function call while the
kernel was not yet relocated.

branches: 1.31.2; 1.31.4;
Don't fall through functions, explicitly jump instead. While here don't
call smap_enable twice (harmless), and add END() markers.

Always use END() markers when declaring functions in assembly, so that ld
can compute the size of the functions. A few remain.

While here, fix a bug in the INTRSTUB macro: we are falling through
resume_, but it is aligned, so it looks like we're executing the inter-
function padding - which probably happens to contain NOPs, but that's
still bad.

Add setds and setes, will be useful in the future.

Add setusergs on Xen, and simplify.

branches: 1.27.8;
Add accessors for available x86 Debug Registers

There are 8 Debug Registers on i386 (available at least since 80386) and 16
on AMD64. Currently DR4 and DR5 are reserved on both cpu-families and
DR9-DR15 are still reserved on AMD64. Therefore add accessors for DR0-DR3,
DR6-DR7 for all ports.

Debug Registers x86:
* DR0-DR3 Debug Address Registers
* DR4-DR5 Reserved
* DR6 Debug Status Register
* DR7 Debug Control Register
* DR8-DR15 Reserved

Access the registers is available only from a kernel (ring 0) as there is
needed top protected access. For this reason there is need to use special
XEN functions to get and set the registers in the XEN3 kernels.

XEN specific functions as defined in NetBSD:
- HYPERVISOR_get_debugreg()
- HYPERVISOR_set_debugreg()

This code extends the existing rdr6() and ldr6() accessor for additional:
- rdr0() & ldr0()
- rdr1() & ldr1()
- rdr2() & ldr2()
- rdr3() & ldr3()
- rdr7() & ldr7()

Traditionally accessors for DR6 were passing vaddr_t argument, while it's
appropriate type for DR0-DR3, DR6-DR7 should be using u_long, however it's
not a big deal. The resulting functionality should be equivalent so stick
to this convention and use the vaddr_t type for all DR accessors.

There was already a function defined for rdr6() in XEN, but it had a nit on
AMD64 as it was casting HYPERVISOR_get_debugreg() to u_int (32-bit on
AMD64), truncating result. It still works for DR6, but for the sake of
simplicity always return full 64-bit value.

New accessors duplicate functionality of the dr0() function available on
i386 within the KSTACK_CHECK_DR0 option. dr0() is a specialized layer with
logic to set appropriate types of interrupts, now accessors are designed to
pass verbatim values from user-land (with simple sanity checks in the
kernel). At the moment there are no plans to make possible to coexist
KSTACK_CHECK_DR0 with debug registers for user applications (debuggers).

options KSTACK_CHECK_DR0
Detect kernel stack overflow using DR0 register. This option uses DR0
register exclusively so you can't use DR0 register for other purpose
(e.g., hardware breakpoint) if you turn this on.

The KSTACK_CHECK_DR0 functionality was designed for i386 and never ported
to amd64.

Code tested on i386 and amd64 with kernels: GENERIC, XEN3_DOMU, XEN3_DOM0.

Sponsored by <The NetBSD Foundation>

Fix rdr6() function on amd64

According to the AMD64 SysV ABI the first returned value is passed in RAX,
not in RDI. Actually RDI is used for the first argument passed to a
function.

So far this function was dead code, it will be used for a ptrace(2)
feature to support CPU watchpoints.

The rdr6() function reads state of the DR6 register and returns its value.

Sponsored by <The NetBSD Foundation>

branches: 1.25.6; 1.25.10;
Change i386 to use x86/fpu.c instead of i386/isa/npx.c
This changes the trap10 and trap13 code to call directly into fpu.c,
removing all the code for T_ARITHTRAP, T_XMM and T_FPUNDA from i386/trap.c
Not all of the code thate appeared to handle fpu traps was ever called!
Most of the changes just replace the include of machine/npx.h with x86/fpu.h
(or remove it entirely).

Best if x86_stmxcsr executes stmxcsr.

Add x86_stmxcsr for amd64.

Add some definitions for cpu 'extended state'.
These are needed for support of the AVX SIMD instructions.
Nothing yet uses them.

branches: 1.21.2; 1.21.12; 1.21.16;
White space fix.

Import rdmsr_safe(msr, *value) for x86 world. It allows reading MSRs
in a safe way by handling the fault that might trigger for certain
register <> CPU/arch combos.

Requested by Jukka. Patch adapted from one found in DragonflyBSD.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.18.2;
Be explicit about the member of the fld family wanted here.

branches: 1.17.2; 1.17.4;
add the guts of TLS support on amd64. based on joerg's patch,
reworked by me to support 32-bit processes as well.
we now keep %fs and %gs loaded with the user values
while in the kernel, which means we don't need to
reload them when returning to user mode.

branches: 1.16.2; 1.16.4;
Fix up mwait/monitor now that gas has been fixed.

branches: 1.15.6; 1.15.10; 1.15.16; 1.15.20;
getss -> x86_getss

branches: 1.14.2;
enable profiling of assembly functions, except for x86_pause().
profiling that one causes the system with profiling on to become so slow
that we get spinlock timeouts.

Don't reload LDTR unless a new value, which only happens for USER_LDT.

Take skew into account for cpu_counter().

Improve x86 tsc handling:

- Ditch the cross-CPU calibration stuff. It didn't work properly, and it's
near impossible to synchronize the CPUs in a running system, because bus
traffic will interfere with any calibration attempt, messing up the
timings.

- Only enable the TSC on CPUs where we are sure it does not drift. If we are
On a known good CPU, give the TSC high timecounter quality, making it the
default.

- When booting CPUs, detect TSC skew and account for it. Most Intel MP
systems have synchronized counters, but that need not be true if the
system has a complicated bus structure. As far as I know, AMD systems
do not have synchronized TSCs and so we need to handle skew.

- While an AP is waiting to be set running, try and make the TSC drift by
entering a reduced power state. If we detect drift, ensure that the TSC
does not get a high timecounter quality. This should not happen and is
only for safety.

- Make cpu_counter() stuff LKM safe.

branches: 1.10.2;
Add support for kernel preeemption to the i386 and amd64 ports. Notes:

- I have seen one isolated panic in the x86 pmap, but otherwise i386
seems stable with preemption enabled.

- amd64 is missing the FPU handling changes and it's not yet safe to
enable it there.

- The usual level for kern.sched.kpreempt_pri will be 128 once enabled
by default. For testing, setting it to 0 helps to shake out bugs.

Remove clause 3 and 4 from TNF licenses

branches: 1.8.2;
+lcr2

branches: 1.7.6; 1.7.8;
breakpoint: set up a stack frame so not to confuse gdb.

add x86_cpuid2, which can specify ecx register.

- Make __cpu_simple_lock and similar real functions and patch at runtime.
- Remove old x86 atomic ops.
- Drop text alignment back to 16 on i386 (really, this time).
- Minor cleanup.

branches: 1.4.4;
Correct argument shuffling in the string I/O functions.

branches: 1.3.2;
Pull up the bouyer-xenamd64 branch to HEAD. This brings in amd64 support
to NetBSD/Xen, both Dom0 and DomU.

Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.

branches: 1.1.2; 1.1.4; 1.1.6; 1.1.8; 1.1.10; 1.1.12; 1.1.14;
x86 changes for pcc and LKMs.

- Replace most inline assembly with proper functions. As a side effect
this reduces the size of amd64 GENERIC by about 120kB, and i386 by a
smaller amount. Nearly all of the inlines did something slow, or something
that does not need to be fast.
- Make curcpu() and curlwp functions proper, unless __GNUC__ && _KERNEL.
In that case make them inlines. Makes curlwp LKM and preemption safe.
- Make bus_space and bus_dma more LKM friendly.
- Share a few more files between the ports.
- Other minor changes.

ddb/amd64: Don't go out of the way to detect invalid addresses.

db_disasm had logic to detect invalid addresses before trying to
disassemble them. But when disassembling a null instruction address,
the logic to detect invalid addresses itself tried to dereference an
invalid address.

db_get_value can already handle this situation gracefully, so there is
no need for this faulty fault-avoidance logic.

Fixes double-fault in ddb on calling null function pointers. With
any luck, this should make diagnosing such bugs easier in the future!

branches: 1.27.4; 1.27.16; 1.27.18;
Start replacing the x86 PTE bits.

- add or adjust /* FALLTHROUGH */ where appropriate
- add __unreachable() after functions that can return but won't in
this case, and thus can't be marked __dead easily

Work around dumb KASSERT in vtopte(), the PTE area can now be above the
MAIN area. I guess the KASSERT should be removed because it doesn't make
a lot of sense.

branches: 1.24.2; 1.24.4;
CID-1364351: Fix uninitialized warnings.

Add support for clac and stac instructions.

When in Rome... use true or false, instead of TRUE and FALSE.

Catch up FreeBSD and OpenBSD's changes. Not completed but it's better than
before:
- Add support rdrand, rdseed, fxrstor, ldmxcsr, stmxcsr, xsare, xrstor,
xsaveopt, cflush and some others.
- Check REX bit correctly.
- Print correct register.
- Fix a lot of bugs.
- KNF.

branches: 1.20.2;
Add support for sysenter, sysexit, vmptrld, vmptrst and getsec.

Fix bugs:
- aaa and daa were reversed. Same as *BSDs.
- fix operand order of shld and shrd. Same as *BSDs.

No functional change.
- 0 -> NULL
- Don't use cast.
- Style fix.
- Fix typos in comment.

KNF. No binary change.

branches: 1.16.12; 1.16.28;
Retire varargs.h support. Move machine/stdarg.h logic into MI
sys/stdarg.h and expect compiler to provide proper builtins, defaulting
to the GCC interface. lint still has a special fallback.
Reduce abuse of _BSD_VA_LIST_ by defining __va_list by default and
derive va_list as required by standards.

Merge db_trace for x86. From: Vladimir Kirillov proger at wilab dot org dot ua

fix misinterpretation of REX prefixes where use of the accumulator
as operand is hardwired into the instruction code,
mostly from Wolfgang Stukenbrock per PR port-amd64/44405

branches: 1.13.4; 1.13.6; 1.13.8;
ANSIfy another 1261 function definitions.
The only ones left in sys are beyond by sed script!
(or in sys/dist or sys/external)
Mostly they have function pointer parameters.

Change about 4500 of the K&R function definitions to ANSI ones.
There are still about 1600 left, but they have ',' or /* ... */
in the actual variable definitions - which my awk script doesn't handle.
There are also many that need () -> (void).
(The script does handle misordered arguments.)

branches: 1.11.10; 1.11.18; 1.11.24;
add swapgs.

Decode cmpxchg8b correctly.

branches: 1.9.2; 1.9.4; 1.9.8;
merge yamt-x86pmap branch.

- reduce differences between amd64 and i386. notably, share pmap.c
between them. it makes several i386 pmap improvements available to
amd64, including tlb shootdown reduction and bug fixes from Stephan Uphoff.
- implement deferred pmap switching for amd64.
- remove LARGEPAGES option. always use large pages if available.
also, make it work on amd64.

Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree. Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches. The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

branches: 1.7.8; 1.7.10; 1.7.12; 1.7.14;
Correct the disassembly of instructions that move immediate values into
64bit targets. Only the 0xb8-0xbf instructions (move immediate to register)
have a 64bit immediate constant.
Correct the disassembly of movq %eax to/from an immediate address - 64bit addr.
Add cmovxx instructions.
Now stands a chance of not getting all misaligned on normal C code.
Add a few others, but there are a lot missing, but probably mainly SIMD ones.

branches: 1.6.4; 1.6.6; 1.6.12;
TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.4.26;
merge ktrace-lwp.

branches: 1.3.2;
Const'ify structure declarations to prevent compiler warnings.

Add disasm support for amd64 (x86-64).
From Cliff Neighbors.

branches: 1.1.2; 1.1.8; 1.1.10; 1.1.12;
Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.

when cpus are paused (spinning) in DDB, call x86_pause() in the loop,
hopefully this can reduce power usage while doing nothing.

ok chs@

XXX: pullup-*
XXX: apply to i386

branches: 1.41.2;
ddb/db_active.h: New home for extern db_active.

This can be included unconditionally, and db_active can then be
queried unconditionally; if DDB is not in the kernel, then db_active
is a constant zero. Reduces need for #include opt_ddb.h, #ifdef DDB.

x86: improve error handling related to idt_vec_alloc()

introduce DDB_END_CMD and replace more than 20 copies of the same
list of NULLs and 0. idea from rillig@.

all touched ports built, several booted.

branches: 1.38.2;
Introduce per-cpu IDTs

This is realized by following modifications:
- Add IDT pages and its allocation maps for each cpu in "struct cpu_info"
- Load per-cpu IDTs at cpu_init_idt(struct cpu_info*)
- Copy the IDT entries for cpu0 to other CPUs at attach
- These are, for example, exceptions, db, system calls, etc.

And, added a kernel option named PCPU_IDT to enable the feature.

Fix boot -c or -d by avoiding ipi handling before the vector is initialized.

branches: 1.36.4; 1.36.6;
Fix NLAPIC, NISA and NIOAPIC related conditional compile errors.

This will allow us to now compile an amd64 kernel without PCI.

No functional changes.

We reorganise definitions for XEN source support as follows:

XEN - common sources required for baseline XEN support.
XENPV - sources required for support of XEN in PV mode.
XENPVHVM - sources required for support for XEN in HVM mode.
XENPVH - sources required for support for XEN in PVH mode.

Make XEN use the same api as native, for idt vector allocation
and registration.

lidt() placed in xenfunc() on maxv@ suggestion.

There should be no functional change due to this commit.

Tested on amd64 native and XEN.

branches: 1.33.2;
Rename the DDB IPI IDT vectors for consistency. ok maxv@

branches: 1.32.2;
Remove double declaration; 'ddb_regs' is already declared as a macro
in db_machdep.h if MULTIPROCESSOR is on, and the macro has higher
priority.

Don't declare 'ddb_regs' locally in this case, because it is misleading.
Part of PR/52964.

Define ddb_regp only if MULTIPROCESSOR (NFC)

Fix NULL pointer dereference via ddb_regs

ddb_regs can be *ddb_regp (see db_machdep.h) so ddb_regp should be NULL-ed
after dereference to ddb_regs.

Also dbreg should be restored to ddb_regp because ddb_regp can be changed
by db_mach_cpu during db_trap.

Fix PR 52964
Helped by nonaka@

make it compile without MULTIPROCESSOR (xen?)

Reset ddb_regp to NULL. Reported by David Binderman in PR/52964.

Rename intrddb -> intrddbipi, like i386.

Reduce the diff between amd64 and i386.

branches: 1.25.2;
x86: Add preliminary x2APIC support.

x2APIC is used only when x2APIC is enabled in BIOS/UEFI.
LAPIC ID is not supported above 256.

branches: 1.24.12; 1.24.30;
make ddb minimally MP aware

branches: 1.23.2;
Clean up excessive #ifdef'age of NMI trap handling for amd64/i386/xen.
Handle NMI in all Xen kernels.

branches: 1.22.2;
Move counting of faults, traps, intrs, soft[intr]s, syscalls, and nswtch
from uvmexp to per-cpu cpu_data and move them to 64bits. Remove unneeded
includes of <uvm/uvm_extern.h> and/or <uvm/uvm.h>.

branches: 1.21.4;
Declare trap name array as extern const char *const [] to match how
it's defined in trap.c. (Should really be in a header, of course.)
Compile-tested.

branches: 1.20.2;
Match declaration of trap_type in trap.c.

branches: 1.19.2;
- Rename cpu_lookup_byindex() to cpu_lookup(). The hardware ID isn't of
interest to MI code. No functional change.
- Change /dev/cpu to operate on cpu index, not hardware ID. Now cpuctl
shouldn't print confused output.

branches: 1.18.4;
Another use of X86_MAXPROCS.

Kill use of X86_MAXPROCS.

- Make cpu_number() return MI index, otherwise the pmap cannot work on
systems with lapic IDs > X86_MAXPROCS.
- Kill cpu_info[] array and use MI cpu_lookup_byindex().

branches: 1.15.14; 1.15.16; 1.15.18; 1.15.20;
Use the new atomic ops.

Reinclude machine/cpuvar.h lost in bouyer-xenamd64 merge; make GENERIC.MP
kernel build again.

Pull up the bouyer-xenamd64 branch to HEAD. This brings in amd64 support
to NetBSD/Xen, both Dom0 and DomU.

Disable TLB shootdown IPIs while in the debugger. Crashdumps may try to
use them, and all but one CPU is paused. Reported and tested by martin@.

branches: 1.11.2;
Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree. Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches. The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

branches: 1.10.2;
Add verbose help for (the one and only) "mach" command.

branches: 1.9.2;
Add a new option DDB_VERBOSE_HELP that adds online help to ddb.
From Adam Hamsik.
Minor modifications by me, all bugs are probably mine.

branches: 1.8.8; 1.8.10;
Ansify, de-__P(), unregister.

branches: 1.7.4; 1.7.6; 1.7.10; 1.7.12;
miscellaneous fallout from the great boolean rototill.

TRUE -> true, FALSE -> false

branches: 1.5.26;
merge ktrace-lwp.

branches: 1.4.2;
adapt to ddb constification

Uppercase CPU, plural is CPUs.

branches: 1.2.2;
Make sure to include opt_foo.h if a defflag option FOO is used.

Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.

{amd64,i386}/db_machdep.c: Don't bury a function call in an if condition

db_frame_info has many arguments and requires contorted line wrapping
that obscures the condition. The same object code is generated
(modulo a local variable moved closer to its only use site).

{amd64,i386}/db_machdep.c: Use DB_SYM_NULL

amd64: Handle __syscall trap frames specially too.

amd64: Print effective syscall number for SYS_syscall stack traces.

This time with compile-testing, at least.

Revert "amd64: Print effective syscall number for SYS_syscall stack traces."

Built the wrong kernel (arm64 is not amd64), oops. Will redo.

amd64: Print effective syscall number for SYS_syscall stack traces.

ddb: fix function names of "noreturn" functions in stack traces.

when looking up function names for stack traces (where the addresses are the
return addresses of function calls), if the address is the first instruction
in the function, assume that the function being called is marked "noreturn"
and that the function containing the call is actually the function immediately
before the address that we looked up. to find the correct function name,
do the lookup again with (address - 1) and then add one to the offset within
the function that we find.

If the frame is not aligned, leave right away. This place probably needs
to be revisited, because %rbp could easily contain garbage.

Reported-by: syzbot+ecb40cf7f8acc102c29b@syzkaller.appspotmail.com

Don't even try to go past a syscall. Fixes severe panic recursions in
KUBSAN.

Rename "handle_" -> "Xhandle_", and add the function names (introduced by
SVS) in db_machdep.c.

Should fix the DDB part of PR/53060.

branches: 1.5.2;
Style, and reduce the diff between i386 and amd64. No functional change.

branches: 1.4.30;
Use db_read_value to read the trapframe fields in db_nextframe.

Fixes SIGSEGV on bt in crash(8) when the stack trace ends in syscall,
and probably other problems as well.

ok christos

branches: 1.3.2;
Neither i386 nor amd64 have had separate syscall_plain/fancy functions
for ages, so this code can't be correct in looking for them.
Change to compare against "syscall".
I don't know if this changes any behaviour anywhere.

fix stack backtraces through traps.

branches: 1.1.2; 1.1.4; 1.1.8; 1.1.10; 1.1.14; 1.1.16;
Merge db_trace for x86. From: Vladimir Kirillov proger at wilab dot org dot ua

Merge i386 and amd64 version of db_memrw.c.

Use this opportunity to skip calculating the VA of the page. Let the CPU
deal with the invalidation itself through invlpg + destination address to
avoid converting between canonical/non canonical forms.

invlpg on a non canonical address is a noop, so no chance to invalidate
the TLB and the CPU will not notice the access right change.

This results in write protection faults in supervisor mode when patching
kernel code through ddb(4) (originally mapped as read only).

Bug reported by David Laight on port-amd64@ (thanks!), patch and test by
me.

i386 is unaffected as PG_LGFRAME does not mask the sign bits. For the
sake of correctness, use VA_SIGN_NEG(...) anyway.

XXX this is the patch that will be pulled-up to -5 and -6.

Use the current destination address to compute PTE, not the address of
origin.

Harmless, except when db_write_text() passes a page boundary.

From Bug Hunting.

XXX has to be pulled up to -5 and -6.

branches: 1.9.2; 1.9.4;
Kill dependency to xpmap_update(), and use setbits/clearbits to update
kernel PTE rights.

branches: 1.8.8;
Move counting of faults, traps, intrs, soft[intr]s, syscalls, and nswtch
from uvmexp to per-cpu cpu_data and move them to 64bits. Remove unneeded
includes of <uvm/uvm_extern.h> and/or <uvm/uvm.h>.

branches: 1.7.2; 1.7.6;
Match mem.c's extern for __data_start.

branches: 1.6.8; 1.6.10;
Remove clause 3 and 4 from TNF licenses

branches: 1.5.14; 1.5.16; 1.5.18;
Pull up the bouyer-xenamd64 branch to HEAD. This brings in amd64 support
to NetBSD/Xen, both Dom0 and DomU.

branches: 1.4.2;
merge yamt-x86pmap branch.

- reduce differences between amd64 and i386. notably, share pmap.c
between them. it makes several i386 pmap improvements available to
amd64, including tlb shootdown reduction and bug fixes from Stephan Uphoff.
- implement deferred pmap switching for amd64.
- remove LARGEPAGES option. always use large pages if available.
also, make it work on amd64.

branches: 1.3.30; 1.3.48; 1.3.50; 1.3.52; 1.3.54;
merge ktrace-lwp.

branches: 1.2.2;
adapt to ddb constification

branches: 1.1.2;
Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.

Merge db_trace for x86. From: Vladimir Kirillov proger at wilab dot org dot ua

branches: 1.18.2;
Remove pfind() and pgfind(), fix locking in various broken uses of these.
Rename real routines to proc_find() and pgrp_find(), remove PFIND_* flags
and have consistent behaviour. Provide proc_find_raw() for special cases.
Fix memory leak in sysctl_proc_corename().

COMPAT_LINUX: rework ptrace() locking, minimise differences between
different versions per-arch.

Note: while this change adds some formal cosmetics for COMPAT_DARWIN and
COMPAT_IRIX - locking there is utterly broken (for ages).

Fixes PR/43176.

branches: 1.17.2; 1.17.4;
Use lwp_getpcb() on x86 MD code, clean from struct user usage.

Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

ansify function definitions

branches: 1.14.2; 1.14.8;
Compile NetBSD/amd64 kernels with "-Wextra". Patches contributed by
Juan RP in PR port-amd64/39266.

branches: 1.13.2;
Remove proc_representative_lwp(), use a simple LIST_FIRST() instead.
OK by <ad>.

branches: 1.12.6; 1.12.10; 1.12.12; 1.12.14;
Define 'struct intrframe' in terms of 'struct trapframe' since the two are
assumed to match by a lot of code (including that which saves the regs).
This only slightly reduces the number of places the trapframe register
layout is defined.

branches: 1.11.22; 1.11.30; 1.11.34;
TRUE -> true, FALSE -> false

Pick up some additional files that were missed before due to conflicts
with newlock2 merge:

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

- Add /a modifier to trace that allows a specific LWP to be selected.
- When doing trace/t, always note which pid/lid combination is being traced.

Change the process/lwp flags seen by userland via sysctl back to the
P_*/L_* naming convention, and rename the in-kernel flags to avoid
conflict. (P_ -> PK_, L_ -> LW_ ). Add back the (now unused) LSDEAD
constant.

Restores source compatibility with pre-newlock2 tools like ps or top.

Reviewed by Andrew Doran.

branches: 1.7.2;
Merge newlock2 to head.

branches: 1.6.2;
Use p_find(addr, PFIND_LOCKED) in case the proclist_lock is held.

branches: 1.5.4; 1.5.8;
merge ktrace-lwp.

branches: 1.4.2;
Function declarations did not match prototypes, fix.

Make tracing work better (through interrupts, etc). Essentially a synch
with i386.

Remove use of NULL as an integer value.

branches: 1.1.2;
Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.

remove an unused file.

merge ktrace-lwp.

branches: 1.8.16;
Overhaul MBR handling (part 1):

<sys/bootblock.h>:
* Added definitions for the Master Boot Record (MBR) used by
a variety of systems (primarily i386), including the format
of the BIOS Parameter Block (BPB).
This information was cribbed from a variety of sources
including <sys/disklabel_mbr.h> which this is a superset of.

As part of this, some data structure elements and #defines
were renamed to be more "namespace friendly" and consistent
with other bootblocks and MBR documentation.
Update all uses of the old names to the new names.

<sys/disklabel_mbr.h>:
* Deprecated in favor of <sys/bootblock.h> (the latter is more
"host tool" friendly).

amd64 & i386:
* Renamed /usr/mdec/bootxx_dosfs to /usr/mdec/bootxx_msdos, to
be consistent with the naming convention of the msdosfs tools.

* Removed /usr/mdec/bootxx_ufs, as it's equivalent to bootxx_ffsv1
and it's confusing to have two functionally equivalent bootblocks,
especially given that "ufs" has multiple meanings (it could be
a synonym for "ffs", or the group of ffs/lfs/ext2fs file systems).

* Rework pbr.S (the first sector of bootxx_*):
+ Ensure that BPB (bytes 11..89) and the partition table
(bytes 446..509) do not contain code.
+ Add support for booting from FAT partitions if BOOT_FROM_FAT
is defined. (Only set for bootxx_msdos).
+ Remove "dummy" partition 3; if people want to installboot(8)
these to the start of the disk they can use fdisk(8) to
create a real MBR partition table...
+ Compile with TERSE_ERROR so it fits because of the above.
Whilst this is less user friendly, I feel it's important
to have a valid partition table and BPB in the MBR/PBR.

* Renamed /usr/mdec/biosboot to /usr/mdec/boot, to be consistent
with other platforms.

* Enable SUPPORT_DOSFS in /usr/mdec/boot (stage2), so that
we can boot off FAT partitions.

* Crank version of /usr/mdec/boot to 3.1, and fix some of the other
entries in the version file.

installboot(8) (i386):
* Read the existing MBR of the filesystem and retain the BIOS
Parameter Block (BPB) in bytes 11..89 and the MBR partition
table in bytes 446..509. (Previously installboot(8) would
trash those two sections of the MBR.)

mbrlabel(8):
* Use sys/lib/libkern/xlat_mbr_fstype.c instead of homegrown code
to map the MBR partition type to the NetBSD disklabel type.


Test built "make release" for i386, and new bootblocks verified to work
(even off FAT!).

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

__KERNEL_RCSID()

branches: 1.5.2;
Change bounds_check_with_label() to take a pointer to the disk structure,
rather than the label itself. This paves the way for some future changes.

Revert previous until I look at some issues regarding default labels.

Don't whine if no disklabel was found, it's not a fatal error. Follows
Jason's commit to the ARM disksubr.c

Change return type of readdisklabel() to const char *
I hope I've found all the correct places!

Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.

Move sys/arch/amd64/amd64/fpu.c and sys/arch/amd64/include/fpu.h
into sys/arch/x86 in preparation for using the same code for i386.

Steal the table from i386/isa/npx.c than selects the highest priority
floating point error when raising a signal.
Multiple errors bits can easily be set if different sub-fields of an
xmm simd operation generate different errors.

Simplify fputrap() considerably.
There is no need to save the fpu state here, and definitely no need
to initialise the fpu.
The code is running with interrupts disabled having trapped on either
an x87 instruction (the one after the one that generated the error)
or on an SSE (etc) instruction that caused the error.
So all it needs to do it obtain the 'error' bits from the relevant
status register, clear the bits, and then raise any signal.
The signal code will save the fp state if the signal itself isn't masked.
It also passes the FP state to the signal handler - which can modify it.
(I suspect that wasn't thecase when this code was written.)
Seems to work for both 64bit and 32bit 'divide by zero' errors.
For the xmm trap, the xmm registers are updated for the result of the
instruction, but the trap returns to re-execute the instruction!
This makes it difficult for the signal handler to do anything sensible.

I've also changed the code to only use unmasked error bits when deciding
the signal code.

Convert the amd64 build to use x86/cpu_extended_state.h so that the fpu
definitions match those of i386.
Mostly just structure and field renames, in addition:
1) process_xmm_to_s87() and process_s87_to_xmm() moved into
x86/convert_xmm_s87.c so they can be used by amd64's netbsd32 code.
2) The linux signal code simplified to use a structure copy for ths fxsave
data - it matches the hardware definition and won't change.