x86: avoid annoying GCC 12 bounds check in curcpu() and curlwp().

these functions read %gs and return an pointer at an offset from this
value (the current cpu, or lwp pointers), and GCC is complaining that
they're accessing a array cpu_info[0] (ie, zero length, no storage.)

several attempts to workaround it have failed, and because of the
asm volatile nature of this, it seems very unlikely a compiler would
take this and do something wrong with it.

amd64: Make curlwp and curcpu() flushable.

The only effect of the `volatile' qualifier on an asm block with
outputs is to force the instructions to appear in the generated code,
even if the outputs end up being unused. Since these instructions
have no (architectural) side effects -- provided %gs is set
correctly, which must be the case here -- there's no need for the
volatile qualifier, so nix it.

In order to prevent _mcount() from being recursively called when built with COPTS=-O0,
sprinkle `__always_inline' to make _mcount() be generated as a single function.

sys/arch/amd64: remove trailing whitespace

Add a redzone between the pcb and the stack. Sent to port-amd64@.

Use the inlines; it is actually fine, since the compiler drops the inlines
if the caller is kmsan-instrumented, forcing a white-listing of the memory
access.

mi_userret(): take care of calling preempt(), set spc_curpriority directly,
and remove MD code that does the same.

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
- "shad", to track uninitialized memory with a bit granularity (1:1).
Each bit set to 1 in the shad corresponds to one uninitialized bit of
real kernel memory.
- "orig", to track the origin of the memory with a 4-byte granularity
(1:1). Each uint32_t cell in the orig indicates the origin of the
associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
- a code designating the type of memory (Stack, Pool, etc), and
- a compressed pointer, which points either (1) to a string containing
the name of the variable associated with the cell, or (2) to an area
in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

We reorganise definitions for XEN source support as follows:

XEN - common sources required for baseline XEN support.
XENPV - sources required for support of XEN in PV mode.
XENPVHVM - sources required for support for XEN in HVM mode.
XENPVH - sources required for support for XEN in PVH mode.

On Xen, copy just the bits we need from the trapframe for hardclock(9)
and statclock(9).

Current, the macros that use the trapframe are:
CLKF_USERMODE()
CLKF_PC()
CLKF_INTR()

Of these, CLKF_INTR() already ignores the frame and uses the ci_idepth
variable to do its job.

Convert the two remaining ones to do this, but only for XEN.

branches: 1.62.2;
Remove the prototypes for cpu_uarea_*, I removed these functions two
minutes ago.

branches: 1.61.2;
Remove the second argument from USERMODE and KERNELMODE, it is unused
now that we don't have vm86 anymore.

branches: 1.60.6; 1.60.40;
allocate uareas contiguously and access them via the direct map.

branches: 1.59.14; 1.59.18;
_LKM -> _MODULE

branches: 1.58.2;
put the contents of these header files around #ifdef __x86_64__, and
#include the <i386/foo.h> in the #else clause, making these files
largely bit-size independant.

branches: 1.57.4;
Mark x86_curlwp() with __attribute__ ((const)), so gcc can CSE it and know
that it does not clobber global data.

Wrap stuff in #ifdef _KERNEL

Share cpu.h between the x86 ports.

Simplify x86 identcpu code, and share between i386/amd64.

Improve x86 tsc handling:

- Ditch the cross-CPU calibration stuff. It didn't work properly, and it's
near impossible to synchronize the CPUs in a running system, because bus
traffic will interfere with any calibration attempt, messing up the
timings.

- Only enable the TSC on CPUs where we are sure it does not drift. If we are
On a known good CPU, give the TSC high timecounter quality, making it the
default.

- When booting CPUs, detect TSC skew and account for it. Most Intel MP
systems have synchronized counters, but that need not be true if the
system has a complicated bus structure. As far as I know, AMD systems
do not have synchronized TSCs and so we need to handle skew.

- While an AP is waiting to be set running, try and make the TSC drift by
entering a reduced power state. If we detect drift, ensure that the TSC
does not get a high timecounter quality. This should not happen and is
only for safety.

- Make cpu_counter() stuff LKM safe.

Make cpu_idle a macro calling a function pointer on x86.
Select the Xen idle routine for Xen, mwait if supported by the CPU and
it is not AMD and halt otherwise. As reported by Christoph Egger,
AMD Barcelona keeps the CPU in C0 state with MWAIT, contrary to HLT,
which uses C1 and therefore much less power.

branches: 1.51.2;
Avoid unneeded AST faults.

Add support for kernel preeemption to the i386 and amd64 ports. Notes:

- I have seen one isolated panic in the x86 pmap, but otherwise i386
seems stable with preemption enabled.

- amd64 is missing the FPU handling changes and it's not yet safe to
enable it there.

- The usual level for kern.sched.kpreempt_pri will be 128 once enabled
by default. For testing, setting it to 0 helps to shake out bugs.

branches: 1.49.2;
- Give ci_want_resched a single cache line, and align. This is for monitor/
mwait. At least one errata sheet from Intel notes that a single line
should be used.
- Align cc_microtime.

Ensure that offsetof() is in scope by including <sys/systm.h>.
Fixes build problem found while building swapnetbsd.o for XEN3_DOM0.

Access Xen's vcpu info structure per-CPU.
Tested on i386 and amd64 (both dom0 and domU) by me.
Xen2 tested (both dom0 and domU) by bouyer.
OK bouyer

branches: 1.46.2;
use POSIX integer types

Remove CTL_MACHDEP_NAMES, it's not used anywhere.

Ok by martin@.

branches: 1.44.2; 1.44.6;
GC i8254_microtime.

remove no longer necessary cpu_maxproc.

- make amd64 use per-cpu tss.
- fix iopl syscall for amd64+xen.

g/c unused members

g/c ci_idle_pcb_paddr

try to detect processor resource sharing topologies. ie. package/core/smt IDs.

Convert many of the uses of __attribute__ to equivalent
__packed, __unused and __dead macros from cdefs.h

Define 'struct intrframe' in terms of 'struct trapframe' since the two are
assumed to match by a lot of code (including that which saves the regs).
This only slightly reduces the number of places the trapframe register
layout is defined.

Add new IPI for saving CPU state explicitly, share high-level part of
ACPI wakeup code and teach it how to start the APs again. As a side
effect the CPU_START interface allows choosing between different
bootstrap codes more easily now.

branches: 1.35.2;
Merge jmcneill-pm branch.

branches: 1.34.2;
Add a CPU local timer based on the LAPIC. This is consistently faster
than TSC, but doesn't suffer from SpeedStep as TSC does.

The default quality is higher than HPET for UP, but -100 for
MULTIPROCESSOR as it needs CPU local state which doesn't exist yet.

branches: 1.33.2;
Pull up the bouyer-xenamd64 branch to HEAD. This brings in amd64 support
to NetBSD/Xen, both Dom0 and DomU.

- cpu_vendor was both an int and char[] on amd64 - fix it.
- Run the errata check/patch on all CPUs, not just the boot processor.

branches: 1.31.2;
Mark cpu_info::ci_tlbstate volatile to ensure that the compiler doesn't
reorder accesses to it. It's updated from the TLB IPI handlers and we don't
block those, so the order in which things are read/updated is important.

Match delay/DELAY on x86 with delay(9). It takes an unsigned int as
argument. Use this and replace the inline assembly (mul + div using the
64bit intermediate result) with normal 32bit multiplication and
division. The compiler can turn the division into a multiplication and
shift, making it even cheaper then the original assembly. For extreme
long delays, just use 64bit arithmetic.

merge yamt-x86pmap branch.

- reduce differences between amd64 and i386. notably, share pmap.c
between them. it makes several i386 pmap improvements available to
amd64, including tlb shootdown reduction and bug fixes from Stephan Uphoff.
- implement deferred pmap switching for amd64.
- remove LARGEPAGES option. always use large pages if available.
also, make it work on amd64.

Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree. Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches. The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

branches: 1.27.2;
x86 changes for pcc and LKMs.

- Replace most inline assembly with proper functions. As a side effect
this reduces the size of amd64 GENERIC by about 120kB, and i386 by a
smaller amount. Nearly all of the inlines did something slow, or something
that does not need to be fast.
- Make curcpu() and curlwp functions proper, unless __GNUC__ && _KERNEL.
In that case make them inlines. Makes curlwp LKM and preemption safe.
- Make bus_space and bus_dma more LKM friendly.
- Share a few more files between the ports.
- Other minor changes.

ci_astpending is no more.

branches: 1.25.2;
Merge most x86 changes from the vmlocking branch, except the threaded soft
interrupt stuff. This is mostly comprised of changes to the pmap modules to
work on multiprocessor systems without kernel_lock, and changes to speed up
tlb shootdowns.

branches: 1.24.4; 1.24.8; 1.24.10;
Revert fs/gs changes until I figure out issues with them.

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

Don't save/restore %fs and %gs in trapframe. The kernel won't touch them.
Instead, save/restore them on context switch. For 32bit processes, save/restore
the selector values only, for 64bit processes, save/restore the appropriate
MSRs. Iff the defaults have been changed.

branches: 1.21.4;
- Remove ci_msr_rvalue, it's not useful anymore as yamt@ pointed out.
- Remove completely debug from msr_ipifuncs, now it's known to work.

MSR read and write IPI handlers for x86. A MSR will be read or written
in all CPUs available in the system. This adds another member
to struct cpu_info, ci_msr_rvalue; it will contain the value of the MSR
in a previous operation.

Tested with clockmod in UP and SMP by me, tested with est in SMP
by Daniel Carosone and Michael Van Elst.

Ok'ed by Andrew Doran and Matthew R. Green.

struct cpu_info: add a ci_feature2_flags member.
identcpu: print extended cpuid features with ci_feature2_flags.

"Looks good" by christos and njoly.

Remove __P(), remove k8_powernow_init proto... it was moved to
x86/include/powernow.h long time ago.

branches: 1.17.2; 1.17.4;
Include sys/simplelock.h, not sys/lock.h.

branches: 1.16.2;
clean up how cpus and ioapics are attached at the mainbus:
Seperate "cpubus" and "ioapicbus" -- while they share a common "address
space" (the apic id), the kernel doesn't use this fact. There are different
data passed to cpus and apics, which caused some ugly polymorphism. This
also saves the special "submatch" functions needed to distingush cpus
and ioapics for autoconf. (And it makes that "apid" locators wired
in the kernel configuration are honored now; this allows one to dumb down
an mp box to singleprocessor by userconfig.)
Print "apid" locators in the buses "print" function "as everyone does",
so the per-port cpu drivers don't need to do it.
Being here, constify "struct cpu_functions" and g/c the unused MP_PICMODE
flag.

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.14.2;
Remove spllowersoftclock() and CLKF_BASEPRI(), and always dispatch callouts
via a soft interrupt. In the near future, softclock will be run from process
context.

Merge newlock2 to head.

branches: 1.12.4; 1.12.8; 1.12.10;
AMD PowerNow!/Cool`n'Quiet driver for NetBSD/amd64,
adapted from OpenBSD.

Tested on a few machines:

http://bigbird.dohd.org:3021/NetBSD/dmesg
http://www.bsd.org.il/netbsd/acpi/dmesg

Thanks to cube, elad and others for testing and fixes.

Enabled by default on GENERIC.

convert to timecounters (from branch simonb-timecounters)

branches: 1.10.6;
delay() is gone, so don't declare it. That way other parts of code that
use a variable named delay (say, netinet6/in6.c) won't shadow something
that doesn't exist anyway.

branches: 1.9.4; 1.9.6; 1.9.8;
bare asm -> __asm

merge ktrace-lwp.

Change all archs that did:

#define clockframe somethingelse

to:

struct clockframe {
struct somethingelse cf_se;
};

and change access macros accordingly.

That means that, at least for that very issue, things will not go
ka-boomy if you don't have the actual definition of struct clockframe
before including systm.h.

branches: 1.6.12;
don't expose cpu_info and friends to userland.

fix a typo in a comment.

move some per-cpu data definitions to MI place so that they can be modified
without touching all ports. discussed on tech-kern@.

Replace the traditional buffer memory management -- based on fixed per buffer
virtual memory reservation and a private pool of memory pages -- by a scheme
based on memory pools.

This allows better utilization of memory because buffers can now be allocated
with a granularity finer than the system's native page size (useful for
filesystems with e.g. 1k or 2k fragment sizes). It also avoids fragmentation
of virtual to physical memory mappings (due to the former fixed virtual
address reservation) resulting in better utilization of MMU resources on some
platforms. Finally, the scheme is more flexible by allowing run-time decisions
on the amount of memory to be used for buffers.

On the other hand, the effectiveness of the LRU queue for buffer recycling
may be somewhat reduced compared to the traditional method since, due to the
nature of the pool based memory allocation, the actual least recently used
buffer may release its memory to a pool different from the one needed by a
newly allocated buffer. However, this effect will kick in only if the
system is under memory pressure.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

branches: 1.1.2;
Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.