Explicitly set -z noseparate-code because the default is about to change
and the boot blocks might not be able to load binaries with more than two
PT_LOAD sections.

Fix GCC flags for Spectre mitigation to really only apply to GCC.

Merge the bouyer-xenpvh branch, bringing in Xen PV drivers support under HVM
guests in GENERIC.
Xen support can be disabled at runtime with
boot -c
disable hypervisor

Make KASAN compatible with LLVM. Same as GCC, except that LLVM aggressively
inlines the shadow checks, and this causes problems at boot time; so we
pass -asan-instrumentation-with-call-threshold=0 to force callbacks instead
of inlines.

branches: 1.82.2;
KCOV doesn't depend on specificdata and cpu_intr_p() anymore, so drop
references.

Add KASAN instrumentation on on-stack VLAs.

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
- "shad", to track uninitialized memory with a bit granularity (1:1).
Each bit set to 1 in the shad corresponds to one uninitialized bit of
real kernel memory.
- "orig", to track the origin of the memory with a 4-byte granularity
(1:1). Each uint32_t cell in the orig indicates the origin of the
associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
- a code designating the type of memory (Stack, Pool, etc), and
- a compressed pointer, which points either (1) to a string containing
the name of the variable associated with the cell, or (2) to an area
in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Merge amd64func.S into cpufunc.S, and clean up.

Allow to use KCOV + kASan together

Stop instrumenting KCOV files with kASan and the other way around.
This fixes booting of the setup of using them together.

In theory the checks could be more fine grained, however there is no good
reason (except extra DIAGNISTIC) for running a kernel sanitizer without
kernel coverage.

Patch by <R3x>

Add support for use-after-scope detection in KASAN. It is available since
GCC7, and we have GCC7 by default now.

Slightly reorder the code, and remove a duplicated KASSERT too.

Tested on amd64-KASAN. Not yet enabled on aarch64-KASAN, but it should
work as-is.

Move the MI parts of KASAN into kern/subr_asan.c. This file includes
machine/asan.h, which contains the MD functions. We use an include rather
than a plain C file, because we want GCC to optimize/inline some functions
into one single block.

The amd64 MD parts of KASAN are moved accordingly.

The naming convention we use is:

kasan_*
a generic kasan object, declared in subr_asan.c
kasan_md_*
an MD kasan object, declared in machine/asan.h, and used
in subr_asan.c
__md_*
an MD object, declared in machine/asan.h, and not used
outside

Overall this makes it easier to add KASAN support on more architectures.

Discussed with several people.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

Add support for monitoring the stack with kASan. This allows us to detect
illegal memory accesses occuring there.

The compiler inlines a piece of code in each function that adds redzones
around the local variables and poisons them. The illegal accesses are then
detected using the usual kASan machinery.

The stack size is doubled, from 4 pages to 8 pages.

Several boot functions are marked with the __noasan flag, to prevent the
compiler from adding redzones in them (because we haven't yet initialized
kASan). The kasan_early_init function is called early at boot time to
quickly create the shadow for the current stack; after this is done, we
don't need __noasan anymore in the boot path.

We pass -fasan-shadow-offset=0xDFFF900000000000, because the compiler
wants to do
shad = shadow-offset + (addr >> 3)
and we do, in kasan_addr_to_shad
shad = KASAN_SHADOW_START + ((addr - CANONICAL_BASE) >> 3)
hence
shad = KASAN_SHADOW_START + (addr >> 3) - (CANONICAL_BASE >> 3)
= [KASAN_SHADOW_START - (CANONICAL_BASE >> 3)] + (addr >> 3)
implies
shadow-offset = KASAN_SHADOW_START - (CANONICAL_BASE >> 3)
= 0xFFFF800000000000 - (0xFFFF800000000000 >> 3)
= 0xDFFF900000000000

In UVM, we add a kasan_free (that is not preceded by a kasan_alloc). We
don't add poisoned redzones ourselves, but all the functions we execute
do, so we need to manually clear the poison before freeing the stack.

With the help of Kamil for the makefile stuff.

Add support for kASan on amd64. Written by me, with some parts inspired
from Siddharth Muralee's initial work. This feature can detect several
kinds of memory bugs, and it's an excellent feature.

It can be enabled by uncommenting these three lines in GENERIC:

#makeoptions KASAN=1 # Kernel Address Sanitizer
#options KASAN
#no options SVS

The kernel is compiled without SVS, without DMAP and without PCPU area.
A shadow area is created at boot time, and it can cover the upper 128TB
of the address space. This area is populated gradually as we allocate
memory. With this design the memory consumption is kept at its lowest
level.

The compiler calls the __asan_* functions each time a memory access is
done. We verify whether this access is legal by looking at the shadow
area.

We declare our own special memcpy/memset/etc functions, because the
compiler's builtins don't add the __asan_* instrumentation.

Initially all the mappings are marked as valid. During dynamic
allocations, we add a redzone, which we mark as invalid. Any access on
it will trigger a kASan error message. Additionally, the compiler adds
a redzone on global variables, and we mark these redzones as invalid too.
The illegal-access detection works with a 1-byte granularity.

For now, we cover three areas:

- global variables
- kmem_alloc-ated areas
- malloc-ated areas

More will come, but that's a good start.

branches: 1.71.2;
Disable sanitizers for the kernel.

Simplify previous, pointed out by mrg.

Fix previous: HAVE_GCC needs to be checked in additon to
SPECTRE_V2_GCC_MITIGATION, but SPECTRE_V2_GCC_MITIGATION being empty is
not a good enough check as it can't be overwritte on the make
command line. Now I can do: "make SPECTRE_V2_GCC_MITIGATION=0" (e.g. when
my gcc is too old for having the indirect-branch options).

Fix copy & pasto

add an SPECTRE_V2_GCC_MITIGATION option to x86 kernels, that turns
on the GCC spectre v2 mitigation options.

XXX: pullup-8.
XXX: turn on in all kernels.

branches: 1.66.2;
Revert previous, we'll need something better (and compatible with Clang).

Add -fno-shrink-wrap, to force GCC to push the frames at the very beginning
of the functions. Otherwise DDB is unable to display a correct stack trace
if a fault occurred in a function before the frame was pushed.

Discussed on tech-kern@, flag suggested by Krister Walfridsson. Should fix
PR/52560.

Support large pages on KASLR kernels, in a way that does not reduce
randomness, but on the contrary that increases it.

The size of the kernel sub-blocks is changed to be 1MB. This produces a
kernel with sections that are always < 2MB in size, that can fit a large
page.

Each section is put in a 2MB physical chunk. In this chunk, there is a
padding of approximately 1MB. The prekern uses a random offset aligned to
sh_addralign, to shift the section in physical memory.

For example, physical memory layout created by the bootloader for .text.4
and .rodata.0:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
|+---------------+ |+---------------+ |
|| .text.4 | PAD || .rodata.0 | PAD |
|+---------------+ |+---------------+ |
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
PA PA+2MB PA+4MB

Then, physical memory layout, after having been shifted by the prekern:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
| P +---------------+ | +---------------+ |
| A | .text.4 | PAD | PAD | .rodata.0 | PAD |
| D +---------------+ | +---------------+ |
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
PA PA+2MB PA+4MB

The kernel maps these 2MB physical chunks with 2MB large pages. Therefore,
randomness is enforced at both the virtual and physical levels, and the
resulting entropy is higher than that of our current implementaion until
now.

The padding around the section is filled by the prekern. Not to consume
too much memory, the sections that are smaller than PAGE_SIZE are mapped
with normal pages - because there is no point in optimizing them. In these
normal pages, the same shift is applied.

This change has two additional advantages: (a) the cache attacks based on
the TLB are mostly mitigated, because even if you are able to determine
that a given page-aligned range is mapped as executable you don't know
where exactly within that range the section actually begins, and (b) given
that we are slightly randomizing the physical layout we are making some
rare physical attacks more difficult to conduct.

NOTE: after this change you need to update GENERIC_KASLR / prekern /
bootloader.

Split each kernel section into sub-blocks of approximately 2MB. The newly
created sections are named .origname.i, for example:

.text -> { .text .text.0 .text.1 .text.2 .text.3 .text.4 }

Each section is randomized independently by the prekern - and in a random
order obviously. As a result we can get intertwined mappings, of the type:

+-------+-----------+------+---------+-----------+-------+-------+------+-
| text1 | NOTMAPPED | bss0 | rodata1 | NOTMAPPED | data2 | text3 | bss1 |
+-------+-----------+------+---------+-----------+-------+-------+------+-

---------+-
rodata0 | ...
---------+-

The CTF section is dropped completely, because (a) when split it becomes
enormous for some reason (that I don't quite understand, verily), and (b)
the kernel expects only one CTF and can't handle several of them.

Remove max-page-size on KASLR, it doesn't play any role.

Use another ld script for kaslr kernels, in which there are no alignment
directives. They don't matter since the bootloader overwrites them.

But, normally we still need to make sure .data.read_mostly is aligned.
Unfortunately I couldn't find any way to force sh_addralign to be 64, so
I'm leaving the alignment there as a useless reminder.

Add linux_sigcode.o, otherwise it doesn't get rebuilt.

Add GENERIC_KASLR, only toolchain parts for now.

Remove several dead entries from the x86 makefiles. Looks like people (me
included) regularly forget to take care of this.

branches: 1.57.6;
Put 2MB alignments between the kernel segments. This way the kernel image
is entirely mapped with large pages, which uniformizes performance and
reduces fluctuation. Sent on port-amd64.

Increase KERNTEXTOFF from 1MB to 2MB on amd64. [1MB; 2MB[ is now handled
by UVM, so there is no physical loss.

On amd64 we always remap the kernel text with 2MB pages, and because of the
1MB start address we were forced to map [0MB; 2MB[ inside the first large
page. The problem is, the lower half is used by UVM to allocate physical
pages, and it is possible that some of these could be used by userland. We
could end up with userland-controllable data mapped into the kernel text on
a privileged page, which is far from being a good idea from a security pov.

I am not fixing i386 yet, because the large page size depends on PAE, and
we probably don't want to have a text located at 4MB on low-memory systems.

(note: I didn't introduce this issue, it was already there when I came in)

branches: 1.55.2; 1.55.4;
Moved hack to ufs_lookup.c

XXX: build ufs_lookup.c with -fno-tree-vrp when using GCC 5, for now.

Define MD start code at the top of files.${MACHINE} relying on config(1)'s
deterministic file output order. No need to define ${MD_OBJS} and its useless,
explicit compile rules.

Define ${MD_OBJS} as relative path. This change makes amd64 kernel buildable
for `-S' mode.

Define ${LINKSCRIPT} in one place.

${KERN_LDSCRIPT} -> ${KERNLDSCRIPT}

Include splash with genfb.
splash.o is only included with options SPLASHSCREEN.
splash_image.o target moved to Makefile.kern.inc.

This commit was approved by matt@

Remove unintended whitespace.

Added a new option for embedding a splash screen into kernel.
Add: options SPLASHSCREEN
makeoptions SPLASHSCREEN_IMAGE="path/to/image"
to your config file. So far it will work on amd64 and RPI/RPI2.

This commit was with ideas, help, and OK from jmcneill@.

branches: 1.46.2;
Use LINKSCRIPT.

Add -fno-omit-frame-pointer to DEFCOPTS.

branches: 1.44.2;
Apparantly clang doesn't like -mno-80387 or -mno-fp-ret-in-387
-msoft-float is equivalent to the former, the latter is probably unlikely
to stop anything not picked up elsewhere.

Change the compiler options to explicitly specify:
-mno-mmx -mno-sse -mno-avx -mno-80387 -mno-fp-ret-in-387
Since no-sse implies no-sse2 that should ensure that the compiler really
doesn't emit any instructions that might trap trying to use the FPU.
On amd64 at least some of those are needed to stop the compiler
saving the registers to stack on every varargs function.
It might be that -mno-sse did that before.

Reduce amount of -no-integrated-as on x86 as .code16 is now supported by
LLVM.

branches: 1.41.2;
do like the i386 for modular.

branches: 1.40.6;
Move Clang-warning flags for ah_regdomain.c into sys/conf, they are
platform independent. Use the selective -Wno-* flags instead of
-Wno-error.

branches: 1.39.2; 1.39.6;
include wakecode after kernel rules so that the .depend rules are defined

Remove a number of now unnecessary -no-integrated-as cases.

Disable LLVM MC for spl.S for now. The different spllower sizes break
patchfunc.

Explicitly disable use of SSE. LLVM generates SSE by default on AMD64
and we certainly don't want that in the kernel.

branches: 1.35.2;
Disable -Werror for ah_regdomain.c if building with clang as workaround
for http://llvm.org/bugs/show_bug.cgi?id=10030.

Drop redundant -I.

Really apply the -no-integrated-as only for clang.

Reuse -x assembler-with-cpp from sys.mk.

LLVM's assembler parser doesn't support .code32 yet, so disable it as
needed.

branches: 1.30.2;
Allow use of traditional CPP to be set on a per platform base in sys.mk.
Honour this for dependency processing in bsd.dep.mk. Switch i386 and
amd64 assembly to use ISO C90 preprocessor concat and drop the
-traditional-cpp on this platform.

branches: 1.29.4;
Enable SSP (Stack Smash Protection) in x86 kernels by default (except
in i386 *TINY kernels). The NetBSD/i386 "ALL" kernel is unconditionally
compiled with SSP enabled.

Change approved by the core team.

It's -z max-page-size, not -z maxpagesize.

MAXPAGESIZE got bumped to 0x200000. Use -z maxpagesize=0x100000 until
someone(tm) decides that kernels should have that alignment, etc.

branches: 1.26.2;
Clean-up makefile stub used to include in the build the binary HAL object

branches: 1.25.2; 1.25.8;
New assym.h dependency.

branches: 1.24.4; 1.24.10; 1.24.12;
Implement an optimized, preemption-safe asm version of tsc_get_timecount().
The C version needs work to be preemption safe. Cuts the clock cycles
for microtime() from 950 down to 300 on a Pentium D.

branches: 1.23.2; 1.23.6;
Refactor in_cksum/in4_cksum/in6_cksum implementations:
- All three functions are included in the kernel by default.
They call a backend function cpu_in_cksum after possibly
computing the checksum of the pseudo header.
- cpu_in_cksum is the core to implement the one-complement sum.
The default implementation is moderate fast on most platforms
and provides a 32bit accumulator with 16bit addends for L32 platforms
and a 64bit accumulator with 32bit addends for L64 platforms.
It handles edge cases like very large mbuf chains (could happen with
native IPv6 in the future) and provides a good base for new native
implementations.
- Modify i386 and amd64 assembly to use the new interface.

This disables the MD implementations on !x86 until the conversion is
done. For Alpha, the portable version is faster.

in_cksum.S needs assym.h.

Remove unnecessary references to config_time.h.

Missing assym.h dependency.

lock_stubs.S needs assym.h.

branches: 1.18.2;
Merge jmcneill-pm branch.

branches: 1.17.2; 1.17.4; 1.17.6;
cpufunc.S depends on assym.h.

Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree. Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches. The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

branches: 1.15.2;
busfunc depends on assym.h

branches: 1.14.16; 1.14.24; 1.14.34; 1.14.36; 1.14.38;
convert to timecounters (from branch simonb-timecounters)

branches: 1.13.2;
Convert existing ath users to new Makefile, except for amd64, which needs
the rules due to needing to conditionally postprocess the HAL object file.

Macppc needs a a non-ELF HAL (EABI) object, so take care of that by default
in the atheros include file.

Reorganize ath layout as requested by sam@ and suggested by dyoung@ in
http://mail-index.netbsd.org/tech-net/2006/03/15/0000.html.

The new layout almost precisely matches FreeBSD, and should make
future imports much easier.

At the same time, import the current 0.9.16.16 HAL from FreeBSD. According
to sam@, this is the proper version we should be using.

branches: 1.11.2;
No need to create opt_ah.h here, config(8) takes care of it.

branches: 1.10.2; 1.10.4;
Before symlinking opt_ah.h, make sure the target doesn't already exist.
This fixes a build problem with UPDATE set.

branches: 1.9.4; 1.9.6;
It seems that make doesn't like testing empty() on undefined variables.
Also objdump DTRT when asked to update a file, so no need to do a mv ourselves.

If we are compiling with -fno-asynchronous-unwind-tables then also
remove the .eh_frame section from athhal-x86_64-elf.hal.o.

branches: 1.7.6;
Add config goo for ath. Tested with a PCI card, detected as:

ath0 at pci0 dev 10 function 0
ath0: interrupting at ioapic0 pin 17 (irq 5)
ath0: 11a rates: 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
ath0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
ath0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36M bps 48Mbps 54Mbps
ath0: mac 5.9 phy 4.3 radio 3.6

Cardbus untested, so it is commented out.

branches: 1.6.2;
s/GENASSYM/GENASSYM_CONF/ so we can use "GENASSYM" as the program name.

Add the %MAKEOPTIONSAPPEND token at the end of the file, after the
common Makefile.kern.inc has been included.

branches: 1.4.2; 1.4.6;
another dependency on assym.h

clock.o depends on config_time.h

Default to -O2.

Rename the x86_64 port to amd64, as this is the actual name used for
the processor family now. x86_64 is kept as the MACHINE_ARCH value,
since it's already widely used (by e.g. the toolchain, etc), and
by other operating systems.