Create a new header lwp_private.h to contain _lwp_getprivate_fast,
_lwp_gettcb_fast, _lwp_settcb and remove them from mcontext.h, so that:
1. we don't need special hacks to hide them
2. we can include <lwp.h> where needed to get the necessary prototypes
without redefining them locally.

branches: 1.4.26;
Provide a trap.h (currently empty)

branches: 1.3.4;
Install aarch64/sljit_machdep.h.

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

s/ablility/ability/ and s/Utilility/Utility/ in comments.

Add a #define for the RES1 bits (just bit 31) in MPIDR_EL1.

Add MDCR_EL2 accessors and bit definitions.

Add sp_el2 accessors.

Remove the XXXNH I had against ESR_EC_LS64.

The FEAT_LS64 instructions are all A64.

Fix various typos in comments.

branches: 1.66.2;
ddress->address in comment.

Add a bunch of system registers and their bit / bit field definitions.
Taken from ryo's nvmm branch with updates from me.

s/Regiser/Register/ and s/regester/register/ in comments.

Improve tprof(4)

- Multiple events can now be handled simultaneously.
- Counters should be configured with TPROF_IOC_CONFIGURE_EVENT in advance,
instead of being configured at TPROF_IOC_START.
- The configured counters can be started and stopped repeatedly by
PROF_IOC_START/TPROF_IOC_STOP.
- The value of the performance counter can be obtained at any timing as a 64bit
value with TPROF_IOC_GETCOUNTS.
- Backend common parts are handled in tprof.c as much as possible, and functions
on the tprof_backend side have been reimplemented to be more primitive.
- The reset value of counter overflows for profiling can now be adjusted.
It is calculated by default from the CPU clock (speed of cycle counter) and
TPROF_HZ, but for some events the value may be too large to be sufficient for
profiling. The event counter can be specified as a ratio to the default or as
an absolute value when configuring the event counter.
- Due to overall changes, API and ABI have been changed. TPROF_VERSION and
TPROF_BACKEND_VERSION were updated.

PMCR.E should not be disabled from tprof.

PMCR.E controls not only performance event counters but also the cycle
counter operation, and the cycle counter may be used for cpu_counter.
Similarly, the 31st bit in PMINTENCLR and PMCNTENCLR controls the cycle
counter, not performance event counters, and should not be modified.

Only print the appropriate PAR fields for PAR.F={0,1}

Group the fields in the header.

fix ID_AA64ISAR0_EL1.ATOMIC field definition

fix build with COPTS=-O0

Typo in comment

Do not try to initialize PMU if ID_AA64DFR0_EL1 reports a non-standard
PMU implementation.

CNTV_CTL_EL0 is a 64-bit register

branches: 1.55.4;
fixed mask width of DBGWVR_MASK, and added definition of DBGBVR_MASK

branches: 1.54.2;
add some fields of ID_AA64ISAR1_EL1 definition (ARMv8.6)

fix typo

Add support for Privileged Access Never (ARMv8.1-PAN).

PAN provides the same functionality as SMAP on x86: it forbids kernel
access to userland pages when PSTATE.PAN=1, and allows such accesses when
PSTATE.PAN=0.

We clear SCTLR_SPAN, to guarantee that PAN=1 each time the kernel is
entered. We catch PAN faults and panic right away without further
processing. In copyin, copyout, etc, we temporarily authorize access to
userland pages.

PAN is a very useful exploit mitigation. Reviewed by ryo@, thanks. Tested
on Qemu. Enabled by default.

The system registers we modify can have an impact on memory accesses, and
we don't want the compiler to randomly re-order the instructions, so add
barriers. Same as WRMSR on x86.

- On some systems with a different cache line size (and DIC,IDC) per CPU, trap "mrs Xt,ctr_el0" instruction
to return the minimum cache line size of the system to userland.
- add CLIDR_EL1 and CTR_EL0 to struct aarch64_sysctl_cpu_id.

On most systems, cache line size is the same for all CPUs, so this mechanism won't be required.
Rather, this is primarily for errata support, which will be committed later.

Add some more id_aa64pfr0_el1 bits.

Add some new CTR_EL0 bits

add ARMv8.1-8.5 definitions of TCR_EL1

cache information can be detected correctly on newer CPUs

- add VPIPT cache type
- adapt to 64-bit CCSIDR (ARMv8.3-CCIDX)
- CCSIDR:[WT,WB,PA,WA] are deprecated
- show number of cache lines when attaching cpu

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

fix typo

- move aarch64 addressspace macros from pmap.h to cpufunc.h
- rename ptr_strip_pac() to aarch64_strip_pac()

"options ARMV83_PAC" is now supported for gcc as well.

- add "-msign-return-address=all" to CFLAGS for gcc when specified options ARMV83_PAC
- AARCH64REG_{READ,WRITE}_INLINE3 macro can now use the APIAKey registers in both gcc and llvm.
llvm requires asm(".arch armv8.3-a"), whereas gcc requires __attribute__((target("arch=armv8.3-a"))).
- use ".arch armv8.3-a" rather than ".arch armv8.3-a+pac" in *.S for llvm.

Fix ID_AA64ISAR0_EL1_RNDR field definition for RNDR support.

ARMv8.5 ARM, p. D13-3232

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.39.2;
Enable the cycle counter when a CPU hatches and store an estimate of the
frequency in ci_data.cpu_cc_freq.

fix missing paren

add more definitions for ARMv8.1-ARMv8.4

widen bit PAR_EL1.PAR_PA from [47:12] to [51:12] for ARMv8.2 (and later).

PAR_EL1:[51:48] is RES0 in ARMv8.1 and ARMv8.0.

BTI definitions.

More SCTLR.

Fetch ID_AA64MMFR2_EL1. Okayed by Nick the other day.

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

More definitions.

branches: 1.30.2;
s/Memroy/Memory/ in comment.

Enable early write acknowledge for device memory mappings.

report A72 errata #859971 workaround status during boot

Move the TCR and TTBR defines into armreg.h where they below. NFCI.

Add support for physical timers and sprinkle isb where needed.

branches: 1.25.2;
Provide icc_pmr_read

- add reg_{s1e0r,s1e0w,s1e1r,s1e1w}_write() macro.
- show the result of AT insn at ddb "machine pte" command.

add gtmr_cntv_cval_write

add support PT_STEP

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

When hardware subnormal support is available, disable flush-to-zero.

Similarly, when hardware NaN propagation is available, disable
default-NaN substitution.

This enables IEEE 754 semantics on any hardware that supports it by
default. Programs that want flush-to-zero or default-NaN substitution
can enable them explicitly.

ok ryo@

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

Provide and use cpu_mpidr_aff_read in psci_fdt_bootstrap

Whitespace

Restore ICC_SRE_EL2 registers lost in previous commit

Add GICv3 system registers

More whitespace

Some whitespace improvements. NFC.

- use #define to define constants instead of static const variables so that
gcc can compile the code.
- fix position of inline, and use __inline

Add more PMC registers

branches: 1.10.2;
Workaround A-008585 errata in GTMR.

Register reads and writes may provide unstable results if the counter
hardware is active at the same time. This results in non-monotonic
counters seen by both the gtmr interrupt and time counter.

The loops are currently applied unconditionally, restricting them to
appropiate FDT markers can be applied later.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

separate cputypes.h for CPU_ID_* from armreg.h,
and add some implementor IDs, CortexA55,73,75 IDs.

(preliminary changes for merging aarch64)

Sprinkle __volatile on asm instructions

Convert decimal to hex to make comparison to arm arm (slightly) easier.

Another harmless typo

Fix harmless typo

branches: 1.3.2;
Trailing whitespace

ARM spells the System Control Register SCTLR

branches: 1.1.4; 1.1.6;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Rename VM_KERNEL_IO_ADDRESS to VM_KERNEL_IO_BASE to match RISC-V

It's less letters, matches other similar variables and will help with
sharing code between the two architectures.

NFCI.

Remove some unnecessary tlb invalidate in pmap_growkernel and ASAN shadow
map. Ensure the shadow map mappings are visible to the TLB walkers.

branches: 1.17.2;
Adjust the kernel virtual address space so that KASAN will map the kernel
seperately from managed kernel virtual memory and not map the unused space
between the two.

branches: 1.16.2;
s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

Mark KASAN shadow pages as LX_BLKPAG_ATTR_NORMAL_WB. NFC as this is zero,
but someone might change it one day.

AA64 is not MIPS.

Change all KSEG references to directmap

branches: 1.13.2;
Use pmap_growkernel(VM_KERNEL_VM_BASE) rather than pmap_virtual_space to
work out what to map initially.

XXX could do better mapping the kernel and modules more accurately

Make __md_palloc pmap agnostic (think sys/uvm/pmap)

kasan: fix the copyright notices

Round of uvm.h cleanup.

The poorly named uvm.h is generally supposed to be for uvm-internal
users only.

- Narrow it to files that actually need it -- mostly files that need
to query whether curlwp is the pagedaemon, which should maybe be
exposed by an external header.

- Use uvm_extern.h where feasible and uvm_*.h for things not exposed
by it. We should split up uvm_extern.h but this will serve for now
to reduce the uvm.h dependencies.

- Use uvm_stat.h and #ifdef UVMHIST uvm.h for files that use
UVMHIST(ubchist), since ubchist is declared in uvm.h but the
reference evaporates if UVMHIST is not defined, so we reduce header
file dependencies.

- Make uvm_device.h and uvm_swap.h independently includable while
here.

ok chs@

Use large pages for the KASAN shadow, same as amd64, discussed with ryo@.

pmapboot_enter simplication
- bootpage_alloc in asm becomes pmapboot_pagealloc in C
- PMAPBOOT_ENTER_NOBLOCK is removed as it's not used
- PMAPBOOT_ENTER_NOOVERWRITE is removed as it's now always on
- physpage_allocator argument is removed as it's always
pmapboot_pagealloc
- Support for EARLYCONS without CONSADDR is removed so that the identity
map for CONSADDR is always known.

For the assembly files:
2 files changed, 40 insertions(+), 89 deletions(-)

LGTM ryo

Rename __MD_CANONICAL_BASE -> __MD_KERNMEM_BASE for clarity.

branches: 1.6.4;
- free empty page tables pages if reach a certain usage.
- need to lock at removing an old pg (_pmap_remove_pv) in _pmap_enter()

- free L1-L3 pages that has been emptied by pmap_remove().
- if no memories, pmap_enter will return correctly ENOMEM if PMAP_CANFAIL, or wait until available any memories if !PMAP_CANFAIL.

These changes improves the stability when we use a huge virtual memory spaces with mmap.

branches: 1.4.2;
add LX_BLKPAG_SH_IS pte attribute for MP

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Provide a kasan_md_unwind

OK maxv

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

Add support for Privileged Access Never (ARMv8.1-PAN).

PAN provides the same functionality as SMAP on x86: it forbids kernel
access to userland pages when PSTATE.PAN=1, and allows such accesses when
PSTATE.PAN=0.

We clear SCTLR_SPAN, to guarantee that PAN=1 each time the kernel is
entered. We catch PAN faults and panic right away without further
processing. In copyin, copyout, etc, we temporarily authorize access to
userland pages.

PAN is a very useful exploit mitigation. Reviewed by ryo@, thanks. Tested
on Qemu. Enabled by default.

"options ARMV83_PAC" is now supported for gcc as well.

- add "-msign-return-address=all" to CFLAGS for gcc when specified options ARMV83_PAC
- AARCH64REG_{READ,WRITE}_INLINE3 macro can now use the APIAKey registers in both gcc and llvm.
llvm requires asm(".arch armv8.3-a"), whereas gcc requires __attribute__((target("arch=armv8.3-a"))).
- use ".arch armv8.3-a" rather than ".arch armv8.3-a+pac" in *.S for llvm.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.6.4;
Replace the two copies of the ADDR macro with a centralised adrl macro.
The adrl name matches the one used by armasm.

branches: 1.5.2;
Add a speculation barrier after the 'eret'.

Some aarch64 cpus speculatively execute instructions after 'eret',
and this potentiates side-channel attack.

from
https://github.com/torvalds/linux/commit/679db70801da9fda91d26caf13bf5b5ccc74e8e8

Don't define register replacements when targetting 32bit ARM.

branches: 1.3.4;
centralize fp,lr definitions

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Really provide bus_funcs.h

branches: 1.2.2; 1.2.6;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Eliminate recursive dependency on <sys/bswap.h> in MD <byte_swap.h>

On certain NetBSD archs, it isn't possible to include <sys/bswap.h>
directly without including <sys/endian.h> first. This has been
broken since at least before netbsd-10, but recent changes elsewhere
have made this problem more visible for some reason.

Fix this by not including <sys/types.h>, eliminating recursive
inclusion - <sys/types.h> includes <machine/endian.h>. It would
be nice to remove that line from types.h, since it's no longer
necessary, but currently the kernel assumes that <sys/endian.h> is
included from <sys/types.h> in various places (and possibly userspace
software does too).

For now we can take the less dangerous option by avoiding
including <sys/types.h> from these bswap headers. They don't need the
full header, and applications that assume transclusion aren't portable
anyway, since architectures like sparc64 don't include <sys/types.h>
via <sys/bswap.h>.

PR pkg/59839 palemoon, firefox, etc., don't build on arm64 any more

branches: 1.4.52;
avoid conversion warnings

branches: 1.3.2; 1.3.4; 1.3.6;
Correct 32 and 64 bit byte swap inlines

branches: 1.2.4;
Use %x/%w as appropriate.

Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

arm64: Enable support for low power idle CPU states on ACPI platforms.

The ACPI CPU driver parses the _LPI package on each CPU and builds a
table of supported low power states. A custom cpu_idle() implementation
is registered that uses the time previously spent idle to select an
entry method for low power on the next idle entry.

A boot option, "nolpi", can be used to ignore _LPI and use the normal
WFI idle method.

This decreases the battery discharge rate on my Snapdragon X1E laptop from
~17W to ~10W when idle.

fix 32-bit arm builds

aarch64: Count RNDRRS failure events and add dtrace probe.

PR port-arm/58572: aarch64 RNDRRS failures should be evcounted and
dtraced

branches: 1.50.2;
port-arm/58194: Resurrect vmt(4) from bitrot

On this architecture vmt(4) used to search for a node "/hypervisor" in the
FDT and probed the VMware hypervisor call only when the node was
found. However, things appear to have changed and VMware no longer provides
the FDT node.

Since vmt(4) doesn't actually need to read anything from FDT, and the
hypervisor call logically resides in virtual CPUs themselves, it would be
better to attach it directly to cpu, just like how it's probed on x86.

aarch64: curcpu() audit.

Sprinkle KASSERT (or KDASSERT in hot paths) for kpreempt_disabled()
when we use curcpu() and it's not immediately obvious that the caller
has preemption disabled but closer scrutiny suggests the caller has.

Note unsafe curcpu()s for syscall event counting. Not sure this is
worth changing.

Possible bugs fixed:

- cpu_irq and cpu_fiq could be preempted while trying to run softints
on this CPU.

- data_abort_handler might incorrectly think it was invoked in
interrupt context when it was only preempted and migrated to
another CPU.

- pmap_fault_fixup might report the wrong CPU logs.

(However, we don't currently run with kpreemption on aarch64, so
these are not yet real bugs fixed except if you patch it to build
with __HAVE_PREEMPTION.)

branches: 1.48.2;
Provide MI PMAP support on AARCH64

Remove GIC_SPLFUNCS.

pic: Update ci_cpl in pic_set_priority callback.

Not all ICs need interrupts disabled to update the priority. DAIF accesses
are not cheap, so push the update of ci_cpl from pic_set_priority to the
IC's pic_set_priority callback, and let the IC driver determine whether
or not it needs interrupts disabled.

In order to prevent _mcount() from being recursively called when built with COPTS=-O0,
sprinkle `__always_inline' to make _mcount() be generated as a single function.

Fix a last minute rebase/merge botch so that the cpu_hatch commit actually
works.

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

Annotate some cpu_info members

Add a comment and adjust whitespace to match style in this file

Use sys/uvm/pmap/pmap_tlb.c on Aarch64 in the same way that some Arm, MIPS,
and some PPC kernels do. This removes the limitation of 256 processes on
CPUs with 8bit ASID field, e.g. Apple M1.

Additionally the following changes have been made

- removed a couple of unnecessary aarch64_tlbi_all calls
- removed any invalidation after freeing page tables due to
_pmap_sweep_pdp. This was never necessary afaict.
- all kernel mappings are marked global and userland mapping not-global.

Performance testing hasn't show a significant difference. The data here
is from building a kernel on an lx2k system with nvme.

before
1489.6u 400.4s 2:40.65 1176.5% 228+224k 0+32289io 57pf+0w
1482.6u 403.2s 2:38.49 1189.9% 228+222k 0+32274io 46pf+0w
1485.4u 402.2s 2:37.27 1200.2% 228+222k 0+32275io 12pf+0w

after
1493.9u 404.6s 2:37.50 1205.4% 227+221k 0+32265io 48pf+0w
1485.0u 408.0s 2:38.54 1194.0% 227+222k 0+32272io 36pf+0w
1484.3u 407.0s 2:35.88 1213.3% 228+224k 0+32268io 14pf+0w

>>> stats.ttest_ind([160.65,158.49,157.27], [157.5,158.54,155.88])
Ttest_indResult(statistic=1.1923622711296888, pvalue=0.2990182944606766)
>>>

gic_splx: performance optimizations

Avoid any kind of register access (DAIF, PMR, etc), barriers, and atomic
operations in the common case where no interrupt fires between spl being
raised and lowered.

This introduces a per-CPU return address (ci_splx_restart) used by the
vector handler to restart a sequence in splx that compares the new ipl
with the per-CPU hardware priority state stored in ci_hwpl.

Improved the performance of kernel profiling on MULTIPROCESSOR, and possible to get profiling data for each CPU.

In the current implementation, locks are acquired at the entrance of the mcount
internal function, so the higher the number of cores, the more lock conflict
occurs, making profiling performance in a MULTIPROCESSOR environment unusable
and slow. Profiling buffers has been changed to be reserved for each CPU,
improving profiling performance in MP by several to several dozen times.

- Eliminated cpu_simple_lock in mcount internal function, using per-CPU buffers.
- Add ci_gmon member to struct cpu_info of each MP arch.
- Add kern.profiling.percpu node in sysctl tree.
- Add new -c <cpuid> option to kgmon(8) to specify the cpuid, like openbsd.
For compatibility, if the -c option is not specified, the entire system can be
operated as before, and the -p option will get the total profiling data for
all CPUs.

Re-apply

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

There is(/was) no need to use atomic operations on the percpu / cpu_info
members, so don't.

Finally removng the use of percpu should help avoid problems with "late"
attaching cpus.

Deal with the pmap limitation of maxproc in a more complete way and
recognise CPUs with only 8bit ASIDs.

Sort includes. NFCI.

branches: 1.34.2; 1.34.4;
Revert recent pic optimizations until I have more time to work on this.

branches: 1.33.2;
Add cpu_dosoftints_ci(). Like cpu_dosoftints(), but takes a cpu_info ptr
so we can avoid the extra tpidr_el1 access if cpu_info is already known.

Keep current hardware priority value in struct cpu_info and use it instead
of reading icc_pmr_el1 in gicv3_set_priority.

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

ACPI Processor UID is 32-bits (ci_acpiid).

Add a per-CPU event counter that counts every time an interrupt handler is
preempted by a higher priority interrupt.

branches: 1.28.2;
fix build error with LLVM

PID_MAX is just an initial value (soft maximum). Don't use it for CTASSERT.
defined __HAVE_CPU_MAXPROC to use function cpu_maxproc().

pointed out by mrg@, thanks.

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

- On some systems with a different cache line size (and DIC,IDC) per CPU, trap "mrs Xt,ctr_el0" instruction
to return the minimum cache line size of the system to userland.
- add CLIDR_EL1 and CTR_EL0 to struct aarch64_sysctl_cpu_id.

On most systems, cache line size is the same for all CPUs, so this mechanism won't be required.
Rather, this is primarily for errata support, which will be committed later.

Switch the Icache sync operation to the necessary and sufficient one according to the CTR_EL0.DIC and CTR_EL0.IDC flags.

If CTR_EL0.DIC=1, Icache invalidation is not required.
If CTR_EL0.IDC=1, Dcache clean before Icache invalidation is not required.
CLIDR_EL1.LoC is 0, or CLIDR_EL1.LoUIS and CLIDR_EL1.LoUU are 0, Dcache clean is not required as well.

SEE ALSO ARMARM, "CTR_EL0 Cache Type Register", and "CLIDR_EL1 Cache Level ID Register"

Draft fpu_kern_enter/leave on aarch64.

protect curcpu/curlwp from _KMEMUSER

Various updates and improvements to cpu start up on arm/aarch64

- start sharing more code around the AP startup messaging.
- call arm_cpu_topology_set early so that ci_core_id is available for
drivers, e.g. bcm2835_intr.c
- both arm and aarch64 now have
- a static cpu_info_store array
- the same arm_cpu_{hatched,mbox}

Define the MULTIPROCESSOR cpu_number() for modules too.

Modules should work whether the main kernel is multiprocessor or not.
In particular, dtrace should not think cpu_number() is 0 while
cpu_index(curcpu()) and curcpu()->ci_index are nonzero, leading to
rather spectacularly bogus results...

port the arm64 cpu topology setup for big.little to arm.

rename arm64 cpu_do_topology() to arm_cpu_do_topology() and
call it from both arm cpu_attach().

replace both aarch64_set_topology() inline code in arm
cpu_attach() with new arm_cpu_do_topology(), which is called
by the arm64 locore as well (possibly not needed, which would
allow it to become static.)

not yet tested on a real big.little armv7 system. tested
on rockpro64 and pinebook pro.

provide some semblance of valid cpu topology for big.little systems.

while attaching cpus, if the FDT provides "capacity-dmips-mhz" track
the fastest set, and call cpu_topology_set() with slow=true for any
cpus that are not the fastest.

bug fix for cpu_topology_set(): actually set ci_is_slow for slow cpus.

with this change, and -current's recent scheduler changes, this means
that long running processes run on the faster cores. on RK3399 based
systems, i am seeing 20-50% speed ups for many tasks.


XXX: all this can be made common with armv7 big.little.

branches: 1.17.2;
Give aarch64 a preemption safe cpu_intr_p().

+ ci_onproc

mi_userret(): take care of calling preempt(), set spc_curpriority directly,
and remove MD code that does the same.

Increase aarch64 MAXCPUS to 256.

branches: 1.13.4;
- add workaround for Cavium ThunderX errata 27456.
- add cpufuncs table in cpu_info. each cpu clusters may have different erratum. (e.g. big.LITTLE)

Provide a LWP_PC for Taylor

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

Provide generic start code that assumes the MMU is off and caches are
disabled as per the linux booting protocol for ARMv6 and ARMv7 boards.
u-boot image type should be changed to 'linux' for correct behaviour.

The new start code builds a minimal "bootstrap" L1PT with cached access
disabled and uses the same table for all processors. AP startup is
performed in less steps and more code is written in C.

The bootstrap tables and stack are placed into an (orphaned) section
"_init_memory" which is given to uvm when it is no longer used.

Various kernels have been converted to use this code and tested. Some
boards were provided by TNF. Thanks!

The GENERIC kernel now boots on boards using the TEGRA, SUNXI and EXYNOS
kernels. The GENERIC kernel will also work on RPI2 using u-boot.

Thanks to martin@ and aymeric@ for testing on parallella and nanosoc
respectively

Add ACPI Processor Unique ID (ci_acpiid) to struct cpu_info, required by
ACPI subsystem.

cleanup aarch64 mpstart and fdt bootstrap
* arm_cpu_hatch_arg is a bad idea. avoid serializing CPU startup, and eliminate arm_cpu_hatch_arg.
in mpstart, resolve own cpu index using array of cpu_mpidr[] (aarch64)
* add support fdt enable-method "spin-table"
* add support fdt enable-method "brcm,bcm2836-smp" (for 32bit RaspberryPi)
* use arm_fdt_cpu_bootstrap() instead of psci_fdt_bootstrap()
* rename "arm/fdt/psci_fdt.h" to "arm/fdt/psci_fdtvar.h" because of conflict of include file for needs-flag
* add devmap for cpu spin-table of raspberrypi3/aarch64
* no need to force hatch APs for raspberrypi3/arm32 ifndef MULTIPROCESSOR.
* fix to work pmap_extract(kerneltext/data/bss) even if before calling pmap_bootstrap

idea to use cpu_mpidr[] by jmcneill@. reviewd by skrll@. thanks.

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

Add fields for per-cpu GICv3 state

rather than using flags to resolve nested locks, reserve pool_cache before locking.

* avoid deadlock. mutex_owned() works only for adaptive lock, therefore we cannot use it for spinlock...
* add more NULL check
* clear pte when pmap_enter() fails

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

PR/56002: aarch64 has a true 64bit CPU cycle counter, we will use it.

This fix solves PR/56002 on aarch64, but this problems can occur on
all other architectures where cpu_counter() is 32bit.

branches: 1.1.4; 1.1.42;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

aarch64: Allow for alternate cpu_idle() implementations

branches: 1.27.2;
Remove ryo@'s mail addresses.

Provide a shared pmap_devmap implementation and convert all pmap_devmap
arrays to use DEVMAP_ENTRY{,_END}

fix misspellings of 'available' and nearby typos

aarch64: Make cpufunc.h includable without sys/cpu.h first.

add support Hardware updates to Access flag and Dirty state (FEAT_HAFDBS)

- The DBM bit of the PTE is now used to determine if it is writable, and
the AF bit is treated entirely as a reference bit. A valid PTE is always
treated as readable. There can be no valid PTE that is not readable.
- LX_BLKPAG_OS_{READ,WRITE} are used only for debugging purposes,
and has been superseded by LX_BLKPAG_AF and LX_BLKPAG_DBM.
- Improve comment

The need for reference/modify emulation has been eliminated,
and access/permission faults have been reduced, however,
there has been little change in overall performance.

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

Whitespace

fix build error with options ARMV85_BTI

branches: 1.19.4; 1.19.6;
Remove unnecessary casts

branches: 1.18.2;
Implement MD ucas(9) (__HAVE_UCAS_FULL)

Add support for Privileged Access Never (ARMv8.1-PAN).

PAN provides the same functionality as SMAP on x86: it forbids kernel
access to userland pages when PSTATE.PAN=1, and allows such accesses when
PSTATE.PAN=0.

We clear SCTLR_SPAN, to guarantee that PAN=1 each time the kernel is
entered. We catch PAN faults and panic right away without further
processing. In copyin, copyout, etc, we temporarily authorize access to
userland pages.

PAN is a very useful exploit mitigation. Reviewed by ryo@, thanks. Tested
on Qemu. Enabled by default.

Switch the Icache sync operation to the necessary and sufficient one according to the CTR_EL0.DIC and CTR_EL0.IDC flags.

If CTR_EL0.DIC=1, Icache invalidation is not required.
If CTR_EL0.IDC=1, Dcache clean before Icache invalidation is not required.
CLIDR_EL1.LoC is 0, or CLIDR_EL1.LoUIS and CLIDR_EL1.LoUU are 0, Dcache clean is not required as well.

SEE ALSO ARMARM, "CTR_EL0 Cache Type Register", and "CLIDR_EL1 Cache Level ID Register"

cache information can be detected correctly on newer CPUs

- add VPIPT cache type
- adapt to 64-bit CCSIDR (ARMv8.3-CCIDX)
- CCSIDR:[WT,WB,PA,WA] are deprecated
- show number of cache lines when attaching cpu

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

- move aarch64 addressspace macros from pmap.h to cpufunc.h
- rename ptr_strip_pac() to aarch64_strip_pac()

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.11.4;
port the arm64 cpu topology setup for big.little to arm.

rename arm64 cpu_do_topology() to arm_cpu_do_topology() and
call it from both arm cpu_attach().

replace both aarch64_set_topology() inline code in arm
cpu_attach() with new arm_cpu_do_topology(), which is called
by the arm64 locore as well (possibly not needed, which would
allow it to become static.)

not yet tested on a real big.little armv7 system. tested
on rockpro64 and pinebook pro.

provide some semblance of valid cpu topology for big.little systems.

while attaching cpus, if the FDT provides "capacity-dmips-mhz" track
the fastest set, and call cpu_topology_set() with slow=true for any
cpus that are not the fastest.

bug fix for cpu_topology_set(): actually set ci_is_slow for slow cpus.

with this change, and -current's recent scheduler changes, this means
that long running processes run on the faster cores. on RK3399 based
systems, i am seeing 20-50% speed ups for many tasks.


XXX: all this can be made common with armv7 big.little.

branches: 1.9.2;
aarch64_cache_info[] is not global

Make cache operations available early.

In pmap_devmap_bootstrap(), cpu_earlydevice_va_p() must not return true until *all* devmap tables have been enabled.
console mapping may be present in the last table.

add checking status of MMU and devmap to make _platform_early_putchar() available at all times.

- add workaround for Cavium ThunderX errata 27456.
- add cpufuncs table in cpu_info. each cpu clusters may have different erratum. (e.g. big.LITTLE)

Add missing include for device_t declaration.

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

* fix icache invalidations.
* "ic ivau" (aarch64_icache_sync_range) with VA generates permission fault in some situations, therefore use KSEG address for now.

branches: 1.1.2; 1.1.4;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

Fix crash(8) build

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

Make the ddb for pmap / pte information pmap agnostic

branches: 1.13.4;
- fixed a problem where hardware {break,watch}points other than #0 could not be cleared
- hardware {break,watch}point addresses are now strictly checked

Add support hardware breakpoint and watchpoint again.

Limited support for hardware watchpoint has been available for some time, but it
has not been working properly. In addition, it stopped working at the time of
the PTRACE support commit on 2018-12-13. This has been fixed to work correctly,
and also fixed to be practical by sharing hardware watchpoints and breakpoints
between CPUs on MULTIPROCESSOR.

Also fixed a bug that causes a malfunction when switching CPUs with
"machine cpu N" when entering ddb mode from other than cpu_Debugger().

I have confirmed that the CPU can be switched by "machine cpu N" and return from
ddb properly in each case where ddb is called triggered by ddb break/watchpoint,
hardware break/watchpoint, and cpu_Debugger().

branches: 1.11.2;
sprinkle LE32TOH to fetch instructions on aarch64eb

Determination of A64,A32,T32 for disasm is now done in strrdisasm() instead of the caller.
correctly disassemble by processor state if defined DEBUG_DUMP_ON_USERFAULT or DEBUG_DDB_ON_USERFAULT.

fix to do backtrace properly for running LWPs and cpu_lwp_fork().
when dump of pcb_tf, only the switchframe part is now displayed instead of the whole trapframe.

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

make kernel-groveling crash(8) work on aarch64

use panic() instead of some printf to show fault status.
useful for ddb "show panic" command.

branches: 1.5.2;
Oops, my previous commit is totally wrong. recast mask/pattern list.
pointed out by David Binderman in PR/53224, thanks.

remove suspicious compare, and cleanup complex conditionals.
pointed out PR/53159 by dcb314, thanks.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.2.16;
PR port-arm/50641: src/sys/arch/aarch64/include/db_machdep.h:67: possible bad if test ?

Fix the bl instruction test.

branches: 1.1.4; 1.1.6;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

aarch64/elf_machdep.h: Permit usage alongside <sys/elfdefinitions.h>.

PR lib/59564

aarch64/elf_machdep.h: keep local and psABI definitions in separate groups.

The content of the file is otherwise unchanged.

PR lib/59788

aarch64: Use the ABI-defined value for R_AARCH64_TLSLD_MOVW_DTPREL_G0_NC.

Kernel source code does not use this constant, so this change should
not affect its runtime behavior. Userland toolchains use the
ABI-defined value for this relocation, via their own definitions.

Remove a duplicate definition for R_AARCH64_TLSLD_ADD_DTPREL_HI12.

Use the ABI value for 'R_AARCH64_TLSLD_LDST128_DTPREL_LO12_NC'.

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

fix relocation type. 277 is R_AARCH64_ADD_ABS_LO12_NC

branches: 1.2.2; 1.2.4;
Cleanup and clarify the ELFSIZE mess:

We now have 2 variables automatically set in elf_machdep.h:

ARCH_ELFSIZE: the size for userland binaries
KERN_ELFSIZE: the size for the kernel binaries

DB_ELFSIZE has been deleted and KERN_ELFSIZE should have always the
same values DB_ELFSIZE used to have.

In sys/exec_elf.h, if ELFSIZE is not set, it is set to KERN_ELFSIZE
for the kernel and ARCH_ELFSIZE for userland. These defaults should
eliminate the need for most manual ELFSIZE setting.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Move aarch64/fpu.h to arm/fpu.h.

Draft fpu_kern_enter/leave on aarch64.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

Define lwp_trapframe() macro

branches: 1.2.2; 1.2.6;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add definition for HCR_E2H bit

Slightly clarify, and style.

branches: 1.1.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.3.2;
Back out last change.

Add formatting for aarch64 as using arm ones errors for ll on 64 bit types

Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

implement dumpsys() and friends for arm64.

this is almost a direct copy of the arm code, which is simply
as the basic structures about physical memory are the same
between arm and arm64. the main change i made was to use
the direct map instead of a virtual dump page that is remapped
to whatever physical page is being dumped.

i also changed the existing cpu_kcore_hdr_t to include the
missing number of ram segments.

note that this is not a complete solution for crash dumps yet,
as the libkvm code needs some work. i'm fairly positive that
this side is correct, as i can see the data i expect to see,
but libkvm's _kvm_kvtop() function returns garbage so far.

there is no "minidump" support here yet, ala amd64, but we
probably want it eventually.


ok skrll@.

branches: 1.1.4; 1.1.28; 1.1.30;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

aarch64/lock.h: Need <sys/param.h> for _HARDKERNEL.

Add include guard while here.

XXX Why does this aarch64-specific file have #ifdef __aarch64__?

Use the yield instruction as SPINLOCK_BACKOFF_HOOK for aarch64.

Use <sys/common_lock.h> for !__arm__

branches: 1.2.2; 1.2.4;
Use __ATOMIC_RELAXED in __cpu_simple_lock_init

Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add DISABLE_INTERRUPT_SAVE(), like DISABLE_INTERRUPT() but also returns
the previous state.

Use DISABLE_INTERRUPT_SAVE()/ENABLE_INTERRUPT() in pic_splfuncs instead
of cpsid()/cpsie(). The difference here is the caller no longer specifies
which bits to disable and enable; on arm32 we continue to use I32_bit and
on aarch64 we now consistently toggle both IRQ and FIQ state.

daif_disable: since we read bits before setting them, if the current state
matches the desired state we can skip the daif write

Use reg_daif{set,clr}_write directly instead of daif_{en,dis}able for
ENABLE_INTERRUPT() and DISABLE_INTERRUPT() macros, to avoid an unnecessary
reg_daif_read().

branches: 1.6.2;
Retire arm_[di]sb in favour of the isb() and dsb(sy) macro invocations.

Include aarch64/machdep.h for arm32 compat.

add MULTIPROCESSOR support

branches: 1.3.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.2.12;
allow pcu_save() and pcu_discard() to be called on other threads,
ptrace needs to use it that way.

branches: 1.1.4; 1.1.6; 1.1.10; 1.1.14;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Create a new header lwp_private.h to contain _lwp_getprivate_fast,
_lwp_gettcb_fast, _lwp_settcb and remove them from mcontext.h, so that:
1. we don't need special hacks to hide them
2. we can include <lwp.h> where needed to get the necessary prototypes
without redefining them locally.

Remove ryo@'s mail addresses.

Add FIQ support.

Fix panic caused by modload. http://mail-index.netbsd.org/port-arm/2020/08/30/msg006960.html

The address space reserved for modules may not be mapped in L1-L3.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

Determination of A64,A32,T32 for disasm is now done in strrdisasm() instead of the caller.
correctly disassemble by processor state if defined DEBUG_DUMP_ON_USERFAULT or DEBUG_DDB_ON_USERFAULT.

- On some systems with a different cache line size (and DIC,IDC) per CPU, trap "mrs Xt,ctr_el0" instruction
to return the minimum cache line size of the system to userland.
- add CLIDR_EL1 and CTR_EL0 to struct aarch64_sysctl_cpu_id.

On most systems, cache line size is the same for all CPUs, so this mechanism won't be required.
Rather, this is primarily for errata support, which will be committed later.

Draft fpu_kern_enter/leave on aarch64.

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

Various updates and improvements to cpu start up on arm/aarch64

- start sharing more code around the AP startup messaging.
- call arm_cpu_topology_set early so that ci_core_id is available for
drivers, e.g. bcm2835_intr.c
- both arm and aarch64 now have
- a static cpu_info_store array
- the same arm_cpu_{hatched,mbox}

branches: 1.9.2;
New function cpu_startup_hook on arm.

Called at end of cpu_startup. Can be defined in, e.g., evbarm to do
additional stuff after cpu_startup. Defined as a weak alias to a
function that does nothing, so optional.

ok jmcneill

branches: 1.8.2;
Add vaddr_t initarm(void *);

Missed in previous commit.

Overhaul the API used to fetch and store individual memory cells in
userspace. The old fetch(9) and store(9) APIs (fubyte(), fuword(),
subyte(), suword(), etc.) are retired and replaced with new ufetch(9)
and ustore(9) APIs that can return proper error codes, etc. and are
implemented consistently across all platforms. The interrupt-safe
variants are no longer supported (and several of the existing attempts
at fuswintr(), etc. were buggy and not actually interrupt-safe).

Also augmement the ucas(9) API, making it consistently available on
all plaforms, supporting uniprocessor and multiprocessor systems, even
those that do not have CAS or LL/SC primitives.

Welcome to NetBSD 8.99.37.

Provide generic start code that assumes the MMU is off and caches are
disabled as per the linux booting protocol for ARMv6 and ARMv7 boards.
u-boot image type should be changed to 'linux' for correct behaviour.

The new start code builds a minimal "bootstrap" L1PT with cached access
disabled and uses the same table for all processors. AP startup is
performed in less steps and more code is written in C.

The bootstrap tables and stack are placed into an (orphaned) section
"_init_memory" which is given to uvm when it is no longer used.

Various kernels have been converted to use this code and tested. Some
boards were provided by TNF. Thanks!

The GENERIC kernel now boots on boards using the TEGRA, SUNXI and EXYNOS
kernels. The GENERIC kernel will also work on RPI2 using u-boot.

Thanks to martin@ and aymeric@ for testing on parallella and nanosoc
respectively

make kernel-groveling crash(8) work on aarch64

Refactor code to split aarch{32,64} kernel page tables and VM setup. This
will help re-build the kernel page tables on aarch64 with correct section
mappings.

Implement TRAP_SIGDEBUG for aarch64...
ptraced programs die with:
data_abort_handler, 257: pid 199.1 (a.out): signal 11 (trap 0x82000006) @pc 0, addr 0x0, error=Instruction Abort (EL0)

add MULTIPROCESSOR support

branches: 1.1.2; 1.1.4;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.3.2;
Remove c&p comment that doesn't apply

Introduce _UC_MACHINE_FP() as a macro

_UC_MACHINE_FP() is a helper macro to extract from mcontext a frame pointer.

Don't rely on this interface as a compiler might strip frame pointer or
optimize it making this interface unreliable.


For hppa assume a small frame context, for larger frames FP might be located
in a different register (4 instead of 3).

For ia64 there is no strict frame pointer, and registers might rotate.
Reuse 79 following:

./gcc/config/ia64/ia64.h:#define HARD_FRAME_POINTER_REGNUM LOC_REG (79)

Once ia64 will mature, this should be revisited.

A macro can encapsulate a real function for extracting Frame Pointer on
more complex CPUs / ABIs.


For the remaining CPUs, reuse standard register as defined in appropriate ABI.

The direct users of this macro are LLVM and GCC with Sanitizers.

Proposed on tech-userlevel@.

Sponsored by <The NetBSD Foundation>

branches: 1.1.4; 1.1.22;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Part III of ad's performance improvements for aarch64

- Assembly language stubs for mutex_enter() and mutex_exit().

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add support of ptrace(2) for COMPAT_NETBSD32.

Now, GDB for arm32 is usable for debugging 32bit applications.

OK ryo@

PR port-arm/54702

Add support for earmv6hf binaries on COMPAT_NETBSD32 for aarch64:

- Emulate ARMv6 instructions with cache operations register (c7), that
are deprecated since ARMv7, and disabled on ARMv8 with LP64 kernel.

- ep_machine_arch (default: earmv7hf) is copied from executables, as we
do for mips64. "uname -p" reports earmv6hf if compiled for earmv6hf;
configure scripts etc can determine the appropriate architecture.

Many thanks to ryo@ for helping me to add support of Thumb-mode,
as well as providing exhaustive test cases:

https://github.com/ryo/mcr_test/

We've confirmed:

- Emulation works in Thumb-mode.
- T32 16-bit length illegal instruction results in SIGILL, even if
it is located nearby a boundary b/w mapped and unmapped pages.
- T32 32-bit instruction results in SIGSEGV if it is located across
a boundary b/w mapped and unmapped pages.

XXX
pullup to netbsd-9

branches: 1.2.4;
add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

branches: 1.1.2; 1.1.4;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

Include "opt_param.h" (ifdef _KERNEL_OPT) everywhere that MSGBUFSIZE is
referenced since some sources include <machine/param.h>.

branches: 1.15.4;
Use 32K as the default NFSv3 read and write data sizes on aarch64, matching
i386 and amd64.

branches: 1.14.6;
G/C

branches: 1.13.2;
PR port-arm/54702

Add support for earmv6hf binaries on COMPAT_NETBSD32 for aarch64:

- Emulate ARMv6 instructions with cache operations register (c7), that
are deprecated since ARMv7, and disabled on ARMv8 with LP64 kernel.

- ep_machine_arch (default: earmv7hf) is copied from executables, as we
do for mips64. "uname -p" reports earmv6hf if compiled for earmv6hf;
configure scripts etc can determine the appropriate architecture.

Many thanks to ryo@ for helping me to add support of Thumb-mode,
as well as providing exhaustive test cases:

https://github.com/ryo/mcr_test/

We've confirmed:

- Emulation works in Thumb-mode.
- T32 16-bit length illegal instruction results in SIGILL, even if
it is located nearby a boundary b/w mapped and unmapped pages.
- T32 32-bit instruction results in SIGSEGV if it is located across
a boundary b/w mapped and unmapped pages.

XXX
pullup to netbsd-9

Increase aarch64 MAXCPUS to 256.

branches: 1.11.4;
Increase MSGBUFSIZE

move DEV_BSIZE, DEV_BSHIFT out of MD param.h, they are same on all ports

also move BLKDEV_IOSIZE, MAXPHYS, but allow override since some ports
have different value (powerpc uses NBPG for BLKDEV_IOSIZE, sun2/sun3
have lower MAXPHYS)

ALIGNBYTES32 should be (8 - 1), not (4 - 1) for EABI:
https://nxr.netbsd.org/xref/src/sys/arch/arm/include/cdefs.h#56

Now, sshd for earmv7hf works without problems.
Also fix other users of cmsg(3) API hopefully.

Expose CACHE_LINE_SIZE (and COHERENCY_UNIT) so that fstat can work

Add CPU_THUNDERX which sets COHERENCY_UNIT and CACHE_LINE_SIZE to 128

Respect the __HIDE_DELAY kludge like on other ports.

Switch to NKMEMPAGES_MAX_UNLIMITED.

This aligns aarch64 with our other modern 64-bit ports. Significantly
improves file caching utilization on aarch64 systems with copious RAM.

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

branches: 1.3.2;
Increase default MSGBUFSIZE to match arm32 defaults

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

branches: 1.1.4; 1.1.28; 1.1.30;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

No need to define cpu_{,set}_tlb_info here - just use the
sys/uvm/pmap/pmap_tlb.h versions.

Provide a shared pmap_devmap implementation and convert all pmap_devmap
arrays to use DEVMAP_ENTRY{,_END}

Provide MI PMAP support on AARCH64

KNF + remove unnecessary brackets

KNF.

Line continuation alignment whitespace. NFC.

Use "non-posted" instead of "strongly ordered" to describe nGnRnE mappings

Rename the following defines:
- _ARM_BUS_SPACE_MAP_STRONGLY_ORDERED to BUS_SPACE_MAP_NONPOSTED
- PMAP_DEV_SO to PMAP_DEV_NP
- LX_BLKPAG_ATTR_DEVICE_MEM_SO to LX_BLKPAG_ATTR_DEVICE_MEM_NP
Rename the following option:
- AARCH64_DEVICE_MEM_STRONGLY_ORDERED to AARCH64_DEVICE_MEM_NONPOSTED

Update to support EFI runtime outside the kernel virtual address space
by creating an EFI RT pmap that can be activated / deactivated when
required.

Adds support for EFI RT to ARM_MMU_EXTENDED (ASID) 32-bit Arm machines.

On Arm64 the usage of pmapboot_enter is reduced and the mappings are
created much later in the boot process -- now in cpu_startup_hook.
Backward compatiblity for KVA mapped RT from old bootaa64.efi is
maintained.

Adding support to other platforms should be easier as a result.

Remove unnecessary brackets

Restore the previous pmap_remove_all behaviour as the new method meant
the n1sdp couldn't complete a build.

No noticeable change in kernel build performance.

Use sys/uvm/pmap/pmap_tlb.c on Aarch64 in the same way that some Arm, MIPS,
and some PPC kernels do. This removes the limitation of 256 processes on
CPUs with 8bit ASID field, e.g. Apple M1.

Additionally the following changes have been made

- removed a couple of unnecessary aarch64_tlbi_all calls
- removed any invalidation after freeing page tables due to
_pmap_sweep_pdp. This was never necessary afaict.
- all kernel mappings are marked global and userland mapping not-global.

Performance testing hasn't show a significant difference. The data here
is from building a kernel on an lx2k system with nvme.

before
1489.6u 400.4s 2:40.65 1176.5% 228+224k 0+32289io 57pf+0w
1482.6u 403.2s 2:38.49 1189.9% 228+222k 0+32274io 46pf+0w
1485.4u 402.2s 2:37.27 1200.2% 228+222k 0+32275io 12pf+0w

after
1493.9u 404.6s 2:37.50 1205.4% 227+221k 0+32265io 48pf+0w
1485.0u 408.0s 2:38.54 1194.0% 227+222k 0+32272io 36pf+0w
1484.3u 407.0s 2:35.88 1213.3% 228+224k 0+32268io 14pf+0w

>>> stats.ttest_ind([160.65,158.49,157.27], [157.5,158.54,155.88])
Ttest_indResult(statistic=1.1923622711296888, pvalue=0.2990182944606766)
>>>

Make even more pmap agnostic

branches: 1.47.2;
Make the ddb for pmap / pte information pmap agnostic

branches: 1.46.2;
Make pmapboot_enter panic if anything goes wrong and any mappings overlap
rather than only doing it in locore.S

branches: 1.45.2;
Improve a comment

implement pmap_remove_all().

The size of struct pv_entry has increased, but speed of kernel build has improved by about 1%
exec and exit should have been improved.

branches: 1.43.2;
Make __md_palloc pmap agnostic (think sys/uvm/pmap)

Part IV of ad's performance improvements for aarch64

- Implement pmap_growkernel(), and update kernel pmap's stats with atomics.

- Then, pmap_kenter_pa() and pmap_kremove() no longer need to allocate
memory nor take pm_lock, because they only modify L3 PTEs.

- Then, pm_lock and pp_lock can be adaptive mutexes at IPL_NONE which are
cheaper than spin mutexes.

- Take the pmap's lock in pmap_extract() if not the kernel's pmap, otherwise
pmap_extract() might see inconsistent state.

pmapboot_enter simplication
- bootpage_alloc in asm becomes pmapboot_pagealloc in C
- PMAPBOOT_ENTER_NOBLOCK is removed as it's not used
- PMAPBOOT_ENTER_NOOVERWRITE is removed as it's now always on
- physpage_allocator argument is removed as it's always
pmapboot_pagealloc
- Support for EARLYCONS without CONSADDR is removed so that the identity
map for CONSADDR is always known.

For the assembly files:
2 files changed, 40 insertions(+), 89 deletions(-)

LGTM ryo

- Fix a lock order reversal in pmap_page_protect().

- Make sure pmap is always locked when updating stats; atomics no longer
needed to do that.

- Remove unneeded traversal of pv list in pmap_enter_pv().

- Shrink struct vm_page from 136 to 128 bytes (cache line sized) and struct
pv_entry from 48 to 32 bytes (power of 2 sized).

- Embed a pv_entry in each vm_page. This means PV entries don't need to
be allocated for private anonymous memory / COW pages / most UBC mappings.
Dynamic PV entries are then used only for stuff like shared libraries and
shared memory.

Proposed on port-arm@.

Use MUTEX_NODEBUG for PV locks as is commonly done. OK ryo.

- move aarch64 addressspace macros from pmap.h to cpufunc.h
- rename ptr_strip_pac() to aarch64_strip_pac()

use PMAP_PAGE_INIT() to initialize mutex in pmap_page.

VM_MDPAGE_INIT() in pmap_free_pdp() had initialized pp_flags,
so it unintentionally cleared PMAP_PAGE_FLAGS_PV_TRACKED.
use PMAP_PAGE_INIT to avoid using PMAP_PAGE_FLAGS_PV_TRACKED.

pointed out by tnn@, thanks

add helper function aarch64_addresspace() and aarch64_untag_address() to check address space, and eliminate address tag

replace KSEG pages mapping code with generic function pmapboot_enter_range()

use LIST(3) instead of TAILQ(3) to save one word in struct vm_page and struct pmap.

pointed out by riastradh@. thanks

add support pmap_pv(9)

Patch originally from jmcneill@. thanks

separate struct vm_page_md into vm_page_md and pmap_page
for preparation pmap_pv(9)

Typo in comment

branches: 1.30.2;
Fix DEVMAP build losage by reducing diffs between arm and aarch64

*sigh*

Drop DEVMAP_{TRUNK_ADDR,ROUND_SIZE} to 4KB pages now that pmap_map_chunk
allows this.

Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

Define PMAP_NEED_PROCWR, providing strategically placed i-cache
synchronization where just-changed memory is about to be executed.

Fixes SIGILLs seen when running Mono 6 on QEMU Cortex-A57.

ok ryo

Use PMAP_DEV in DEVMAP_ENTRY rather than pmap_map_chunk. It's clearer and
means pmap_map_chunk can be made to map other memory types.

branches: 1.24.4;
- free empty page tables pages if reach a certain usage.
- need to lock at removing an old pg (_pmap_remove_pv) in _pmap_enter()

- add ddb command "machine ttbr" to dump MMU tables.
- tidy up descriptions, usages and messages.

- free L1-L3 pages that has been emptied by pmap_remove().
- if no memories, pmap_enter will return correctly ENOMEM if PMAP_CANFAIL, or wait until available any memories if !PMAP_CANFAIL.

These changes improves the stability when we use a huge virtual memory spaces with mmap.

improve pmap_remove
- don't lock/unlock per page in pmap_remove()
- speedup pte lookup for continuous addresses
- bring out pool_cache_put(&_pmap_pv_pool, pv) from lock/unlock section

re-apply rev. 1.18, now tested by Jonathan Kollasch and Ryo Shimizu - no
problems observed, and about 2x speedup for cached read

Implement PMAP_DIRECT / pmap_direct_process() in support of experimental
UBC optimization

PR kern/53124

revert PMAP_DIRECT until tested; requested by mrg@

Implement PMAP_DIRECT / pmap_direct_process() in support of experimental
UBC optimizations (compile-tested only for now)

PR kern/53124

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

Provide generic start code that assumes the MMU is off and caches are
disabled as per the linux booting protocol for ARMv6 and ARMv7 boards.
u-boot image type should be changed to 'linux' for correct behaviour.

The new start code builds a minimal "bootstrap" L1PT with cached access
disabled and uses the same table for all processors. AP startup is
performed in less steps and more code is written in C.

The bootstrap tables and stack are placed into an (orphaned) section
"_init_memory" which is given to uvm when it is no longer used.

Various kernels have been converted to use this code and tested. Some
boards were provided by TNF. Thanks!

The GENERIC kernel now boots on boards using the TEGRA, SUNXI and EXYNOS
kernels. The GENERIC kernel will also work on RPI2 using u-boot.

Thanks to martin@ and aymeric@ for testing on parallella and nanosoc
respectively

- define PMAP_{MAP,UNMAP}_POOLPAGE for performance
- define __HAVE_MM_MD_KERNACC and add mm_md_kernacc()

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

rewrite pmap_pte_lookup() to share similar code.

cleanup locore, and changed the way to map memories during boot.
- add functions bootpage_enter() and bootpage_alloc() to adapt various layout
of physical memory map. especially for 64bit physical memory layout.
pmapboot_alloc() allocates pagetable pages from _end[].
- changed to map only the required amount for PA=VA identity mapping
(kernel image, UART device, and FDT blob) with L2_BLOCK(2Mbyte).
- changing page permission for kernel image, and making KSEG mapping are done
at cpu_kernel_vm_init() instead of at locore.
- optimize PTE entries with PTE Contiguous bit. it is enabled on devmap only for now.

reviewed by skrll@, thanks.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

make kernel-groveling crash(8) work on aarch64

Rename _pmap_alloc_pdp -> pmap_alloc_pdp, and make it public.

treat kernel-exec attr and user-exec attr separately.
kernel cannot execute userland exec page, and user cannot execute kernel page.

set kernel text/rodata readonly by default.
add function db_write_text() for setting ddb breakpoint.

changes of pmap.c r1.13 seems to be unstable.
In order to invalidate icache, not to invalidate all icache,
but temporary to make the page writable and invalidate target address only.

branches: 1.5.2;
Provide bs_mmap implementations for bcm283x based boards.

PR: port-arm/53283
Submitted by: Nick Hudson

fix instability behavior of bufcache on aarch64.
* fix to return correct ref/mod when PMAP_WIRED.
* changed to keep wired flags in pte instead of pv_entry, and cleanup.

Fix encoding of MMAP flags for generic_bs_mmap

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

There is no meed to dsb(ishst) after every pte_pde_cas - the necessary
dsb will be performed later when a pte is added,

Allow building w/o options EFI_RUNTIME

branches: 1.8.6;
blank line audit

G/C pmap_md_kernel_*

Reduce #ifdefs

Wrap long lines in a comment block.

spaces to tabs.

Provide a shared pmap_devmap implementation and convert all pmap_devmap
arrays to use DEVMAP_ENTRY{,_END}

Rename pmap_md_pdetab_destroy to pmap_md_pdetab_fini to match
pmap_md_pdetab_init.

Call pmap_md_pdetab_fini from pmap_segtab_destroy.

Provide MI PMAP support on AARCH64

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.5.6;
part of PR port-arm/54702

Having md_march32 unconditionally in struct mdproc, in order to
make libkvm happy.

XXX
pullup to netbsd-9

PR port-arm/54702

Add support for earmv6hf binaries on COMPAT_NETBSD32 for aarch64:

- Emulate ARMv6 instructions with cache operations register (c7), that
are deprecated since ARMv7, and disabled on ARMv8 with LP64 kernel.

- ep_machine_arch (default: earmv7hf) is copied from executables, as we
do for mips64. "uname -p" reports earmv6hf if compiled for earmv6hf;
configure scripts etc can determine the appropriate architecture.

Many thanks to ryo@ for helping me to add support of Thumb-mode,
as well as providing exhaustive test cases:

https://github.com/ryo/mcr_test/

We've confirmed:

- Emulation works in Thumb-mode.
- T32 16-bit length illegal instruction results in SIGILL, even if
it is located nearby a boundary b/w mapped and unmapped pages.
- T32 32-bit instruction results in SIGSEGV if it is located across
a boundary b/w mapped and unmapped pages.

XXX
pullup to netbsd-9

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Oh...the name of the mcount call was different between gcc and llvm.
gcc calls it as "_mconut", llvm calls as "__mcount".

Change the main name of mcount to "mcount()", and created "_mcount" and "__mcount" entries
to work regardless of which compiler the object was created with.

add support kernel profiling on aarch64

- add MCOUNT_ENTER, MCOUNT_EXIT macro
- __mcount() function should be aligned
- add "-fno-optimize-sibling-calls" option when PROF. for accurate profiling, it is better to suppress the tail call.

branches: 1.2.2;
Fix userland gprof profiling on aarch64.

Adjusts _PROF_PROLOGUE to match OpenBSD; reworks our MCOUNT to retrieve
frompc placed on stack by the prologue, and to streamline sp manipulation
when preserving argument registers.

branches: 1.1.4; 1.1.40;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Fixed a bug that pte's __BIT(63,48) could be set when accessing addresses above 0x0001000000000000 in /dev/mem with mmap().

Use sys/uvm/pmap/pmap_tlb.c on Aarch64 in the same way that some Arm, MIPS,
and some PPC kernels do. This removes the limitation of 256 processes on
CPUs with 8bit ASID field, e.g. Apple M1.

Additionally the following changes have been made

- removed a couple of unnecessary aarch64_tlbi_all calls
- removed any invalidation after freeing page tables due to
_pmap_sweep_pdp. This was never necessary afaict.
- all kernel mappings are marked global and userland mapping not-global.

Performance testing hasn't show a significant difference. The data here
is from building a kernel on an lx2k system with nvme.

before
1489.6u 400.4s 2:40.65 1176.5% 228+224k 0+32289io 57pf+0w
1482.6u 403.2s 2:38.49 1189.9% 228+222k 0+32274io 46pf+0w
1485.4u 402.2s 2:37.27 1200.2% 228+222k 0+32275io 12pf+0w

after
1493.9u 404.6s 2:37.50 1205.4% 227+221k 0+32265io 48pf+0w
1485.0u 408.0s 2:38.54 1194.0% 227+222k 0+32272io 36pf+0w
1484.3u 407.0s 2:35.88 1213.3% 228+224k 0+32268io 14pf+0w

>>> stats.ttest_ind([160.65,158.49,157.27], [157.5,158.54,155.88])
Ttest_indResult(statistic=1.1923622711296888, pvalue=0.2990182944606766)
>>>

Fix pmap to work correctly with tagged addresses

- when fault, untag from address before passing to uvm/pmap functions
- pmap_extract() checks more strictly and consider the address tag

BTI definitions.

branches: 1.10.2;
Define PRIxPTE

Move the TCR and TTBR defines into armreg.h where they below. NFCI.

- Fix TCR_TG0 field definitions to match Armv8 ARM
- Rename TCR_IPS_64TB to TCR_IPS_16TB, add TCR_IPS_4PB
- Whitespace fixes

Indent the field value defines. NFCI.

Add DBM

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

fix build with aarch64 gcc/gas

branches: 1.3.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.2.12;
Correct definitions for TCR.

Values from ARM Cortex A-53 MPCore Processor Technical Reference Manual
4.3.48. Translation Control Register, EL1

branches: 1.1.4; 1.1.6; 1.1.10; 1.1.14;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Oops. revert my previous commit. AArch64 instructions are always LE.

need swap for aarch64be

Add support of ptrace(2) for COMPAT_NETBSD32.

Now, GDB for arm32 is usable for debugging 32bit applications.

OK ryo@

branches: 1.9.2;
Introduce PTRACE_REG_FP() a helper macro to retrieve the frame pointer

The macro is dummy for ia64 (the FP register is unknown and can change
freely) and sparc/sparc64 (not stored in struct reg).

add support PT_STEP

don't depend endian.

flip the byte order

branches: 1.5.10; 1.5.12;
Add new macro PTRACE_BREAKPOINT_ASM in <sys/ptrace.h> MD part

This macro ships with a MD-specific assembly instruction triggering
a software breakpoint.

Missing instruction for powerpc targets.

This code is used in ATF tests (lib/libc/sys/t_ptrace_wait).

Original patch by Nick Hudson, thanks!

branches: 1.4.2; 1.4.4;
For processors that have memory breakpoints, add macros for them to help
libproc

Provide access to pc/sp/syscall-return registers like we have for mcontext

branches: 1.2.2; 1.2.4;
#include <arm/ptrace.h> instead of <arm/asm.h>
(opps)

Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Use __uint128_t conditionally in aarch64 reg.h

Check whether __uint128_t is available checking __SIZEOF_INT128__ in
preprocessor.
Move __aligned attribute to the whole structure.

No functional change for current NetBSD/aarch64 users of GCC and Clang.

This change allows to use the aarch64 target with rumpkernel on Linux
aarch64 hosts, in a toolchain configuration with 128-bit variables.

OK from <martin> and <christos>

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Don't futz with tpidr_el0 in {set,long}jmp as it breaks TLS as seen in
qemu

branches: 1.1.4; 1.1.34;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Normally, to support COMPAT_NETBSD32 we need to define
__HAVE_STRUCT_SIGCONTEXT in order to support the old
"sigcontext" style of handlers for 32-bit binaries.
However, we only support 32-bit EABI binaries on AArch64,
and by happy accident (due to a libc bug introduced in
2006), 32-bit NetBSD EABI binaries never used "sigcontext"
style handlers. So, we don't need to carry any of this
baggage forward.

This addresses the AArch64 case of PR kern/56487.

- In sendsig() and sigaction1(), don't hard-code signal trampoline
versions. Instead, use the version constants from <sys/signal.h>
and automatically (and correctly) handle cases where multiple versions
of a particular trampoline flavor exist. Conditionalize support
for sigcontext trampolines on __HAVE_STRUCT_SIGCONTEXT.
- aarch64 and amd64 don't use sigcontext natively, but do need to
support it for 32-bit compatibility; define __HAVE_STRUCT_SIGCONTEXT
conditionally on _KERNEL.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

aarch64/sljit_machdep.h: Make this work in compat32 context.

Should fix clang build of compat32 eabi libsljit:

dependall ===> compat/arm/eabi/../../../lib/../external/bsd/sljit/lib
In file included from /home/source/ab/HEAD-llvm/src/sys/external/bsd/sljit/dist/sljit_src/sljitLir.c:1678:
/home/source/ab/HEAD-llvm/src/sys/external/bsd/sljit/dist/sljit_src/sljitNativeARM_64.c:142:54: error: shift count >= width of type [-Werror,-Wshift-count-overflow]
FAIL_IF(push_inst(compiler, MOVK | RD(dst) | (((imm >> 32) & 0xffff) << 5) | (2 << 21)));
^ ~~

bsd.own.mk: Enable MKLSJIT on aarch64.

Make sure there's only one copy of the conditional, in bsd.own.mk;
just make sys/modules/Makefile conditional on MKSLJIT so we don't
have to keep these in sync.

As a workaround for PR 58106, tweak the conditional definition of
SLJIT_CACHE_FLUSH to use cpu_icache_sync_range only in _HARDKERNEL,
and use __builtin___clear_cache in userland and in rump kernels.

PR 58103: bpfjit.kmod is not built on aarch64

branches: 1.3.18;
s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

branches: 1.2.4; 1.2.6; 1.2.14;
Switch to __builtin___clear_cache() in userspace.

aarch64_sync_icache() doesn't exist because there no libarm equivalent
on aarch64.

branches: 1.1.2;
Add SLJIT to aarch64.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.3.2;
Provide a trap.h (currently empty)

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Provide MI PMAP support on AARCH64

Use sys/uvm/pmap/pmap_tlb.c on Aarch64 in the same way that some Arm, MIPS,
and some PPC kernels do. This removes the limitation of 256 processes on
CPUs with 8bit ASID field, e.g. Apple M1.

Additionally the following changes have been made

- removed a couple of unnecessary aarch64_tlbi_all calls
- removed any invalidation after freeing page tables due to
_pmap_sweep_pdp. This was never necessary afaict.
- all kernel mappings are marked global and userland mapping not-global.

Performance testing hasn't show a significant difference. The data here
is from building a kernel on an lx2k system with nvme.

before
1489.6u 400.4s 2:40.65 1176.5% 228+224k 0+32289io 57pf+0w
1482.6u 403.2s 2:38.49 1189.9% 228+222k 0+32274io 46pf+0w
1485.4u 402.2s 2:37.27 1200.2% 228+222k 0+32275io 12pf+0w

after
1493.9u 404.6s 2:37.50 1205.4% 227+221k 0+32265io 48pf+0w
1485.0u 408.0s 2:38.54 1194.0% 227+222k 0+32272io 36pf+0w
1484.3u 407.0s 2:35.88 1213.3% 228+224k 0+32268io 14pf+0w

>>> stats.ttest_ind([160.65,158.49,157.27], [157.5,158.54,155.88])
Ttest_indResult(statistic=1.1923622711296888, pvalue=0.2990182944606766)
>>>

Make tlb_asid_t unsigned int as pmap_tlb.c expects tlb_asid_t to be able to
hold ASID_MAX + 1.

s/depreciated/deprecated/g

branches: 1.17.2;
Add __HAVE_BUS_SPACE_8

branches: 1.16.2;
PID_MAX is just an initial value (soft maximum). Don't use it for CTASSERT.
defined __HAVE_CPU_MAXPROC to use function cpu_maxproc().

pointed out by mrg@, thanks.

Implement MD ucas(9) (__HAVE_UCAS_FULL)

sort __HAVE_* defines. NFCI

branches: 1.13.2;
Remove __HAVE_CPU_LWP_SETPRIVATE from aarch64

aarch64 specific cpu_lwp_setprivate() is redundant with its caller
lwp_setprivate() and there are no MD bits.

- define PMAP_{MAP,UNMAP}_POOLPAGE for performance
- define __HAVE_MM_MD_KERNACC and add mm_md_kernacc()

Be consistent and explicitly size register32_t too.

match declaration types for registers from reg.h

Remove the kernel PMC code. Sent yesterday on tech-kern@.

This change:

* Removes "options PERFCTRS", the associated includes, and the associated
ifdefs. In doing so, it removes several XXXSMPs in the MI code, which is
good.

* Removes the PMC code of ARM XSCALE.

* Removes all the pmc.h files. They were all empty, except for ARM XSCALE.

* Reorders the x86 PMC code not to rely on the legacy pmc.h file. The
definitions are put in sysarch.h.

* Removes the kern/sys_pmc.c file, and along with it, the sys_pmc_control
and sys_pmc_get_info syscalls. They are marked as OBSOL in kern,
netbsd32 and rump.

* Removes the pmc_evid_t and pmc_ctr_t types.

* Removes all the associated man pages. The sets are marked as obsolete.

branches: 1.8.2;
Define __HAVE_OLD_DISKLABEL for compatibility with the arm32 port.

define __HAVE_ATOMIC64_OPS
pointed out by nonaka@, thanks

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.5.16;
Reorder using register_t to the point where it is defined.

expose the kernel types for standalone code.

Hide {p,v}{addr,size}_t and register_t (and a couple more types that
are machine-specific) from userland unless _KERNEL/_KMEMUSER and a
new _KERNTYPES variables is defined. The _KERNTYPES should be fixed
for many subsystems that should not be using it (rump)...

Fix PTHREAD_FOO_INITIALIZER for C++ by not using volatile in the relevant
pthread types in C++ builds, attempt 2.

The problem with attempt 1 was making assumptions of what the MD
__cpu_simple_lock_t (declared volatile) looks like. To get a same type
except non-volatile, we change the MD type to __cpu_simple_lock_nv_t
and typedef __cpu_simple_lock_t as a volatile __cpu_simple_lock_nv_t.
IMO, __cpu_simple_lock_t should not be volatile at all, but changing it
now is too risky.

Fixes at least Rumprun w/ gcc 5.1/5.2. Furthermore, the mpd application
(and possibly others) will no longer require NetBSD-specific patches.

Tested: build.sh for i386, Rumprun for x86_64 w/ gcc 5.2.

Based on the patch from Christos in lib/49989.

branches: 1.1.4; 1.1.6;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

aarch64: Bump VM_PHYSSEG_MAX to match DRAM_BANKS / FDT_MEMORY_RANGES.

On aarch64 there is a single free list, so VM_PHYSSEG_MAX needs to be
the same as FDT_MEMORY_RANGES (which needs to be the same as DRAM_BANKS).
Future cleanup should be done to fold these into a single define.

branches: 1.20.6;
Rename VM_KERNEL_IO_ADDRESS to VM_KERNEL_IO_BASE to match RISC-V

It's less letters, matches other similar variables and will help with
sharing code between the two architectures.

NFCI.

branches: 1.19.4;
Update to support EFI runtime outside the kernel virtual address space
by creating an EFI RT pmap that can be activated / deactivated when
required.

Adds support for EFI RT to ARM_MMU_EXTENDED (ASID) 32-bit Arm machines.

On Arm64 the usage of pmapboot_enter is reduced and the mappings are
created much later in the boot process -- now in cpu_startup_hook.
Backward compatiblity for KVA mapped RT from old bootaa64.efi is
maintained.

Adding support to other platforms should be easier as a result.

Adjust the kernel virtual address space so that KASAN will map the kernel
seperately from managed kernel virtual memory and not map the unused space
between the two.

branches: 1.17.2;
AA64 is not MIPS.

Change all KSEG references to directmap

branches: 1.16.2;
GC unused MAXTSIZ32

Readability of a comment

Define VM_KERNEL_VM_{BASE,SIZE} for aarch64 and remove an #ifdef in
fdt/platform.h

NFCI

G/C AARCH64_KMEMORY_BASE

Fix a comment

change kernel vm base address to use more than 256GB of memory. (up to 64TB)

also enlarge KSEG(direct map) region from 512GB to 64TB.
KASAN works ok.

Note: -fasan-shadow-offset=
KASAN_SHADOW_START - (CANONICAL_BASE >> 3) =
0xFFFF400000000000 - (0xFFFF000000000000 >> 3) =
0xDFFF600000000000

Bump UBC defaults on sparc64 & aarch64, which already have a large pager_map.

Switch aarch64 to use a single freelist.

branches: 1.8.6;
Document the VA range reserved for EFI runtime services.

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

define VM_KERNEL_IO_SIZE for clarity

Increase VM_PHYSSEG_MAX to 64

branches: 1.4.2;
Increase PAGER_MAP_DEFAULT_SIZE to 512MB (from 16MB)

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.2.2; 1.2.20;
Add some definitions for building RUMP libraries with MKCOMPAT.

Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.