branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove an item

Implement MD ucas(9) (__HAVE_UCAS_FULL)

remove DTrace from TODO list

Add DTrace.

COMPAT_NETBSD32 to work on also thumbmode

some items are already done. update and sync with reality.

Overhaul the API used to fetch and store individual memory cells in
userspace. The old fetch(9) and store(9) APIs (fubyte(), fuword(),
subyte(), suword(), etc.) are retired and replaced with new ufetch(9)
and ustore(9) APIs that can return proper error codes, etc. and are
implemented consistently across all platforms. The interrupt-safe
variants are no longer supported (and several of the existing attempts
at fuswintr(), etc. were buggy and not actually interrupt-safe).

Also augmement the ucas(9) API, making it consistently available on
all plaforms, supporting uniprocessor and multiprocessor systems, even
those that do not have CAS or LL/SC primitives.

Welcome to NetBSD 8.99.37.

update TODO
* done: kernel text/rodata mapping with correct permission. implemented.
* add: kernel preemption

check off SMP

branches: 1.1.2; 1.1.4;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

Remove ryo@'s mail addresses.

Arrange to update cached LWP credentials in userret() rather than during
syscall/trap entry, eliminating a test+branch on every syscall/trap.

This wasn't possible in the 3.99.x timeframe when l->l_cred came about
because there wasn't a reliable/timely way to force an ONPROC LWP running on
a remote CPU into the kernel (which is just about the only new thing in
this scheme).

aarch64: curcpu() audit.

Sprinkle KASSERT (or KDASSERT in hot paths) for kpreempt_disabled()
when we use curcpu() and it's not immediately obvious that the caller
has preemption disabled but closer scrutiny suggests the caller has.

Note unsafe curcpu()s for syscall event counting. Not sure this is
worth changing.

Possible bugs fixed:

- cpu_irq and cpu_fiq could be preempted while trying to run softints
on this CPU.

- data_abort_handler might incorrectly think it was invoked in
interrupt context when it was only preempted and migrated to
another CPU.

- pmap_fault_fixup might report the wrong CPU logs.

(However, we don't currently run with kpreemption on aarch64, so
these are not yet real bugs fixed except if you patch it to build
with __HAVE_PREEMPTION.)

add support COMPAT_LINUX32 for aarch64

Wrap long line. No binary changes.

Fix __syscall(2) for COMPAT_NETBSD32 on aarch64{,eb}.

The 1st argument for __syscall(2) is quad_t, which is stored in r0 and r1.

Now, tests/lib/libc/t_syscall:mmap___syscall passes for COMPAT_NETBSD32.

branches: 1.3.4; 1.3.18; 1.3.20;
COMPAT_NETBSD32 to work on also thumbmode

no need to include opt_multiprocessor.h here.

branches: 1.1.2;
add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

Refactor the "platform" defitions into fdt_platform.h

aarch64: Omit needless xcfunc_t casts by using xcfunc_t correctly.

No functional change intended, except for avoiding possible undefined
behaviour that could have made demons come flying out your nose.

G/C an outdated comment.

Rename VM_KERNEL_IO_ADDRESS to VM_KERNEL_IO_BASE to match RISC-V

It's less letters, matches other similar variables and will help with
sharing code between the two architectures.

NFCI.

Rename ARM_PLATFORM to FDT_PLATFORM and make it available outside arm.

Fixed a bug that pte's __BIT(63,48) could be set when accessing addresses above 0x0001000000000000 in /dev/mem with mmap().

No need to call arm_fdt_platform twice.

add support Hardware updates to Access flag and Dirty state (FEAT_HAFDBS)

- The DBM bit of the PTE is now used to determine if it is writable, and
the AF bit is treated entirely as a reference bit. A valid PTE is always
treated as readable. There can be no valid PTE that is not readable.
- LX_BLKPAG_OS_{READ,WRITE} are used only for debugging purposes,
and has been superseded by LX_BLKPAG_AF and LX_BLKPAG_DBM.
- Improve comment

The need for reference/modify emulation has been eliminated,
and access/permission faults have been reduced, however,
there has been little change in overall performance.

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

Use BOOT_FLAG() to recognize standard boot options.

Two fixes for loading free pages into UVM

- Only consider a boot_physmem (inner loop) range that has its end
(bp_end) after the bootconfig.dram (outer loop) range start (start).
This was harmless as a later condition correctly checks there is only
something to do if start < bp_end.

- Stop processing boot_physmem ranges if all the bootconfig.dram range has
been passed to UVM. This fixes a boot problem for simon@

branches: 1.60.2; 1.60.6;
More debug

Update a comment to reflect reality

Adjust the kernel virtual address space so that KASAN will map the kernel
seperately from managed kernel virtual memory and not map the unused space
between the two.

Tweak a comment

branches: 1.56.2;
Move evbarm/fdt/fdt_memory.[ch] to sys/dev/fdt and simplify the api and
some operations. This allows other architectures to use it.

Remove unnecessary aarch64_dcache_wbinv_all now that pmapboot_enter does
dsb(ish)

AA64 is not MIPS.

Change all KSEG references to directmap

branches: 1.53.2;
Use the dmb/dsb/isb macros... if nothing else they're all now consistent
about the "memory" assembler contraint.

No binary change

Simplify the cpufunc.h header, i.e. always use #include <arm/cpufunc.h>

KNF

G/C

Improve a comment

G/C AARCH64_KMEMORY_BASE

Fix a comment

Add support for Privileged Access Never (ARMv8.1-PAN).

PAN provides the same functionality as SMAP on x86: it forbids kernel
access to userland pages when PSTATE.PAN=1, and allows such accesses when
PSTATE.PAN=0.

We clear SCTLR_SPAN, to guarantee that PAN=1 each time the kernel is
entered. We catch PAN faults and panic right away without further
processing. In copyin, copyout, etc, we temporarily authorize access to
userland pages.

PAN is a very useful exploit mitigation. Reviewed by ryo@, thanks. Tested
on Qemu. Enabled by default.

pmapboot_enter simplication
- bootpage_alloc in asm becomes pmapboot_pagealloc in C
- PMAPBOOT_ENTER_NOBLOCK is removed as it's not used
- PMAPBOOT_ENTER_NOOVERWRITE is removed as it's now always on
- physpage_allocator argument is removed as it's always
pmapboot_pagealloc
- Support for EARLYCONS without CONSADDR is removed so that the identity
map for CONSADDR is always known.

For the assembly files:
2 files changed, 40 insertions(+), 89 deletions(-)

LGTM ryo

- On some systems with a different cache line size (and DIC,IDC) per CPU, trap "mrs Xt,ctr_el0" instruction
to return the minimum cache line size of the system to userland.
- add CLIDR_EL1 and CTR_EL0 to struct aarch64_sysctl_cpu_id.

On most systems, cache line size is the same for all CPUs, so this mechanism won't be required.
Rather, this is primarily for errata support, which will be committed later.

Add support for installing modules that were loaded by the bootloader.

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.40.4;
add support userspace tagged address for aarch64 (experimental)

'sysctl machdep.tagged_address' to set/clear TCR_EL1.TBI0 to enable/disable address tagging.
with 'machdep.tagged_address=1', some syscalls may cause problems?

replace KSEG pages mapping code with generic function pmapboot_enter_range()

Fixup a comment

branches: 1.37.2;
fix panic when modload.

>panic: kernel diagnostic assertion "!pmap_extract(pmap_kernel(), loopva, NULL)" failed: file "../../../../uvm/uvm_km.c", line 674 loopva=0xffffffc001000000'

The space allocated by bootpage_alloc() is only used as a physical page
for pagetable pages, so there is no need to map it with KVA.
And kernend_extra should not have consumed any KVA space.

Flush the cache and disable TTBR0 translations once we're done with
them in cpu_kernel_vm_init

The Cortex A72s in RPI4 need the cache flush for some reason.

New function cpu_startup_hook on arm.

Called at end of cpu_startup. Can be defined in, e.g., evbarm to do
additional stuff after cpu_startup. Defined as a weak alias to a
function that does nothing, so optional.

ok jmcneill

Mark several kASan functions with __nothing, to avoid annoying #ifdefs.
Same as kCSan and kMSan.

Don't include "opt_kasan.h" when there's already <sys/asan.h> included.

Whitespace

used L3 even if L2 could cover the range. fix to use larger block if possible good enough.
pointed out by jmcneill@. thanks.

use L1-L3 blocks/pages for KSEG mappings to fit dramblocks exactly.
r1.29 and this changes avoid over cache prefetch problem (perhaps) with PMAP_MAP_POOLPAGE/KSEG on CortexA72, and be more stable for rockpro64.

Do not assume that DRAM is linear when creating KSEG mappings. Instead,
create L2 blocks to cover all ranges specified in the memory map.

branches: 1.28.4;
Merge the [pgoyette-compat] branch

KNF

redo the previous using ptoa(). also apply to another instance of
the same integer overflow, and now savecore actually does something
in the OD1K.

avoid integer overflow when calculating the end address of a ram
block. fixes a bug when a PhysMem range covers more than 4GB.

with this, my OD1K (8GB ram) is almost able to properly coredump.
savecore finds the core, but can't read it properly.

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

support boot option "-1" to disable multiprocessor boot, and "-z" to set AB_SILENT flag.

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

Physical end should be the end of the last range, not the first.

need to invalidate TLBs after pmapboot_enter(). pmapboot_protect() requires KSEG mappings.

implement dumpsys() and friends for arm64.

this is almost a direct copy of the arm code, which is simply
as the basic structures about physical memory are the same
between arm and arm64. the main change i made was to use
the direct map instead of a virtual dump page that is remapped
to whatever physical page is being dumped.

i also changed the existing cpu_kcore_hdr_t to include the
missing number of ram segments.

note that this is not a complete solution for crash dumps yet,
as the libkvm code needs some work. i'm fairly positive that
this side is correct, as i can see the data i expect to see,
but libkvm's _kvm_kvtop() function returns garbage so far.

there is no "minidump" support here yet, ala amd64, but we
probably want it eventually.


ok skrll@.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

Implement parse_mi_bootargs for aarch64

changes of r1.14 was incomplete. use bootconfig.dram[] to resolve valid memory range.
pmap(1) failed to access kvm on some environment.

Use __nothing

- define PMAP_{MAP,UNMAP}_POOLPAGE for performance
- define __HAVE_MM_MD_KERNACC and add mm_md_kernacc()

Add optional ap_startup callback to struct arm_platform. This allows for
late (post-UVM init) initialization of platform specific stuff.

cleanup locore, and changed the way to map memories during boot.
- add functions bootpage_enter() and bootpage_alloc() to adapt various layout
of physical memory map. especially for 64bit physical memory layout.
pmapboot_alloc() allocates pagetable pages from _end[].
- changed to map only the required amount for PA=VA identity mapping
(kernel image, UART device, and FDT blob) with L2_BLOCK(2Mbyte).
- changing page permission for kernel image, and making KSEG mapping are done
at cpu_kernel_vm_init() instead of at locore.
- optimize PTE entries with PTE Contiguous bit. it is enabled on devmap only for now.

reviewed by skrll@, thanks.

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

Sprinkle __unused

MODULAR support

Refactor code to split aarch{32,64} kernel page tables and VM setup. This
will help re-build the kernel page tables on aarch64 with correct section
mappings.

Keep in .data by using a section attribute.

kern_vtopdiff is stored in fdt_start.S. that is before cleaning bss.
decl "kern_vtopdiff = 0" for keep in .data section.

remove unused variables, add missing casts

branches: 1.4.2;
style

add sysctl for machdep.{cpu_id,id_revidr,id_mvfr,id_mpidr,id_aa64isar,id_aa64mmfr,id_aa64pfr}
each corresponding to system registers MIDR_EL1, REVIDR_EL1, MVFR*_EL1, MPIDR_EL1, ID_AA64ISAR*_EL1, ID_AA64MMFR*_EL1, ID_AA64PFR*_EL1.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove the advertising clause from Microsoft's licenses.

Thanks to Scott Hanselman at Microsoft for turning the wheels to get this
approved!

Move the at-shutdown call to resettodr() from cpu_reboot() to kern_reboot().

It's a small step, but it's a step.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

branches: 1.3.12;
implement dumpsys() and friends for arm64.

this is almost a direct copy of the arm code, which is simply
as the basic structures about physical memory are the same
between arm and arm64. the main change i made was to use
the direct map instead of a virtual dump page that is remapped
to whatever physical page is being dumped.

i also changed the existing cpu_kcore_hdr_t to include the
missing number of ram segments.

note that this is not a complete solution for crash dumps yet,
as the libkvm code needs some work. i'm fairly positive that
this side is correct, as i can see the data i expect to see,
but libkvm's _kvm_kvtop() function returns garbage so far.

there is no "minidump" support here yet, ala amd64, but we
probably want it eventually.


ok skrll@.

branches: 1.2.2;
docpureset() doesn't return anything, so mark it void.
(probably could also be __dead.)

branches: 1.1.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

Fix length of memset in tlb_record_asids

Use sys/uvm/pmap/pmap_tlb.c on Aarch64 in the same way that some Arm, MIPS,
and some PPC kernels do. This removes the limitation of 256 processes on
CPUs with 8bit ASID field, e.g. Apple M1.

Additionally the following changes have been made

- removed a couple of unnecessary aarch64_tlbi_all calls
- removed any invalidation after freeing page tables due to
_pmap_sweep_pdp. This was never necessary afaict.
- all kernel mappings are marked global and userland mapping not-global.

Performance testing hasn't show a significant difference. The data here
is from building a kernel on an lx2k system with nvme.

before
1489.6u 400.4s 2:40.65 1176.5% 228+224k 0+32289io 57pf+0w
1482.6u 403.2s 2:38.49 1189.9% 228+222k 0+32274io 46pf+0w
1485.4u 402.2s 2:37.27 1200.2% 228+222k 0+32275io 12pf+0w

after
1493.9u 404.6s 2:37.50 1205.4% 227+221k 0+32265io 48pf+0w
1485.0u 408.0s 2:38.54 1194.0% 227+222k 0+32272io 36pf+0w
1484.3u 407.0s 2:35.88 1213.3% 228+224k 0+32268io 14pf+0w

>>> stats.ttest_ind([160.65,158.49,157.27], [157.5,158.54,155.88])
Ttest_indResult(statistic=1.1923622711296888, pvalue=0.2990182944606766)
>>>

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove ryo@'s mail addresses.

Use "non-posted" instead of "strongly ordered" to describe nGnRnE mappings

Rename the following defines:
- _ARM_BUS_SPACE_MAP_STRONGLY_ORDERED to BUS_SPACE_MAP_NONPOSTED
- PMAP_DEV_SO to PMAP_DEV_NP
- LX_BLKPAG_ATTR_DEVICE_MEM_SO to LX_BLKPAG_ATTR_DEVICE_MEM_NP
Rename the following option:
- AARCH64_DEVICE_MEM_STRONGLY_ORDERED to AARCH64_DEVICE_MEM_NONPOSTED

Fix the problem "pcictl pci0 list" causes "panic: trap_el1h_error" on rockpro64.

The panic occures in bus_space_barrier() in rk3399_pcie.c:rkpcie_conf_read().
We expected bus_space_peek_4() to trap and recover in the path
trap_el1h_sync() -> data_abort_handler(), but In fact, the read is delayed
until bus_space_barrier(), and we get an SError interrupt (trap_el1h_error)
instead of a Synchronous Exception (trap_el1h_sync).

To catch this correctly, An implicit barrier in bus_space_peek have been added,
and trap the SError interrupt to recover from.

branches: 1.15.2;
Add a note about completion vs ordering barrier as well.

Add a big comment in generic_bs_barrier about mappings and what barriers
are really required and why we cheat. Inspired by a similar comment in
x86/bus_space.c

Use full system DSB ops for bs barrier.

The bus_space(9) man page is not clear whether barriers should enforce
ordering or completion. To be safe, use dsb here instead of dmb.

branches: 1.11.2;
Reduce scope of memory barriers use in bus_space_barrier() implementation.

Instead of always "dsb sy", use "dsb ishld" for reads, "dsb ishst" for
writes, and "dsh ish" for reads and writes.

Ok skrll@

Adjust aarch64 bus_space tags to also work on aarch64eb

Do not use Early Write Acknowledge for PCIe I/O and config space.

branches: 1.8.4;
Merge the [pgoyette-compat] branch

On second thought, get rid of "bs_base" from struct bus_space and use a
custom bs_map for acpipchb instead.

Add a "bs_base" field to struct bus_space. If present, use it to translate
mappings by appending the value to the pa passed to bus_space_map.

branches: 1.5.2;
initialize bs_cookie for generic_dsb tags

Provide bs_mmap implementations for bcm283x based boards.

PR: port-arm/53283
Submitted by: Nick Hudson

Fix encoding of MMAP flags for generic_bs_mmap

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove ryo@'s mail addresses.

fixed a bug that bus_space_read_region_{2,4,8}_swap() accesses wrong address.

branches: 1.5.18;
Fix typo in comment

branches: 1.4.2;
fix *_bs_rm_4_swap(). it was only reading 2 bytes, not 4 bytes.

pointed out by skrll@ thanks.

fix bugs in *_bs_rm_8_swap(). it was only reading 4 bytes, not 8 bytes.

Fix mis-incrementing pointer size in bus_space_read_region_{4,8}

pointed out by jmcneill@. thanks.

branches: 1.1.2; 1.1.4; 1.1.8; 1.1.10;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

Remove ryo@'s mail addresses.

branches: 1.1.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

aarch64: Remove empty source file and references to it.

branches: 1.1.4; 1.1.46;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

include "opt_gprof.h" so that _PROF_PROLOGUE works properly in ENTRY() macro in *.S files

branches: 1.15.2;
Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

Add support for Privileged Access Never (ARMv8.1-PAN).

PAN provides the same functionality as SMAP on x86: it forbids kernel
access to userland pages when PSTATE.PAN=1, and allows such accesses when
PSTATE.PAN=0.

We clear SCTLR_SPAN, to guarantee that PAN=1 each time the kernel is
entered. We catch PAN faults and panic right away without further
processing. In copyin, copyout, etc, we temporarily authorize access to
userland pages.

PAN is a very useful exploit mitigation. Reviewed by ryo@, thanks. Tested
on Qemu. Enabled by default.

Make copystr() a MI C function, part of libkern and shared on all
architectures.

Notes:

- On alpha and ia64 the function is kept but gets renamed locally to avoid
symbol collision. This is because on these two arches, I am not sure
whether the ASM callers do not rely on fixed registers, so I prefer to
keep the ASM body for now.
- On Vax, only the symbol is removed, because the body is used from other
functions.
- On RISC-V, this change fixes a bug: copystr() was just a wrapper around
strlcpy(), but strlcpy() makes the operation less safe (strlen on the
source beyond its size).
- The kASan, kCSan and kMSan wrappers are removed, because now that
copystr() is in C, the compiler transformations are applied to it,
without the need for manual wrappers.

Could test on amd64 only, but should be fine.

change copystr() to asm so that we don't have to add __noasan.
Also copyinout.S is the right place for copystr().

changed kcopy() to asm to avoid replacement memcpy() to kasan_memcpy() when defined KASAN.

fix copy{in,out}str to return ENAMETOOLONG if the string is longer than len bytes.

copy(9) had returned -1 if a bad address is encountered. fix to return EFAULT in that case.

centralize fp,lr definitions

fix build with aarch64 gcc/gas

branches: 1.3.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.2.2;
retire copyinout.S and fusu.S

branches: 1.1.4; 1.1.6;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

use lwp_trapframe() macro. NFC.

Move all non-emulation-specific coredump code into the coredump module,
and remove all #ifdef COREDUMP conditional compilation. Now, the
coredump module is completely separated from the emulation modules, and
they can all be independently loaded and unloaded.

Welcome to 9.99.18 !

branches: 1.4.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.3.2;
KNF

Use correct MID_ value

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

aarch64: Identify Cortex-A520, Cortex-A720 cores

arm64: Identify Qualcomm Oryon CPUs

CPU ID strings for Arm Cortex-A710, Neoverse V1, Neoverse N2, and Fujitsu A64FX

refine previous

Add Ampere 1 and 1A CPU IDs

aarch64: Count RNDRRS failure events and add dtrace probe.

PR port-arm/58572: aarch64 RNDRRS failures should be evcounted and
dtraced

aarch64: print NUMA ID

branches: 1.76.2;
kern/58195: arm: Support drvctl -d and -r for cpufeaturebus

This is required for detaching and re-attaching the vmt(4) driver on aarch64.

port-arm/58194: Resurrect vmt(4) from bitrot

On this architecture vmt(4) used to search for a node "/hypervisor" in the
FDT and probed the VMware hypervisor call only when the node was
found. However, things appear to have changed and VMware no longer provides
the FDT node.

Since vmt(4) doesn't actually need to read anything from FDT, and the
hypervisor call logically resides in virtual CPUs themselves, it would be
better to attach it directly to cpu, just like how it's probed on x86.

Remove ryo@'s mail addresses.

Remove useless/harmful casts in debug messages. MPIDR AFF3 would not
be printed before.

PMCR_EL0.LC should be set. ARM deprecates use of PMCR_EL0.LC=0

Explicitly disable overflow interrupts before enabling the cycle counter.

branches: 1.70.4;
fix build without options DDB

arm: Use device_set_private for cpuN.

For cpu at fdt, nix the fdt softc -- this was leaked and never used
for anything. The device's private storage is the cpu_info.

Print a big warning about trying to run on early ThunderX parts

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

G/C MD_CPU_HATCH. It's old evbarm (<= armv7)

style. NFCI.

Remove some newlines

Need to call pmap_tlb_info_attach for each CPU. Missed in previous
commit.
CVS ----------------------------------------------------------------------

Add a KASSERT

Identify Apple M1 "Icestorm" and "Firestorm" CPU types.

Do not try to initialize PMU if ID_AA64DFR0_EL1 reports a non-standard
PMU implementation.

branches: 1.59.4;
Add support hardware breakpoint and watchpoint again.

Limited support for hardware watchpoint has been available for some time, but it
has not been working properly. In addition, it stopped working at the time of
the PTRACE support commit on 2018-12-13. This has been fixed to work correctly,
and also fixed to be practical by sharing hardware watchpoints and breakpoints
between CPUs on MULTIPROCESSOR.

Also fixed a bug that causes a malfunction when switching CPUs with
"machine cpu N" when entering ddb mode from other than cpu_Debugger().

I have confirmed that the CPU can be switched by "machine cpu N" and return from
ddb properly in each case where ddb is called triggered by ddb break/watchpoint,
hardware break/watchpoint, and cpu_Debugger().

Improve a comment

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

branches: 1.56.2;
Fix detection of FP and SIMD features on Armv8.2+.

Only touch PMC registers if Performance Monitor Extensions are present.

Implement ChaCha with NEON on ARM.

XXX Needs performance measurement.
XXX Needs adaptation to arm32 neon which has half the registers.

Split aes_impl declarations out into aes_impl.h.

This will make it less painful to add more operations to struct
aes_impl without having to recompile everything that just uses the
block cipher directly or similar.

- On some systems with a different cache line size (and DIC,IDC) per CPU, trap "mrs Xt,ctr_el0" instruction
to return the minimum cache line size of the system to userland.
- add CLIDR_EL1 and CTR_EL0 to struct aarch64_sysctl_cpu_id.

On most systems, cache line size is the same for all CPUs, so this mechanism won't be required.
Rather, this is primarily for errata support, which will be committed later.

Switch the Icache sync operation to the necessary and sufficient one according to the CTR_EL0.DIC and CTR_EL0.IDC flags.

If CTR_EL0.DIC=1, Icache invalidation is not required.
If CTR_EL0.IDC=1, Dcache clean before Icache invalidation is not required.
CLIDR_EL1.LoC is 0, or CLIDR_EL1.LoUIS and CLIDR_EL1.LoUU are 0, Dcache clean is not required as well.

SEE ALSO ARMARM, "CTR_EL0 Cache Type Register", and "CLIDR_EL1 Cache Level ID Register"

New permutation-based AES implementation using ARM NEON.

Also derived from Mike Hamburg's public-domain vpaes code.

Implement AES in kernel using ARMv8.0-AES on aarch64.

Draft fpu_kern_enter/leave on aarch64.

Add some more id_aa64pfr0_el1 bits.

sctlr_el1 and ctr_el0 are 64-bit registers

Add support for the ARMv8.5-RNG CPU random number generator.

We use the RNDRRS system register. I made the following two
wild-arse guesses about the architecture of real implementations,
which might not exist yet:

1. There's only one physical source per CPU package, so not worth
attaching one per core.

2. Like other CPU RNGs -- RDSEED, VIA C3 -- this probably gives about
half a bit of entropy per bit of data (although perhaps we should
say zero and revisit this once it arrives on real silicon).

Tested in qemu as well as I can, using `-cpu max' (which doesn't get
to userland for unrelated reasons).

This uses the numeric notation `mrs %0, s3_3_c2_c4_1' for the rndrrs
system register instead of the more legible `mrs %0, rndrrs' as
suggested in the ARMv8.5 ARM. Why?

- clang doesn't like `mrs %0, rndrrs' for reasons unclear to me.

- gas only likes it with `.arch armv8.5-a+rng', but there's no clear
way to keep that scoped; the `.set push/pop' stack that would be an
obvious choice for this works only on mips.

- gcc supports __attribute__((target("arch=..."))) on functions, but
the version we use doesn't yet know about armv8.5-a+rng.

Later on, we should replace this by a target attribute and the more
obvious `mrs %0, rndrrs' notation.

ok nick

Print RNDR support in verbose CPU feature identification.

Cleanup CPU attach output:
- Always print the core's vendor and product name.
- Print the CPU ID on the same line as the name. Single line of dmesg
per core.
- Use aprint_verbose for reporting additional details.

Enable the cycle counter when a CPU hatches and store an estimate of the
frequency in ci_data.cpu_cc_freq.

Various updates and improvements to cpu start up on arm/aarch64

- start sharing more code around the AP startup messaging.
- call arm_cpu_topology_set early so that ci_core_id is available for
drivers, e.g. bcm2835_intr.c
- both arm and aarch64 now have
- a static cpu_info_store array
- the same arm_cpu_{hatched,mbox}

#if 0 / #endif -> a comment

Fetch ID_AA64MMFR2_EL1. Okayed by Nick the other day.

NVIDIA's breakaway marketing dept have been in touch.

Identify the Denver2 CPU in the Nvidia TX2

Trailing whitespace

KNF

port the arm64 cpu topology setup for big.little to arm.

rename arm64 cpu_do_topology() to arm_cpu_do_topology() and
call it from both arm cpu_attach().

replace both aarch64_set_topology() inline code in arm
cpu_attach() with new arm_cpu_do_topology(), which is called
by the arm64 locore as well (possibly not needed, which would
allow it to become static.)

not yet tested on a real big.little armv7 system. tested
on rockpro64 and pinebook pro.

provide some semblance of valid cpu topology for big.little systems.

while attaching cpus, if the FDT provides "capacity-dmips-mhz" track
the fastest set, and call cpu_topology_set() with slow=true for any
cpus that are not the fastest.

bug fix for cpu_topology_set(): actually set ci_is_slow for slow cpus.

with this change, and -current's recent scheduler changes, this means
that long running processes run on the faster cores. on RK3399 based
systems, i am seeing 20-50% speed ups for many tasks.


XXX: all this can be made common with armv7 big.little.

When attaching the first fdtbus, use the root "comptabile" (or failing that:
"model") property to set the cpu model (in userland aka sysctl hw.model).
When attaching the first cpu, do not overwrite a cpu model if it already
had been set.

branches: 1.31.2;
Identify Arm Neoverse E1 and N1 CPUs.

Fix build.

Add a missing newline

Fix build break (ci->ci_dev is not available on every port).

Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

Make cache operations available early.

Use separate cacheline aligned arrays for mbox and hatched as before.

Invalidate dcache before polling AP hatched status

Increase aarch64 MAXCPUS to 256.

Remove the A72 errata #859971 detection, it causes an illegal instruction on AWS A1 (virtualized)

report A72 errata #859971 workaround status during boot

branches: 1.20.2;
Need CPU_PARTMASK for eMAG CPU ID

Add Ampere eMAG 8180 cpuid

add several cortex CPU implementations found in their TRMs:
- A32 R1 (aarch32 only, not supported)
- A35 R1
- A65 R0
- A76AE R1
- A77

add the aarch64 ones to cpu.c for identification.

add cortex A-76 detection.

Use ci_{package,core,smt}_id instead of ci_data.cpu_{package,core,smt}_id

NFC

- add workaround for Cavium ThunderX errata 27456.
- add cpufuncs table in cpu_info. each cpu clusters may have different erratum. (e.g. big.LITTLE)

support boot option "-1" to disable multiprocessor boot, and "-z" to set AB_SILENT flag.

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

Use __nothing

remove XXX delay to attach cpus in order

Another space that hurts Jared's eyes.

Fix some product names and details as suggested by jmcneill

Identify some Cavium ThunderX CPUs

cleanup aarch64 mpstart and fdt bootstrap
* arm_cpu_hatch_arg is a bad idea. avoid serializing CPU startup, and eliminate arm_cpu_hatch_arg.
in mpstart, resolve own cpu index using array of cpu_mpidr[] (aarch64)
* add support fdt enable-method "spin-table"
* add support fdt enable-method "brcm,bcm2836-smp" (for 32bit RaspberryPi)
* use arm_fdt_cpu_bootstrap() instead of psci_fdt_bootstrap()
* rename "arm/fdt/psci_fdt.h" to "arm/fdt/psci_fdtvar.h" because of conflict of include file for needs-flag
* add devmap for cpu spin-table of raspberrypi3/aarch64
* no need to force hatch APs for raspberrypi3/arm32 ifndef MULTIPROCESSOR.
* fix to work pmap_extract(kerneltext/data/bss) even if before calling pmap_bootstrap

idea to use cpu_mpidr[] by jmcneill@. reviewd by skrll@. thanks.

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

Use __SHIFTOUT to extract MPIDR affinity levels

Define and use VPRINTF

add default statements, use PRI?64 instead of ll?

add MULTIPROCESSOR support

branches: 1.1.2; 1.1.4;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

Delete aarch64 cpu_in_cksum.S draft.

This isn't actually used in the kernel; it is only used to cause the
in_cksum tests to fail.

If you want to revive it and make it work, you can pull it out of the
attic.

branches: 1.1.4; 1.1.30; 1.1.36;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

aarch64: Allow for alternate cpu_idle() implementations

branches: 1.15.2;
kern/58149: aarch64: Cannot return from a signal handler if SP was misaligned when the signal arrived

Apply the kernel diff from the PR

1. sendsig_siginfo() previously assumed that user SP was always aligned to
16 bytes and could call signal handlers with SP misaligned. This is a
wrong assumption because aarch64 demands that SP is aligned *only while*
it's being used to access memory. Now it properly aligns it before
pusing anything on the stack.

2. cpu_mcontext_validate() used to check if _REG_SP was aligned and
considered the ucontext invalid otherwise. This meant if a signal was
sent to a process whose SP was misaligned, the signal handler would fail
to return because the ucontext passed from the kernel was an invalid
one. Now setcontext(2) doesn't complain about misaligned SP.

aarch64: curcpu() audit.

Sprinkle KASSERT (or KDASSERT in hot paths) for kpreempt_disabled()
when we use curcpu() and it's not immediately obvious that the caller
has preemption disabled but closer scrutiny suggests the caller has.

Note unsafe curcpu()s for syscall event counting. Not sure this is
worth changing.

Possible bugs fixed:

- cpu_irq and cpu_fiq could be preempted while trying to run softints
on this CPU.

- data_abort_handler might incorrectly think it was invoked in
interrupt context when it was only preempted and migrated to
another CPU.

- pmap_fault_fixup might report the wrong CPU logs.

(However, we don't currently run with kpreemption on aarch64, so
these are not yet real bugs fixed except if you patch it to build
with __HAVE_PREEMPTION.)

branches: 1.13.4;
aarch64: Refactor splhigh and restore in dosoftints.

No functional change intended. splhigh always returns ci->ci_cpl,
which should not be changing at this point. Makes the bracketing by
splhigh/splx clearer.

use lwp_trapframe() macro. NFC.

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

Fix a problem that setcontext(2) sometimes fail on ARMv8.5-BTI cpu.

fixed to always get 0 for SPSR.BTYPE with getcontext(2).
a non-zero SPSR.BTYPE cannot be set with setcontext(2).

aarch64: handle _UC_SETSTACK and _UC_CLRSTACK like on arm32

ok ryo@

cpu_need_resched():

- Remove all code that should be MI, leaving the bare minimum under arch/.
- Make the required actions very explicit.
- Pass in LWP pointer for convenience.
- When a trap is required on another CPU, have the IPI set it locally.
- Expunge cpu_did_resched().

mi_userret(): take care of calling preempt(), set spc_curpriority directly,
and remove MD code that does the same.

branches: 1.6.4;
don't set lwp->l_private if no _UC_TLSBASE flag.
atf lib/libc/sys/t_swapcontext Passed.

add missing cast

add MULTIPROCESSOR support

branches: 1.3.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.2.16;
__HAVE_PREEEMPTION -> __HAVE_PREEMPTION

branches: 1.1.4; 1.1.6;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove ryo@'s mail addresses.

KASSERT(kpreempt_disabled()) before accessing curcpu() to reflect why
preemption needs to be disabled more clearly.

aarch64: curcpu() audit.

Sprinkle KASSERT (or KDASSERT in hot paths) for kpreempt_disabled()
when we use curcpu() and it's not immediately obvious that the caller
has preemption disabled but closer scrutiny suggests the caller has.

Note unsafe curcpu()s for syscall event counting. Not sure this is
worth changing.

Possible bugs fixed:

- cpu_irq and cpu_fiq could be preempted while trying to run softints
on this CPU.

- data_abort_handler might incorrectly think it was invoked in
interrupt context when it was only preempted and migrated to
another CPU.

- pmap_fault_fixup might report the wrong CPU logs.

(However, we don't currently run with kpreemption on aarch64, so
these are not yet real bugs fixed except if you patch it to build
with __HAVE_PREEMPTION.)

add support Hardware updates to Access flag and Dirty state (FEAT_HAFDBS)

- The DBM bit of the PTE is now used to determine if it is writable, and
the AF bit is treated entirely as a reference bit. A valid PTE is always
treated as readable. There can be no valid PTE that is not readable.
- LX_BLKPAG_OS_{READ,WRITE} are used only for debugging purposes,
and has been superseded by LX_BLKPAG_AF and LX_BLKPAG_DBM.
- Improve comment

The need for reference/modify emulation has been eliminated,
and access/permission faults have been reduced, however,
there has been little change in overall performance.

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

Sprinkle some const

Shrink a panic message to avoid a long line

Remove unnecessary brackets in a conditional

Print the cache information in similar formats and arm and aarch64, e.g.

arm before
[ 1.0000000] cpu0: 32KB/64B 2-way L1 PIPT Instruction cache
[ 1.0000000] cpu0: 32KB/64B 2-way write-back-locking-C L1 PIPT Data cache
[ 1.0000000] cpu0: 2304KB/64B 16-way write-through L2 PIPT Unified cache

arm after
[ 1.0000000] cpu0: L1 32KB/64B 2-way (256 set) PIPT Instruction cache
[ 1.0000000] cpu0: L1 32KB/64B 2-way (256 set) write-back-locking-C PIPT Data cache
[ 1.0000000] cpu0: L2 2304KB/64B 16-way (2304 set) write-through PIPT Unified cache

aarch64 before
[ 1.0000030] cpu1: L1 48KB/64B*256L*3W PIPT Instruction cache
[ 1.0000030] cpu1: L1 32KB/64B*256L*2W PIPT Data cache
[ 1.0000030] cpu1: L2 2048KB/64B*2048L*16W PIPT Unified cache

aarch64 after
[ 1.0000030] cpu1: L1 48KB/64B 3-way (256 set) PIPT Instruction cache
[ 1.0000030] cpu1: L1 32KB/64B 2-way (256 set) PIPT Data cache
[ 1.0000030] cpu1: L2 2048KB/64B 16-way (2048 set) PIPT Unified cache

Small simplification

branches: 1.26.2;
Use the dmb/dsb/isb macros... if nothing else they're all now consistent
about the "memory" assembler contraint.

No binary change

Simplify the cpufunc.h header, i.e. always use #include <arm/cpufunc.h>

Add support for Privileged Access Never (ARMv8.1-PAN).

PAN provides the same functionality as SMAP on x86: it forbids kernel
access to userland pages when PSTATE.PAN=1, and allows such accesses when
PSTATE.PAN=0.

We clear SCTLR_SPAN, to guarantee that PAN=1 each time the kernel is
entered. We catch PAN faults and panic right away without further
processing. In copyin, copyout, etc, we temporarily authorize access to
userland pages.

PAN is a very useful exploit mitigation. Reviewed by ryo@, thanks. Tested
on Qemu. Enabled by default.

Fix previous; add missing <uvm/uvm.h> include.

Fix uvmexp.ncolors for some big.LITTLE configuration; it is uncertain
which CPU is used as primary, and as a result, secondary CPUs can
require larger number of colors.

In order to solve this problem, update uvmexp.ncolors via
uvm_page_recolor(9) when secondary CPUs are attached, as done for
other ports like x86.

Pointed out by jmcneill@, and discussed on port-arm@:
http://mail-index.netbsd.org/port-arm/2020/07/03/msg006837.html

Tested and OK'd by ryo@.

Switch the Icache sync operation to the necessary and sufficient one according to the CTR_EL0.DIC and CTR_EL0.IDC flags.

If CTR_EL0.DIC=1, Icache invalidation is not required.
If CTR_EL0.IDC=1, Dcache clean before Icache invalidation is not required.
CLIDR_EL1.LoC is 0, or CLIDR_EL1.LoUIS and CLIDR_EL1.LoUU are 0, Dcache clean is not required as well.

SEE ALSO ARMARM, "CTR_EL0 Cache Type Register", and "CLIDR_EL1 Cache Level ID Register"

cache information can be detected correctly on newer CPUs

- add VPIPT cache type
- adapt to 64-bit CCSIDR (ARMv8.3-CCIDX)
- CCSIDR:[WT,WB,PA,WA] are deprecated
- show number of cache lines when attaching cpu

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.16.2;
Cleanup CPU attach output:
- Always print the core's vendor and product name.
- Print the CPU ID on the same line as the name. Single line of dmesg
per core.
- Use aprint_verbose for reporting additional details.

port the arm64 cpu topology setup for big.little to arm.

rename arm64 cpu_do_topology() to arm_cpu_do_topology() and
call it from both arm cpu_attach().

replace both aarch64_set_topology() inline code in arm
cpu_attach() with new arm_cpu_do_topology(), which is called
by the arm64 locore as well (possibly not needed, which would
allow it to become static.)

not yet tested on a real big.little armv7 system. tested
on rockpro64 and pinebook pro.

provide some semblance of valid cpu topology for big.little systems.

while attaching cpus, if the FDT provides "capacity-dmips-mhz" track
the fastest set, and call cpu_topology_set() with slow=true for any
cpus that are not the fastest.

bug fix for cpu_topology_set(): actually set ci_is_slow for slow cpus.

with this change, and -current's recent scheduler changes, this means
that long running processes run on the faster cores. on RK3399 based
systems, i am seeing 20-50% speed ups for many tasks.


XXX: all this can be made common with armv7 big.little.

- Many small tweaks to the SMT awareness in the scheduler. It does a much
better job now at keeping all physical CPUs busy, while using the extra
threads to help out. In particular, during preempt() if we're using SMT,
try to find a better CPU to run on and teleport curlwp there.

- Change the CPU topology stuff so it can work on asymmetric systems. This
mainly entails rearranging one of the CPU lists so it makes sense in all
configurations.

- Add a parameter to cpu_topology_set() to note that a CPU is "slow", for
where there are fast CPUs and slow CPUs, like with the Rockwell RK3399.
Extend the SMT awareness to try and handle that situation too (keep fast
CPUs busy, use slow CPUs as helpers).

branches: 1.12.2;
Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

fix build error by my previous commit

MAX_CACHE_LEVEL * struct aarch64_cache_info are required to statically allocate for cpu0.

avoid "cpu0: L2 512KB/64B 16-way write-back read-allocate write-allocate PIPT *UNK* cache" by r1.8

Take the basic CPU topology information we already collect, and use it
to make circular lists of CPU siblings in the same core, and in the
same package. Nothing fancy, just enough to have a bit of fun in the
scheduler trying out different tactics.

Make cache operations available early.

in many device attach paths, allocate memory with KM_SLEEP instead of KM_NOSLEEP
and remove code to handle failures that can no longer happen.

Do not attempt to change coherency_unit at runtime. Instead, if the
required coherency unit is greater than COHERENCY_UNIT in a MULTIPROCESSOR
kernel, just panic instead.

This makes non-MULTIPROCESSOR kernels work again.

branches: 1.5.4;
- add workaround for Cavium ThunderX errata 27456.
- add cpufuncs table in cpu_info. each cpu clusters may have different erratum. (e.g. big.LITTLE)

Update coherency_unit if needed.

Pointed out by skrll@

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

add default statement

branches: 1.1.2; 1.1.4;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

include "opt_gprof.h" so that _PROF_PROLOGUE works properly in ENTRY() macro in *.S files

branches: 1.7.2;
fix build error with LLVM.

Switch the Icache sync operation to the necessary and sufficient one according to the CTR_EL0.DIC and CTR_EL0.IDC flags.

If CTR_EL0.DIC=1, Icache invalidation is not required.
If CTR_EL0.IDC=1, Dcache clean before Icache invalidation is not required.
CLIDR_EL1.LoC is 0, or CLIDR_EL1.LoUIS and CLIDR_EL1.LoUU are 0, Dcache clean is not required as well.

SEE ALSO ARMARM, "CTR_EL0 Cache Type Register", and "CLIDR_EL1 Cache Level ID Register"

even if the line size of Icache and Dcache is different, it was fixed to work correctly.

- MAX(IcacheShift,DcacheShift) is wrong and should be MIN(IcacheShift,DcacheShift).
Dcache and Icache are now done in independent loops instead of in the same loop.
- simplify the handling of cache_handle_range() macro arguments.
- cache_handle_range macro doesn't include "ret" anymore.

even if "no options MULTIPROCESSOR" requires isb after tlbi op. since it should be harmless, dsb is also added.
fixed a problem that rockpro64 doesn't boot without MULTIPROCESSOR.

branches: 1.3.4;
- add workaround for Cavium ThunderX errata 27456.
- add cpufuncs table in cpu_info. each cpu clusters may have different erratum. (e.g. big.LITTLE)

* fix icache invalidations.
* "ic ivau" (aarch64_icache_sync_range) with VA generates permission fault in some situations, therefore use KSEG address for now.

branches: 1.1.2; 1.1.4;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

aarch64: Optimization: Omit needless membar when triggering softint.

When we are triggering a softint, it can't already hold any mutexes.
So any path to mutex_exit(mtx) must go via mutex_enter(mtx), which is
always done with atomic r/m/w, and we need not issue any explicit
barrier between ci->ci_curlwp = softlwp and a potential load of
mtx->mtx_owner in mutex_exit.

PR kern/57240

XXX pullup-9
XXX pullup-10

aarch64: Add missing barriers in cpu_switchto.

Details in comments.

Note: This is a conservative change that inserts a barrier where
there was a comment saying none is needed, which is probably correct.
The goal of this change is to systematically add barriers to be
confident in correctness; subsequent changes may remove some bariers,
as an optimization, with an explanation of why each barrier is not
needed.

PR kern/57240

XXX pullup-9
XXX pullup-10

branches: 1.39.4;
Move cpu_Debugger() into a more suitable file, from cpuswitch.S to db_interface.c.

On aarch64, ddb backtrace can be performed without framepointer by specifying
the /s modifier to the ddb trace command (trace/s, bt/s).
The default is trace with framepointer (same as before).

This allows backtracing even on kernels compiled with -fomit-frame-pointer.

use stp if possible.

optimize. reduce 2 instructions.

make a frame pointer to show a backtrace correctly.

Sprinkle isb after modifying system regs of pointer auth.
With options ARMV83_PAC, it now works on native Mac M1.

TODO: Multiple ISBs should be combined in one place.

Add support hardware breakpoint and watchpoint again.

Limited support for hardware watchpoint has been available for some time, but it
has not been working properly. In addition, it stopped working at the time of
the PTRACE support commit on 2018-12-13. This has been fixed to work correctly,
and also fixed to be practical by sharing hardware watchpoints and breakpoints
between CPUs on MULTIPROCESSOR.

Also fixed a bug that causes a malfunction when switching CPUs with
"machine cpu N" when entering ddb mode from other than cpu_Debugger().

I have confirmed that the CPU can be switched by "machine cpu N" and return from
ddb properly in each case where ddb is called triggered by ddb break/watchpoint,
hardware break/watchpoint, and cpu_Debugger().

Always issue isb after cpacr_el1 writes since it is a context-changing
operation.

branches: 1.31.2;
Use the correct (more relaxed) membar_exit barrier in cpu_switchto_softint

Use corrcet membar_exit barrier

move #include "opt_compat_netbsd32.h" to where it's required

Move el[01]_trap_exit into vectors.S where the callers exist

Use 'lr' instead of 'x30' in an instruction for clarity

Fix a comment

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

Reduce the window of having interrupts disabled in cpu_switchto{,_softint}
and ensure astpending is checked with interrupts disabled.

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

fix to do backtrace properly for running LWPs and cpu_lwp_fork().
when dump of pcb_tf, only the switchframe part is now displayed instead of the whole trapframe.

use ldp if possible

Meant to do a store here, not a load. Ie we want to replace the initial
weak key by the stronger one we just generated.

Rototilled this place too many times.

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.15.4;
oldlwp is always non-NULL in cpu_switchto so remove the test for NULL.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.13.2;
Add a speculation barrier after the 'eret'.

Some aarch64 cpus speculatively execute instructions after 'eret',
and this potentiates side-channel attack.

from
https://github.com/torvalds/linux/commit/679db70801da9fda91d26caf13bf5b5ccc74e8e8

Trailing whitespace

branches: 1.11.4;
make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

- need to save/restore interrupt mask when entering/exiting to/from cpu_switchto_softint().
- when call dosoftints from cpu_idle, interrupts should be disabled.

rarely, lwp stack had been exhausted when high interrupts.
reported by alnsn@. thanks.

need to save/restore also x1. x1 is in-use as ipl.

modifying curlwp->l_md_ktf, curlwp->l_md_cpacr, and curlwp should be protected by a critical section.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

centralize fp,lr definitions

fix build with aarch64 gcc/gas

remove unused code

branches: 1.1.2; 1.1.4;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

Remove ryo@'s mail addresses.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

branches: 1.10.2;
fix build error of /usr/sbin/crash

pointed out by rjs@, thanks.

Determination of A64,A32,T32 for disasm is now done in strrdisasm() instead of the caller.
correctly disassemble by processor state if defined DEBUG_DUMP_ON_USERFAULT or DEBUG_DDB_ON_USERFAULT.

don't read memory directly.
In particular, userland memory may be unmapped at the time of reading.

Format string annotation for strdisasm_printf

Merge the [pgoyette-compat] branch

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

make kernel-groveling crash(8) work on aarch64

use panic() instead of some printf to show fault status.
useful for ddb "show panic" command.

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove ryo@'s mail addresses.

Relax the TLB invalidation from full to by va for writing to kernel text
in db_write_text.

Restore a '\n' I accidentally removed in 1.16

KNF

Use "non-posted" instead of "strongly ordered" to describe nGnRnE mappings

Rename the following defines:
- _ARM_BUS_SPACE_MAP_STRONGLY_ORDERED to BUS_SPACE_MAP_NONPOSTED
- PMAP_DEV_SO to PMAP_DEV_NP
- LX_BLKPAG_ATTR_DEVICE_MEM_SO to LX_BLKPAG_ATTR_DEVICE_MEM_NP
Rename the following option:
- AARCH64_DEVICE_MEM_STRONGLY_ORDERED to AARCH64_DEVICE_MEM_NONPOSTED

Move cpu_Debugger() into a more suitable file, from cpuswitch.S to db_interface.c.

Use the PAR register to check for accessibility in db_(read|write)_bytes().

db_(read|write)_bytes() uses the TTBR[01] at that time, so it must check
if it is accessible in context at that time, not pmap_extract()
which uses the struct pmap of the process.

- It also checks if the address is writable.
- db_write_bytes() also requires ARMV81_PAN control.

In ddb, fixed "trace/u" and user process memory read/write to work correctly.

In the softint context, curlwp points the kernel lwp, so to get the pmap
of a user process, we had to use curcpu()->ci_onproc->l_proc instead of
curproc (curlwp->l_proc). Adviced by ad@.

Make even more pmap agnostic

Reduce characters to print in db_pte_print and unwrap some short lines.

branches: 1.14.2;
Remove unnecssary brackets. Same binary before and after.

Make the ddb for pmap / pte information pmap agnostic

branches: 1.12.4;
Avoid duplicate definition of ddb_regs in crash(8).

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

branches: 1.10.2;
sprinkle LE32TOH to fetch instructions on aarch64eb

Improve a comment

Add support for Privileged Access Never (ARMv8.1-PAN).

PAN provides the same functionality as SMAP on x86: it forbids kernel
access to userland pages when PSTATE.PAN=1, and allows such accesses when
PSTATE.PAN=0.

We clear SCTLR_SPAN, to guarantee that PAN=1 each time the kernel is
entered. We catch PAN faults and panic right away without further
processing. In copyin, copyout, etc, we temporarily authorize access to
userland pages.

PAN is a very useful exploit mitigation. Reviewed by ryo@, thanks. Tested
on Qemu. Enabled by default.

Merge the [pgoyette-compat] branch

fixed to keep PROT_EXECUTE when writing the page/block.
this is required when the L2 block to which the target address belongs
and the L2 block to which this function itself belongs are the same.

set kernel text/rodata readonly by default.
add function db_write_text() for setting ddb breakpoint.

branches: 1.4.2;
PR/53338: David Binderman: Widen shift to the LHS type.

implement properly branch_taken() and inst_unconditional_flow_transfer().

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

ddb/db_active.h: New home for extern db_active.

This can be included unconditionally, and db_active can then be
queried unconditionally; if DDB is not in the kernel, then db_active
is a constant zero. Reduces need for #include opt_ddb.h, #ifdef DDB.

- Display "cpu[<CPUINDEX>]" instead of "cpu[<CPUID>]".
- Also add cpu_info->ci_onproc to display.

Only print the appropriate PAR fields for PAR.F={0,1}

Group the fields in the header.

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

When booted with "boot netbsd -1" (disable multiprocessor boot),
"ddb>continue" didn't work when ddb was started by breakpoint trap.

Make the ddb for pmap / pte information pmap agnostic

branches: 1.39.4;
Numeric modifiers conflict with the syntax interpretation of ddb, so use 'b', 'w', 'l', 'q' instead.
Also, change load/store('l','s') to 'r','w' like the other arch.

>db{0}> machine watch/1 hostname
>Bad modifier

>db{0}> machine watch/s1 hostname
>add watchpoint 0 as ffffc00001087848

- fixed a problem where hardware {break,watch}points other than #0 could not be cleared
- hardware {break,watch}point addresses are now strictly checked

Add support hardware breakpoint and watchpoint again.

Limited support for hardware watchpoint has been available for some time, but it
has not been working properly. In addition, it stopped working at the time of
the PTRACE support commit on 2018-12-13. This has been fixed to work correctly,
and also fixed to be practical by sharing hardware watchpoints and breakpoints
between CPUs on MULTIPROCESSOR.

Also fixed a bug that causes a malfunction when switching CPUs with
"machine cpu N" when entering ddb mode from other than cpu_Debugger().

I have confirmed that the CPU can be switched by "machine cpu N" and return from
ddb properly in each case where ddb is called triggered by ddb break/watchpoint,
hardware break/watchpoint, and cpu_Debugger().

"machine cpu" command shows pc of trapframe and the symbol

match the macro name to the order of the arguments. NFC.

introduce DDB_END_CMD and replace more than 20 copies of the same
list of NULLs and 0. idea from rillig@.

all touched ports built, several booted.

Avoid duplicate definition of ddb_regs in crash(8).

Fix build as crash(8); Protect db_md_meminfo_cmd() by defined(_KERNEL).

add a command to dump the bootconfig passed meminfo.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

Provide and use a sev() macro for the sev instruction.

While here use the correct barrier to ensure completion of memory accesses
before a couple of the sev() calls.

branches: 1.28.2;
Use the dmb/dsb/isb macros... if nothing else they're all now consistent
about the "memory" assembler contraint.

No binary change

Simplify the cpufunc.h header, i.e. always use #include <arm/cpufunc.h>

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

Add ddb "mach reset" command for Arm ports.

fix to do backtrace properly for running LWPs and cpu_lwp_fork().
when dump of pcb_tf, only the switchframe part is now displayed instead of the whole trapframe.

instead of reading memory directly, db_read_bytes() is used to avoid faults in ddb.

for "mach cpuinfo", print ci_biglock_count too.

add the case of kdb_trap(-1) called from pic_ipi_ddb().
it depended on the update timing of 'db_recover'.

branches: 1.20.4;
use macro

prevent switching to CPUs that are not responding to IPI_DDB.

add "machine cpuinfo/a" to show cpuinfo[] of all cpus

Align output from db_md_cpuinfo_cmd

- add reg_{s1e0r,s1e0w,s1e1r,s1e1w}_write() macro.
- show the result of AT insn at ddb "machine pte" command.

- add ddb command "machine ttbr" to dump MMU tables.
- tidy up descriptions, usages and messages.

Merge the [pgoyette-compat] branch

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

Comment out implementation specific registers to avoid illegal instruction trap on ThunderX

don't pass illegal cpu index to cpu_lookup(). it may cause KASSERT.

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

make kernel-groveling crash(8) work on aarch64

no need machine local ddb command pmaphist any more.

use DDB_EXPR_FMT. fix typo.

add missing casts; use PRI?64 instead of ll?

fix compile error

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

KNF. (sort #includes)

Remove ryo@'s mail addresses.

oops, my previous commit is bad. revert previous.
<frame-address> is a frame pointer, not a trapframe, and it worked correctly. (e.g., trace $x29)

Specifying the frame address "trace <frame-address>" was not working.

If there was a "bl <func>" instruction at the end of a function block,
the stack analysis backtrace (bt/s) would fail because $lr would point
to the beginning of the next function.

Fixed stack analyzing backtrace (bt/s) correctly for nested trapframes.

Functionalize frame pointer backtrace.

On aarch64, ddb backtrace can be performed without framepointer by specifying
the /s modifier to the ddb trace command (trace/s, bt/s).
The default is trace with framepointer (same as before).

This allows backtracing even on kernels compiled with -fomit-frame-pointer.

tidy up backtrace from crash(9) on aarch64

- fix to dump trapframe when backtracing from crash(8).
- use db_read_bytes() when reading kernel memory.

Display the trap type of trapframe when backtracing.

Simplified termination conditions for ddb backtrace.

Exit backtrace when the user trapframe is invalid. (Mainly in kernel threads).

aarch64: Fix stack traces from jump-to-null.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

branches: 1.12.2;
Fix build failure due to -Werror=stack-usage.

Use db_read_bytes() against particular member of structure in use,
by which we can avoid to have whole structure in the stack.

Now, stack usage shrinks: 4240 --> 1296

fix to do backtrace properly for running LWPs and cpu_lwp_fork().
when dump of pcb_tf, only the switchframe part is now displayed instead of the whole trapframe.

- move aarch64 addressspace macros from pmap.h to cpufunc.h
- rename ptr_strip_pac() to aarch64_strip_pac()

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.8.10;
Merge the [pgoyette-compat] branch

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

make kernel-groveling crash(8) work on aarch64

aarch64/db_trace.c: annotate w/ __printflike; fix discovered problems

don't depend on clang code to backtrace. keep trapframe as framepointer if DDB.

add missing casts

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove ryo@'s mail addresses.

adjust tab. NFC

fix wrong target register size of "ldrsh"

"ldrsh Xt, [Xn, Xm]" was being output as "ldrsh Wt, [Xn, Xm]"

make more system registers are disassemblable

The immediate offset of "ldtrb", "ldtrh", "sttrb", and "sttrh" was always output as unsigned.
Correctly, it is 9bit signed.

The register operand size for "smnegl" and "smsubl' was wrong.
not "smsubl Xd, Xn, Xm, Xa", but "smsubl Xd, Wn, Wm, Xa".

branches: 1.10.2;
AArch64 instructions are always LE: swap if we're BE

make more ARMv8.x system registers are disassemblable

disassemblable bti insns

disassemblable pointer authentication insns

disassemblable sha512 insns

make kernel-groveling crash(8) work on aarch64

add support for disasm pmull,aes,sha insns. most SIMD insns are not yet.

use panic() instead of some printf to show fault status.
useful for ddb "show panic" command.

branches: 1.2.2;
Widen shift to the LHS type.
same as aarch64/db_interface.c r1.4, PR/53338.

branches: 1.1.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

Remove ryo@'s mail addresses.

branches: 1.1.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

Factor out some fdt(4) features from {,evb}arm into dev/fdt.

Now, FDT-based support to efirt, initrd, rndseed, and efirng can be
used from, e.g., riscv.

Mostly from Nick Hudson.

XXX
As Nick comments, there can be some optimizations for fdt_map_range().
efiboot may also be modified to load these objects into aligned PAs.

Style. NFCI.

since pmap_activate_efirt() rewrites TTBR0, it is necessary to pmap_activate() again after pmap_deactivate_efirt() to restore the original TTBR0.

- Fix to do pmap_{de,}activate() before/after pmap_{,de}activate_efirt().
- moved kpreempt_{disable,enable}() to the caller since everything between
arm_efirt_md_enter() and arm_efirt_md_exit() should be kpreempt disabled.

ok skrll@

Update to support EFI runtime outside the kernel virtual address space
by creating an EFI RT pmap that can be activated / deactivated when
required.

Adds support for EFI RT to ARM_MMU_EXTENDED (ASID) 32-bit Arm machines.

On Arm64 the usage of pmapboot_enter is reduced and the mappings are
created much later in the boot process -- now in cpu_startup_hook.
Backward compatiblity for KVA mapped RT from old bootaa64.efi is
maintained.

Adding support to other platforms should be easier as a result.

Remove the unnecessary invalidation code in arm_efirt_md_map_range.

pmapboot_enter will panic if any overlapping mappings existed before and
a full TLB invalidate was done as part of turning the MMU on in locore.

Don't mark EFI runtime pages LX_BLKPAG_OS_READ | LX_BLKPAG_OS_WRITE as
these bits are only used by the current pmap fault code and these are
wired pages which will never fault.

branches: 1.8.2; 1.8.4;
Use the dmb/dsb/isb macros... if nothing else they're all now consistent
about the "memory" assembler contraint.

No binary change

Simplify the cpufunc.h header, i.e. always use #include <arm/cpufunc.h>

pmapboot_enter simplication
- bootpage_alloc in asm becomes pmapboot_pagealloc in C
- PMAPBOOT_ENTER_NOBLOCK is removed as it's not used
- PMAPBOOT_ENTER_NOOVERWRITE is removed as it's now always on
- physpage_allocator argument is removed as it's always
pmapboot_pagealloc
- Support for EARLYCONS without CONSADDR is removed so that the identity
map for CONSADDR is always known.

For the assembly files:
2 files changed, 40 insertions(+), 89 deletions(-)

LGTM ryo

Enable FP access for EFI RT and improve error handling.

Trailing whitespace

branches: 1.3.2; 1.3.6; 1.3.8;
EFI runtime code section needs to be writable, otherwise we fail with a permission fault at shutdown on QEMU when writing to the RTC

Setup mappings for EFI runtime mmio ranges.

Add support for EFI runtime services on aarch64.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

use lwp_trapframe() macro. NFC.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

branches: 1.8.2;
BE32 binaries are no longer supported for ARMv7 and later, and
therefore for aarch64eb.

Reject them with ENOEXEC, rather than causing illegal instruction
exceptions due to unexpected binary format.

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

part of PR port-arm/54702

Make sure that md_march32 and ep_machine_arch have same size.

XXX
pullup to netbsd-9

PR port-arm/54702

Add support for earmv6hf binaries on COMPAT_NETBSD32 for aarch64:

- Emulate ARMv6 instructions with cache operations register (c7), that
are deprecated since ARMv7, and disabled on ARMv8 with LP64 kernel.

- ep_machine_arch (default: earmv7hf) is copied from executables, as we
do for mips64. "uname -p" reports earmv6hf if compiled for earmv6hf;
configure scripts etc can determine the appropriate architecture.

Many thanks to ryo@ for helping me to add support of Thumb-mode,
as well as providing exhaustive test cases:

https://github.com/ryo/mcr_test/

We've confirmed:

- Emulation works in Thumb-mode.
- T32 16-bit length illegal instruction results in SIGILL, even if
it is located nearby a boundary b/w mapped and unmapped pages.
- T32 32-bit instruction results in SIGSEGV if it is located across
a boundary b/w mapped and unmapped pages.

XXX
pullup to netbsd-9

don't exec 32bit binary on the cpu that has no aarch32.

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove ryo@'s mail addresses.

aarch64: curcpu() audit.

Sprinkle KASSERT (or KDASSERT in hot paths) for kpreempt_disabled()
when we use curcpu() and it's not immediately obvious that the caller
has preemption disabled but closer scrutiny suggests the caller has.

Note unsafe curcpu()s for syscall event counting. Not sure this is
worth changing.

Possible bugs fixed:

- cpu_irq and cpu_fiq could be preempted while trying to run softints
on this CPU.

- data_abort_handler might incorrectly think it was invoked in
interrupt context when it was only preempted and migrated to
another CPU.

- pmap_fault_fixup might report the wrong CPU logs.

(However, we don't currently run with kpreemption on aarch64, so
these are not yet real bugs fixed except if you patch it to build
with __HAVE_PREEMPTION.)

fix various typos in comments.

whitespace

add support Hardware updates to Access flag and Dirty state (FEAT_HAFDBS)

- The DBM bit of the PTE is now used to determine if it is writable, and
the AF bit is treated entirely as a reference bit. A valid PTE is always
treated as readable. There can be no valid PTE that is not readable.
- LX_BLKPAG_OS_{READ,WRITE} are used only for debugging purposes,
and has been superseded by LX_BLKPAG_AF and LX_BLKPAG_DBM.
- Improve comment

The need for reference/modify emulation has been eliminated,
and access/permission faults have been reduced, however,
there has been little change in overall performance.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

branches: 1.20.2;
For cpu_jump_onfault() in data_abort_handler(), stop returning
hard-coded EFAULT and use return value from uvm_fault() instead.

There are some paths that do not call uvm_fault():

(1) For fatalabort case, use EFAULT as before.
(2) When va range is invalid, use EFAULT instead of EINVAL.

These change fixes bytes_transfer_eof_* tests in
sys/lib/libc/sys/t_ptrace_wait*.

Note that without (2) above, some tests like
sys/lib/libc/sys/t_wait:write_error become newly failing.

I've confirmed that there's no new regression in full ATF run.

OK ryo

Don't use %s in UVMHIST_PRINT. Remove an unnecessary #ifdef UVMHIST while
I'm here

No need to recover from fault from within a hardware interrupt handler.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

Add support for Privileged Access Never (ARMv8.1-PAN).

PAN provides the same functionality as SMAP on x86: it forbids kernel
access to userland pages when PSTATE.PAN=1, and allows such accesses when
PSTATE.PAN=0.

We clear SCTLR_SPAN, to guarantee that PAN=1 each time the kernel is
entered. We catch PAN faults and panic right away without further
processing. In copyin, copyout, etc, we temporarily authorize access to
userland pages.

PAN is a very useful exploit mitigation. Reviewed by ryo@, thanks. Tested
on Qemu. Enabled by default.

Determination of A64,A32,T32 for disasm is now done in strrdisasm() instead of the caller.
correctly disassemble by processor state if defined DEBUG_DUMP_ON_USERFAULT or DEBUG_DDB_ON_USERFAULT.

- move aarch64 addressspace macros from pmap.h to cpufunc.h
- rename ptr_strip_pac() to aarch64_strip_pac()

Fix pmap to work correctly with tagged addresses

- when fault, untag from address before passing to uvm/pmap functions
- pmap_extract() checks more strictly and consider the address tag

fix behaviour mmap()/mprotect() when passed only PROT_EXEC.

when mmap()/mprotect() with only PROT_EXEC, syscall will be successful,
but the page actually hadn't been mapped.
it should be mapped with PROT_READ|PROT_EXEC implicitly. (r-x)

branches: 1.10.2; 1.10.4;
since uvm_faut() will fail if cache maintain instructions (e.g., "dc cvau".
that has ESR.WnR=1 = write access) for a read only page causes a data abort trap,
It shoulde be treat the access type as read access in a data abort with
cache operation (ESR_ISS_DATAABORT_CM=1)

pointed out and tested by mrg@. thanks

Overhaul the API used to fetch and store individual memory cells in
userspace. The old fetch(9) and store(9) APIs (fubyte(), fuword(),
subyte(), suword(), etc.) are retired and replaced with new ufetch(9)
and ustore(9) APIs that can return proper error codes, etc. and are
implemented consistently across all platforms. The interrupt-safe
variants are no longer supported (and several of the existing attempts
at fuswintr(), etc. were buggy and not actually interrupt-safe).

Also augmement the ucas(9) API, making it consistently available on
all plaforms, supporting uniprocessor and multiprocessor systems, even
those that do not have CAS or LL/SC primitives.

Welcome to NetBSD 8.99.37.

Switch discriminates between fsc values and should check the masked fsc value,
not the whole register.

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

return correct signo and code in accordance with return value of uvm_fault.

avoid double-fault caused by reading the instruction when panic

Implement TRAP_SIGDEBUG for aarch64...
ptraced programs die with:
data_abort_handler, 257: pid 199.1 (a.out): signal 11 (trap 0x82000006) @pc 0, addr 0x0, error=Instruction Abort (EL0)

use panic() instead of some printf to show fault status.
useful for ddb "show panic" command.

fix uninitialized, add missing casts

branches: 1.1.2; 1.1.4;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

fpu_kern_enter/leave: Disable IPL assertions.

These don't work because mutex_enter/exit on a spin lock may raise an
IPL but not lower it, if another spin lock was already held. For
example,

mutex_enter(some_lock_at_IPL_VM);
printf("foo\n");
fpu_kern_enter();
...
fpu_kern_leave();
mutex_exit(some_lock_at_IPL_VM);

will trigger the panic, because printf takes a lock at IPL_HIGH where
the IPL wil remain until the mutex_exit. (This was a nightmare to
track down before I remembered that detail of spin lock IPL
semantics...)

x86, arm: Allow fpu_kern_enter/leave while cold.

Normally these are forbidden above IPL_VM, so that FPU usage doesn't
block IPL_SCHED or IPL_HIGH interrupts. But while cold, e.g. during
builtin module initialization at boot, all interrupts are blocked
anyway so it's a moot point.

Also initialize x86 cpu_info_primary.ci_kfpu_spl to -1 so we don't
trip over an assertion about it while cold -- the assertion is meant
to detect reentrance into fpu_kern_enter/leave, which is prohibited.

Also initialize cpu0's ci_kfpu_spl.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

branches: 1.10.2;
Use the dmb/dsb/isb macros... if nothing else they're all now consistent
about the "memory" assembler contraint.

No binary change

Simplify the cpufunc.h header, i.e. always use #include <arm/cpufunc.h>

Add kthread_fpu_enter/exit support to aarch64.

Use pcu_save_all_on_cpu, not pcu_save.

We don't care what curlwp is here; we care whose state is in the fpu
registers.

Limit aarch64 fpu_kern_enter/leave to IPL_VM or below.

Move aarch64/fpu.h to arm/fpu.h.

Draft fpu_kern_enter/leave on aarch64.

When hardware subnormal support is available, disable flush-to-zero.

Similarly, when hardware NaN propagation is available, disable
default-NaN substitution.

This enables IEEE 754 semantics on any hardware that supports it by
default. Programs that want flush-to-zero or default-NaN substitution
can enable them explicitly.

ok ryo@

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.