Support __HAVE_PIC_SET_PRIORITY for Armada XP.

Update to reflect the move of armadaxpreg.h

branches: 1.2.2; 1.2.4;
Add CVS IDs where appropriate.

Add support for Armada XP PIC.

Obtained from Marvell, Semihalf.

Support __HAVE_PIC_SET_PRIORITY for Armada XP.

branches: 1.5.2;
Add support for Armada XP PIC.

Obtained from Marvell, Semihalf.

branches: 1.4.2;
Add PIC hooks for MP and for the ARM Generic Interrupt Controller.

branches: 1.3.4; 1.3.12;
split arch/arm/pic/pic.c so that we can implement more efficient version of splfoo() while using pic framework.

branches: 1.2.18; 1.2.22; 1.2.28;
Merge kernel changes in matt-armv6 to HEAD.

branches: 1.1.2; 1.1.20; 1.1.22; 1.1.24;
file files.pic was initially added on branch matt-armv6.

arm/pic: Sprinkle dtrace probes on interrupt handlers like x86.

arm/pic: Use a local variable to reduce #ifdef scope.

Avoids straddling a conditional this way.

arm/pic: Mark definitions static to match declarations.

arm/pic: Assert ci_cpl doesn't lower across interrupt handler.

Catch up with the per lwp astpending (rather than per cpu) flag in the
preemption code. NFC as preemption (still) isn't enabled.

pic: Update ci_cpl in pic_set_priority callback.

Not all ICs need interrupts disabled to update the priority. DAIF accesses
are not cheap, so push the update of ci_cpl from pic_set_priority to the
IC's pic_set_priority callback, and let the IC driver determine whether
or not it needs interrupts disabled.

arm: No #if DIAGNOSTIC needed any more.

Compiler can (and gcc does) flush call to strlen when the result is
unused, so no performance impact here.

Fix a bug where pic_establish_intr would fail to call pic_establish_irq
if a free pic__iplsources slot was found, i.e. an interrupt handler at
the same ipl had been disestablished previously.

G/C pic_iplsource

KNF

Only perform pic_unblock_percpu if all of the following are true
- mp_online, i.e. APs are running.
- is_mpsafe, i.e. the interrupt handler is MP safe
- is_percpu, i.e. the interrupt actually requires it!

The last one (is_percpu) is true for GIC PPI+SGI only.

Assert we can sleep in pic_add

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

If an SGI or PPI is established after interrupts are enabled, make sure
we unblock the source on _all_ CPUs and not just the CPU that is
establishing the interrupt.

Re-apply

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

There is(/was) no need to use atomic operations on the percpu / cpu_info
members, so don't.

Finally removng the use of percpu should help avoid problems with "late"
attaching cpus.

Revert recent pic optimizations until I have more time to work on this.

branches: 1.69.2;
Inline pic_set_priority and use cpu_dosoftints_ci when available.

Fixup with __HAVE_PREEMPTION code which is currently unused

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag, and should fix
booting on Raspberry Pi 3.

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

pic: avoid pic_do_pending_ints if pic_mark_pending_* has never been called

splx: use pic_set_priority_psw in interrupts disabled case to skip a few
more daif accesses.

pic: reduce the number of daif accesses in pic_do_pending_ints

The caller has already provided daif state. No need to keep updating
daif via pic_set_priority if it's already in the state we need (interrupts
disabled).

Use ENABLE_INTERRUPT() / DISABLE_INTERRUPT() instead of cpsie()/cpsid() in
places where we don't care about the cpsie() return value.

branches: 1.61.2;
intr_ipi_send: assert that kcp is either NULL or contains exactly one CPU

Improve a comment

KNF

KASSERT -> KASSERTMSG

Trailing whitespace

KNF

Reduce some ifdefs.

pic_pending_zero is unnecessary; percpu_alloc already zeroes.

Switch arm pic allocation and initialization to percpu_create.

branches: 1.52.2;
Update pic_add to allocate and return an irqbase if passed
PIC_IRQBASE_ALLOC.

Make pic_sourcebase static

Add reference counts to intr_mask/intr_unmask as calls can be nested, spotted by thorpej

Implement acpi_md_intr_mask and acpi_md_intr_unmask

Add intr_establish_xname support to arm and expose it to intrctl

Fix intrctl for pics with non-0 irqbase

Add support for intrctl(8).

Implement intr_string(9)

Add support for setting and getting interrupt affinity.

fix false positive KASSERT.
the case that secondary processors failed to hatch, and not attached.

when "cpu1 at cpus0: disabled (unresponsive)"

branches: 1.42.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.41.2;
Trailing whitespace

spaces to TAB

decouple pic_ipi_kpreempt() from DDB

The assertion at the bottom of intr_ipi_send is invalid for UP configs; add
a text for arm_cpu_max == 1

remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

unblock interrupts also when reusing a previously freed irq slot.

Serialise work in pic_add.

Add separate IPI routines for IPI_AST and IPI_KPREEMPT.

__HAVE_PREEEMPTION -> __HAVE_PREEMPTION

Don't send IPIs to ourselves if sending to everyone.

Don't need pic_ipi_sender anymore.

Use right kcpuset call.

Adapt pic to deal with the BCM2836 interrupts.
Move pic_pending_pics, pic_pending_ipls, and pic_blocked_irqs into a
structure and make then per-cpu. There is no global interrupt state anymore.

Add __HAVE_PIC_HAVE_PENDING_INTRS and define it if __HAVE_PIC_SET_PRIORITY
is undefined (also define in mvsoc_intr.h since their use of the latter is
peculiar). This new define controls whether the pending interrupt logic is
compiled. The GIC doesn't use pending interrupts since it uses the priority
level on the GIC to control delivery of interrupts, thus there can never
be a pending interrupt. The kernel shrinks about 4KB with the removal of
the pending interupt support,

Don't clear CI_ASTPENDING in exception return, do it in ast() instead.
Add basic support for __HAVE_PREEMPTION.
Use atomic ops for ci_astpending if __HAVE_PREEMPTION is defined.
Use kpreempt_disable/kpreempt_enable

Add aarch64 support (mostly remapping I32_bit and F32_bit onto aarch64 bits).

branches: 1.25.2;
Include opt_multiprocessor.h

In the MULTIPROCESSOR case mark all interrupt handlers not at IPL_VM as
mpsafe, so the kernel_lock is only taken at IPL_VM as per spl(9).

Comment out a printf that was confusing entering ddb output.

branches: 1.22.2;
Implement MI IPI interface with cross-call support.

branches: 1.21.2;
Support MPSAFE interrupts.

Add a mpsafe flag to the intrsource

Mark a diagnostic-only variable

Mark rv as unused (for now)

Use KASSERTMSG

Whitespace

branches: 1.15.2; 1.15.6;
Fix a bug that incorrect arg is passed to pic_block_irqs() on disestablish.
Fixes PR#45371 by Yuichiro Goto.

branches: 1.14.2;
Add #ifdef DIAGNOSTIC / #endif

Add PIC hooks for MP and for the ARM Generic Interrupt Controller.

Fix botched change to use right value for ipl. Thanks jak

Add hooks for __HAVE_PIC_SET_PRIORITY which allows updating of a hardware
(PIC) priority based on current IPL.

Convert a KASSERT to a KASSERTMSG

Add a KASSERT

branches: 1.8.4; 1.8.10; 1.8.12;
split arch/arm/pic/pic.c so that we can implement more efficient version of splfoo() while using pic framework.

Increment ci->ci_intr_depth for the duration of pic_do_pending_ints().
Fixes locking catastrophy commonly noticed with mvsata(4) on evbarm.

branches: 1.6.2; 1.6.4;
Fix lost interrupt. (2/2)
+ It is likely to lose sight of interrupt when the interrupt of irq_base
that is smaller than the same at the level is generated if PIC_MAXSOURCES
is 33 or more.

Fix lost interrupt. (1/2)
+ Change blocked_irqs that dispach if change pending_irqs in the loop.

branches: 1.4.4; 1.4.6;
Use atomic ops to manipulate the bitmasks.

branches: 1.3.8; 1.3.16;
Remove clause 3 and 4 from TNF licenses

Merge kernel changes in matt-armv6 to HEAD.

branches: 1.1.2; 1.1.20; 1.1.22; 1.1.24;
file pic.c was initially added on branch matt-armv6.

Remove GIC_SPLFUNCS.

pic: splx performance improvement

Skip disabling interrupts and check for pending hard interrupts if old
ipl was < IPL_VM.

Make _splraise/_spllower/splx functions available to modules again.

arm: pic: allow overriding _splraise/_spllower/splx

Revert recent pic optimizations until I have more time to work on this.

branches: 1.19.2;
Add DISABLE_INTERRUPT_SAVE(), like DISABLE_INTERRUPT() but also returns
the previous state.

Use DISABLE_INTERRUPT_SAVE()/ENABLE_INTERRUPT() in pic_splfuncs instead
of cpsid()/cpsie(). The difference here is the caller no longer specifies
which bits to disable and enable; on arm32 we continue to use I32_bit and
on aarch64 we now consistently toggle both IRQ and FIQ state.

Make the splx fast path smaller.

Inline pic_set_priority and use cpu_dosoftints_ci when available.

Only disable/enable interrupts if processing pending interrupts, and
inline pic_set_pending_psw.

Adjust ci_intr_depth when processing pending ints

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag, and should fix
booting on Raspberry Pi 3.

pic: avoid pic_do_pending_ints if pic_mark_pending_* has never been called

splx: use pic_set_priority_psw in interrupts disabled case to skip a few
more daif accesses.

splx: restore priority even if interrupts are disabled

splx: only dispatch hard interrupts if interrupts are enabled

splx: only dispatch softints if interrupts are enabled

branches: 1.8.14;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.7.16;
#include <sys/lwp.h>

Add aarch64 support (mostly remapping I32_bit and F32_bit onto aarch64 bits).

branches: 1.5.4;
Do softints after restoring interrupts

branches: 1.4.2; 1.4.4; 1.4.12; 1.4.14;
Add PIC hooks for MP and for the ARM Generic Interrupt Controller.

Add hooks for __HAVE_PIC_SET_PRIORITY which allows updating of a hardware
(PIC) priority based on current IPL.

branches: 1.2.4; 1.2.6; 1.2.14;
Restore ci_intr_depth increment for duration of pic_do_pending_ints() lost
when this code was moved.

branches: 1.1.2;
split arch/arm/pic/pic.c so that we can implement more efficient version of splfoo() while using pic framework.

Remove GIC_SPLFUNCS.

If an SGI or PPI is established after interrupts are enabled, make sure
we unblock the source on _all_ CPUs and not just the CPU that is
establishing the interrupt.

Make _splraise/_spllower/splx functions available to modules again.

arm: pic: allow overriding _splraise/_spllower/splx

Revert recent pic optimizations until I have more time to work on this.

branches: 1.33.2;
machine/cpufunc.h -> arm/cpufunc.h for the benefit of non-evbarm ports

Unfortunately we need to disable interrupts in pic_set_priority to keep
hardware and ci_cpl in sync.

Inline pic_set_priority and use cpu_dosoftints_ci when available.

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag, and should fix
booting on Raspberry Pi 3.

remove "pic_do_pending_int() prototype; no matching function

Provide a pic_set_priority_psw in the case that __HAVE_PIC_SET_PRIORITY
is not defined.

splx: use pic_set_priority_psw in interrupts disabled case to skip a few
more daif accesses.

branches: 1.26.8;
Update pic_add to allocate and return an irqbase if passed
PIC_IRQBASE_ALLOC.

Add reference counts to intr_mask/intr_unmask as calls can be nested, spotted by thorpej

Implement acpi_md_intr_mask and acpi_md_intr_unmask

enlarge pic_name[] from 14 to 16. bcm2835_intr.c@1.20 used to the limit.
(sizeof struct pic_softc was not changed. it's just same as padding)

Add intr_establish_xname support to arm and expose it to intrctl

Increase size of is_irq and pic_irqbase

Implement intr_string(9)

Increase the size of is_irq from 8- to 16-bits to allow for > 256 IRQs per pic.

I added is_affinity to intrsource in the previous commit but it is not used. Remove it.

Add support for setting and getting interrupt affinity.

branches: 1.16.16; 1.16.18;
Protect #include "opt_multiprocessor.h" with _KERNEL_OPT

Add separate IPI routines for IPI_AST and IPI_KPREEMPT.

Add a pic_cpus to the softc which specifies which cpus the pic can send
IPIs to. For GIC, initialize pic_cpus to kcpuset_running since it can handle
all the cpus.

Only if we __HAVE_PIC_PENDING_INTRS do we need the variables to track them.

Add __HAVE_PIC_HAVE_PENDING_INTRS and define it if __HAVE_PIC_SET_PRIORITY
is undefined (also define in mvsoc_intr.h since their use of the latter is
peculiar). This new define controls whether the pending interrupt logic is
compiled. The GIC doesn't use pending interrupts since it uses the priority
level on the GIC to control delivery of interrupts, thus there can never
be a pending interrupt. The kernel shrinks about 4KB with the removal of
the pending interupt support,

branches: 1.11.2;
Sprinkle #include "opt_multiprocessor.h"

branches: 1.10.2;
Implement MI IPI interface with cross-call support.

branches: 1.9.2;
Support MPSAFE interrupts.

Add a mpsafe flag to the intrsource

branches: 1.7.2; 1.7.4; 1.7.8;
Add PIC hooks for MP and for the ARM Generic Interrupt Controller.

Add hooks for __HAVE_PIC_SET_PRIORITY which allows updating of a hardware
(PIC) priority based on current IPL.

branches: 1.5.8; 1.5.16;
fix compile error about missing evcnt definition.
picvar.h is now included very early because of uebayashi's change to sys/param.h.

branches: 1.4.4; 1.4.6;
Use atomic ops to manipulate the bitmasks.

branches: 1.3.8; 1.3.16;
Remove clause 3 and 4 from TNF licenses

Merge kernel changes in matt-armv6 to HEAD.

branches: 1.1.2; 1.1.20; 1.1.22; 1.1.24;
file picvar.h was initially added on branch matt-armv6.

branches: 1.1.2;
file softintr.c was initially added on branch matt-armv6.

branches: 1.1.2;
file softintr.h was initially added on branch matt-armv6.