libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.
For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.

XXX pullup-10

branches: 1.23.14;
Add missing atomic_and_{8,16}_nv_cas.c for __sync_and_and_fetch_{1,2}.
XXX why is not only atomic_and_* symmetric unlike the others?
(in common/lib/libc/atomic/)

branches: 1.22.16;
Use sparc64 code always on 32-bit sparc64 kernels since %psr read/write
instructions used in sparc's atomic_cas.S don't exist in SPARC-V9.

When using clang, do not use the sparc64 code as it doesn't provide the
full set of symbols and clang in -m32 defaults to sparcv8, not
sparcv8plus.

Include C11 load/store helpers.

Fix 64-bit atomic operations on 32-bit kernel and userland, and add
support for missing __sync_*_8 builtins.

Add C++ 2011 <atomic> support functions.

branches: 1.17.4;
Provide the missing __sync_* ops for sparc.

Provide _atomic_cas_{16,8}_up

Provide an assembler version of _atomic_cas_up for sparc - the C code
does not compile to something usable in a RAS. See PR 38482.

branches: 1.14.6; 1.14.12;
Revert test from MACHINE_ARCH to MACHINE, as this can be called with
with MACHINE_ARCH=sparc MACHINE=sparc64 when building 32bit kernels for
ultrasparc hardware. Picked up by nakayama@ - thanks.
Add a comment explaining this odd case

Prefer MACHINE_ARCH to MACHINE in some tests

branches: 1.12.2;
Include userland version of atomic cas routines in librump.
(the kernel version disables interrupts and therefore can't be used)

allow inclusion of atomic ops in librump

Allow atomic ops to be built as part of libpthread.

oops

In libc, use the MI atomic_cas(). The assembly version is faster, but I
think there are PIC issues with it and I don't have a way to fix and test.

branches: 1.7.2;
Only build atomic ops for libkern/libc.

Enable the atomic ops in userspace.

Search files in arch/sparc64/atomic before arch/sparc/atomic in case of
sparc64 32-bit kernel.

Make sparc64 32-bit kernel works again.

branches: 1.4.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.

Use the CAS-based inc/dec variants, since these CPUs don't have atomic
add in hardware (does arm?).

sparc atomic ops

branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_add.c was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_and.c was initially added on branch thorpej-atomic.

branches: 1.13.26;
Provide the missing __sync_* ops for sparc.

Pass PICFLAGS down to cc-as-as and use __PIC__ to decide if it is small
vs big PIC mode. Retire -DPIC and -DBIGPIC.

branches: 1.11.4; 1.11.10;
Add PIC variant for fetching the lock address.

librump.so now works on sparc

branches: 1.10.4;
Fix typo. (s/_HARKDERNEL/_HARDKERNEL/)

branches: 1.9.2;
include sys/param.h for _HARDKERNEL instead of homegrown def.

Include userland version of atomic cas routines in librump.
(the kernel version disables interrupts and therefore can't be used)

enable profiling of assembly functions.

branches: 1.6.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.5.4;
Add atomic_cas_foo_ni().

branches: 1.4.4;
Add some goo to make this work in userspace.

objdump -d to the rescue. I think this should work now.

sparc atomic ops

branches: 1.1.2;
file atomic_cas.S was initially added on branch thorpej-atomic.

Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.

branches: 1.2.26;
Provide _atomic_cas_{16,8}_up

branches: 1.1.2; 1.1.6;
Provide an assembler version of _atomic_cas_up for sparc - the C code
does not compile to something usable in a RAS. See PR 38482.

branches: 1.1.2;
file atomic_dec.c was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_inc.c was initially added on branch thorpej-atomic.

Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.

branches: 1.6.48;
Move SPARC and SPARC64 to modern CPP. Update UPDATING note for this and
Alpha.

Remove clause 3 and 4 from TNF licenses

branches: 1.4.4;
Use __arch64__ instead of __sparc64__, since -D__sparc64__ is appended
to build option of sparc64 32-bit kernel.

branches: 1.3.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.

sparc atomic ops

branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_or.c was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_swap.c was initially added on branch thorpej-atomic.

paravirt_membar_sync(9): New memory barrier.

For use in paravirtualized drivers which require store-before-load
ordering -- irrespective of whether the kernel is built for a single
processor, or whether the (virtual) machine is booted with a single
processor.

This is even required on architectures that don't even have a
store-before-load ordering barrier, like m68k; adding, e.g., a virtio
bus is _as if_ the architecture has been extended with relaxed memory
ordering when talking with that new bus. Such architectures need
some way to request the hypervisor enforce that ordering -- on m68k,
that's done by issuing a CASL instruction, which qemu maps to an
atomic r/m/w with sequential consistency ordering in the host.

PR kern/59618: occasional virtio block device lock ups/hangs

branches: 1.8.2; 1.8.10;
Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.

sparc/membar_ops: Upgrade membar_enter from R/RW to RW/RW.

This will be deprecated soon but let's avoid leaving rakes to trip on
with it arising from disagreement over the documentation (W/RW) and
implementation and usage (R/RW).

sparc: Fix membar_sync with LDSTUB.

membar_sync is required to be a full sequential consistency barrier,
equivalent to MEMBAR #StoreStore|LoadStore|StoreLoad|LoadLoad on
sparcv9. LDSTUB and SWAP are the only pre-v9 instructions that do
this and SWAP doesn't exist on all v7 hardware, so use LDSTUB.

Note: I'm having a hard time nailing down a reference for the
ordering implied by LDSTUB and SWAP. I'm _pretty sure_ SWAP has to
imply store-load ordering since the SPARCv8 manual recommends it for
Dekker's algorithm (which notoriously requires store-load ordering),
and the formal memory model treats LDSTUB and SWAP the same for
ordering. But the v8 and v9 manuals aren't clear.

GCC issues STBAR and LDSTUB, but (a) I don't see why STBAR is
necessary here, (b) STBAR doesn't exist on v7 so it'd be a pain to
use, and (c) from what I've heard (although again it's hard to nail
down authoritative references here) all actual SPARC hardware is TSO
or SC anyway so STBAR is a noop in all the silicon anyway.

Either way, certainly this is better than what we had before, which
was nothing implying ordering at all, just a store!

Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.

branches: 1.4.62;
enable profiling of assembly functions.

branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.2.4; 1.2.8;
Fix the aliases.

sparc atomic ops