branches: 1.8.26;
Provide most missing __sync_*64 primitives for i386

branches: 1.7.8; 1.7.14;
allow inclusion of atomic ops in librump

Allow atomic ops to be built as part of libpthread.

Only build atomic ops for libkern/libc.

Enable the atomic ops in userspace.

branches: 1.3.2;
64-bit atomic ops for i386.

x86 atomic ops.

branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.

paravirt_membar_sync(9): New memory barrier.

For use in paravirtualized drivers which require store-before-load
ordering -- irrespective of whether the kernel is built for a single
processor, or whether the (virtual) machine is booted with a single
processor.

This is even required on architectures that don't even have a
store-before-load ordering barrier, like m68k; adding, e.g., a virtio
bus is _as if_ the architecture has been extended with relaxed memory
ordering when talking with that new bus. Such architectures need
some way to request the hypervisor enforce that ordering -- on m68k,
that's done by issuing a CASL instruction, which qemu maps to an
atomic r/m/w with sequential consistency ordering in the host.

PR kern/59618: occasional virtio block device lock ups/hangs

branches: 1.37.2;
xen: Don't hotpatch away LOCK prefix in xen_mb, even on UP boots.

Both xen_mb and membar_sync are designed to provide store-before-load
ordering, but xen_mb has to provide it in synchronizing guest with
hypervisor, while membar_sync only has to provide it in synchronizing
one (guest) CPU with another (guest) CPU.

It is safe to hotpatch away the LOCK prefix in membar_sync on a
uniprocessor boot because membar_sync is only designed to coordinate
between normal memory on multiple CPUs, and is never necessary when
there's only one CPU involved.

But xen_mb is used to coordinate between the guest and the `device'
implemented by a hypervisor, which might be running on another
_physical_ CPU even if the NetBSD guest only sees one `CPU', i.e.,
one _virtual_ CPU. So even on `uniprocessor' boots, xen_mb must
still issue an instruction with store-before-load ordering on
multiprocessor systems, such as a LOCK ADD (or MFENCE, but MFENCE is
costlier for no benefit here).

No need to change xen_wmb (release ordering, load/store-before-store)
or xen_rmb (acquire ordering, load-before-load/store) because every
x86 store is a store-release and every x86 load is a load-acquire,
even on multiprocessor systems, so there's no hotpatching involved
anyway.

PR kern/57199

branches: 1.36.2; 1.36.8;
x86: Eliminate mfence hotpatch for membar_sync.

The more-compatible LOCK ADD $0,-N(%rsp) turns out to be cheaper
than MFENCE anyway. Let's save some space and maintenance and rip
out the hotpatching for it.

Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.

i386/membar_ops: Upgrade membar_enter from R/RW to RW/RW.

This will be deprecated soon but let's avoid leaving rakes to trip on
with it arising from disagreement over the documentation (W/RW) and
implementation and usage (R/RW).

x86: Add a note on membar_sync and mfence.

x86: Omit needless store in membar_producer/exit.

On x86, every store is a store-release, so there is no need for any
barrier. But this wasn't a barrier anyway; it was just a store,
which was redundant with the store of the return address to the stack
implied by CALL even if issuing a store made a difference.

x86: Every load is a load-acquire, so membar_consumer is a noop.

lfence is only needed for MD logic, such as operations on I/O memory
rather than normal cacheable memory, or special instructions like
RDTSC -- never for MI synchronization between threads/CPUs. No need
for hot-patching to do lfence here.

(The x86_lfence function might reasonably be patched on i386 to do
lfence for MD logic, but it isn't now and this doesn't change that.)

Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.

Use the hotpatch framework when patching _atomic_cas_64.

Use the hotpatch framework for LFENCE/MFENCE.

Remove unused argument in macro.

Remove unused.

Drop the hardcoded array, use the hotpatch section.

Merge the bouyer-xenpvh branch, bringing in Xen PV drivers support under HVM
guests in GENERIC.
Xen support can be disabled at runtime with
boot -c
disable hypervisor

branches: 1.23.8;
On Xen, always alias _atomic_cas_64 to _atomic_cas_cx8. AFAIK Xen doesn't
support CPUs that don't support cx8.
i386 XENPAE_DOMU boots again.

branches: 1.22.22; 1.22.24;
Put missing END() markers to set ELF symbol size.

The kernel uses 64 bit atomic ops.

branches: 1.20.2;
Provide most missing __sync_*64 primitives for i386

branches: 1.19.6; 1.19.12;
Allow use of traditional CPP to be set on a per platform base in sys.mk.
Honour this for dependency processing in bsd.dep.mk. Switch i386 and
amd64 assembly to use ISO C90 preprocessor concat and drop the
-traditional-cpp on this platform.

Use strong alias within the kernel namespace regardless of if we're
dealing with a hard or soft kernel (kernel linker doesn't support
weak symbols).

So that profile kernel runs again,
- Adjust the size of functions used to patch.
- Fix the jump offset of mcount call when patching functions.

Approved by Andrew Doran.

branches: 1.16.2;
include sys/param.h for _HARDKERNEL instead of homegrown def.

Opt for libc versions in case of _KERNEL && !_RUMPKERNEL.
(kernel version uses sti/cli and is not PIC)

PR kern/40213 my i386 machine can't boot because of tsc

- Patch in atomic_cas_64() twice. The first patch is early and makes it
the MP-atomic version available if we have cmpxchg8b. The second patch
strips the lock prefix if ncpu==1.

- Fix the i486 atomic_cas_64() to not unconditionally enable interrupts.

branches: 1.13.4;
enable profiling of assembly functions.

branches: 1.12.2;
whitespace.

rename END to ENDLABEL. i'll use END for other purpose. ok by Andrew Doran.

Remove clause 3 and 4 from TNF licenses

branches: 1.9.4;
Add atomic_cas_foo_ni().

membar_enter was doing the wrong thing. For x86 we can alias:

membar_enter -> membar_consumer
membar_exit -> membar_producer

branches: 1.7.2;
- Make __cpu_simple_lock and similar real functions and patch at runtime.
- Remove old x86 atomic ops.
- Drop text alignment back to 16 on i386 (really, this time).
- Minor cleanup.

64-bit atomic ops for i386.

Add missing strong aliases.

atomic_add_* takes signed integers, the others take unsigned.

A lock prefix on xchg is meaningless.

Fix up a few minor problems.

x86 atomic ops.

branches: 1.1.2;
file atomic_add.S was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_and.S was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_cas_32.S was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_dec.S was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_inc.S was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_or.S was initially added on branch thorpej-atomic.

branches: 1.1.2;
file atomic_swap.S was initially added on branch thorpej-atomic.

branches: 1.1.2;
file membar_ops.S was initially added on branch thorpej-atomic.

branches: 1.4.24;
Put missing END() markers to set ELF symbol size.

branches: 1.3.34; 1.3.48;
Use xchgb, bswapl and not rotate/shift.

branches: 1.2.18;
don't rename bswap{16,32} if defined(_KERNEL) || defined(_STANDALONE).
(they are hidden by gcc-optimized versions, though.)

Merge libkern + libc common files. As requested by core.

branches: 1.4.24;
Put missing END() markers to set ELF symbol size.

branches: 1.3.34; 1.3.48;
Use xchgb, bswapl and not rotate/shift.

branches: 1.2.18;
don't rename bswap{16,32} if defined(_KERNEL) || defined(_STANDALONE).
(they are hidden by gcc-optimized versions, though.)

Merge libkern + libc common files. As requested by core.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.

x86 uses the C versions of bcmp() and memcmp() now.

branches: 1.3.26; 1.3.30;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.2.28; 1.2.34;
Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.

branches: 1.1.18;
Merge libkern + libc common files. As requested by core.

branches: 1.4.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.3.28; 1.3.34;
Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.

branches: 1.2.18;
prefix local labels with ".L".

Merge libkern + libc common files. As requested by core.

branches: 1.1.94;
Merge libkern + libc common files. As requested by core.

branches: 1.5.24;
Put missing END() markers to set ELF symbol size.

branches: 1.4.26; 1.4.40;
Convert to new 2 clause license

branches: 1.3.6;
Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.

branches: 1.2.18;
prefix local labels with ".L".

Merge libkern + libc common files. As requested by core.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.

branches: 1.3.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.2.6; 1.2.12;
Change all archs so that strchr.[cS] and strrchr.[cS] exist and generate
duplicate symbols for index() and rindex().

branches: 1.1.36;
Merge libkern + libc common files. As requested by core.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.

s/Westley/Wesley/ in a book reference (in comments).

s/efficent/efficient/ in comments.

remove duplicate the article in comments.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.

branches: 1.3.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.2.6; 1.2.12;
Change all archs so that strchr.[cS] and strrchr.[cS] exist and generate
duplicate symbols for index() and rindex().

branches: 1.1.36;
Merge libkern + libc common files. As requested by core.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.2; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.

branches: 1.2.24;
Put missing END() markers to set ELF symbol size.

branches: 1.1.8; 1.1.20;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.

branches: 1.1.44;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.2; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.

branches: 1.3.16;
Look for the character to locate before checking for the NUL character

As documented in PR port-i386/49208, this fixes strchr(s, '\0'), as used by
the FAT first-stage bootloader on x86 (bootxx_msdos).
strchr(s, '\0') is otherwise equivalent to strlen(string), which would
probably look nicer in the original file, dosfs.c from libsa.

Confirmed working in qemu and on real hardware.
ok joerg@

XXX pull-up to netbsd-6 and netbsd-7

branches: 1.2.4;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.2; 1.1.4; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.2; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.2; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.

branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.

branches: 1.1.2; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.