Home | History | Annotate | only in /src/common/lib/libc/arch
History log of /src/common/lib/libc/arch
RevisionDateAuthorComments
 1.5 06-Aug-2022  riastradh aarch64: Implement __aarch64_casN_sync.

gcc generates calls to this symbol in programs that use
__sync_*_compare_and_swap, which require full sequential consistency
barriers, including store-before-load ordering on both sides of the
atomic; none of the release/acquire operations guarantee that, so we
have to insert explicit DMB instructions.

Note: gcc's own definition omits some of the DMB instructions, but I
can't prove that it's correct that way -- stores preceding the CAS
must complete before the load part of the CAS, and the store part of
the CAS must complete before loads following the CAS. Maybe there's
some way to prove that one of these orderings is guaranteed some
other way than a DMB but I'm not seeing it, and store-before-load
ordering is hard to understand.

Patch by skrll@ based on a patch by mrg@, soliloquy in commit message
by me.
 1.4 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.3 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file Makefile.inc was added on branch tls-maxphys on 2014-08-19 23:45:11 +0000
 1.7 06-Aug-2022  riastradh aarch64: Implement __aarch64_casN_sync.

gcc generates calls to this symbol in programs that use
__sync_*_compare_and_swap, which require full sequential consistency
barriers, including store-before-load ordering on both sides of the
atomic; none of the release/acquire operations guarantee that, so we
have to insert explicit DMB instructions.

Note: gcc's own definition omits some of the DMB instructions, but I
can't prove that it's correct that way -- stores preceding the CAS
must complete before the load part of the CAS, and the store part of
the CAS must complete before loads following the CAS. Maybe there's
some way to prove that one of these orderings is guaranteed some
other way than a DMB but I'm not seeing it, and store-before-load
ordering is hard to understand.

Patch by skrll@ based on a patch by mrg@, soliloquy in commit message
by me.
 1.6 23-Jul-2022  skrll whitespace
 1.5 18-Jun-2022  skrll be consistent about comparing loaded value against expected old value
register ordering
 1.4 18-Jun-2022  skrll Fix some register usage
 1.3 16-Jun-2022  skrll remove stray 'w'
 1.2 08-Aug-2021  skrll Whitespace
 1.1 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 27-Apr-2021  skrll Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.
 1.1 21-Apr-2021  skrll Do previous differently as the API is different.
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_add_16.S was added on branch tls-maxphys on 2014-08-19 23:45:11 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_add_32.S was added on branch tls-maxphys on 2014-08-19 23:45:11 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_add_64.S was added on branch tls-maxphys on 2014-08-19 23:45:11 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_add_8.S was added on branch tls-maxphys on 2014-08-19 23:45:11 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_and_16.S was added on branch tls-maxphys on 2014-08-19 23:45:11 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_and_32.S was added on branch tls-maxphys on 2014-08-19 23:45:11 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_and_64.S was added on branch tls-maxphys on 2014-08-19 23:45:11 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_and_8.S was added on branch tls-maxphys on 2014-08-19 23:45:11 +0000
 1.4 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3 07-Oct-2020  skrll Comment nit
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_cas_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.4 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3 07-Oct-2020  skrll Comment nit
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_cas_32.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.6 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5 07-Oct-2020  skrll Comment nit
 1.4 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.3 19-Feb-2019  rin branches: 1.3.2;
Sort STRONG_ALIAS's in the same manner as ATOMIC_OP_ALIAS's.
No functional changes.
 1.2 19-Feb-2019  rin Export _atomic_cas_64 as atomic_cas_64_ni.

Note that _atomic_cas_64 is already exported as atomic_cas_{ulong,prt}_ni.

Fix build error of test/lib/atomic/t_atomic_cas, which is successfully
passed on RPI3B+ now.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.26.2 21-Apr-2020  martin Sync with HEAD
 1.1.26.1 10-Jun-2019  christos Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_cas_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.3.2.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3 07-Oct-2020  skrll Comment nit
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_cas_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_dec_32.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_dec_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_inc_32.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_inc_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 06-Jul-2021  skrll One more s/pte/ptr/
 1.3 04-Jul-2021  skrll Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows

{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.3 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.28.2 08-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1319):

common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.4

One more s/pte/ptr/
 1.1.28.1 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1314):

common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.4

Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows
{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.

typo in comment s/pte/ptr/
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_nand_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 05-Jul-2021  skrll typo in comment s/pte/ptr/
 1.3 04-Jul-2021  skrll Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows

{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.28.1 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1314):

common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.4

Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows
{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.

typo in comment s/pte/ptr/
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_nand_32.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 05-Jul-2021  skrll typo in comment s/pte/ptr/
 1.3 04-Jul-2021  skrll Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows

{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.28.1 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1314):

common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.4

Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows
{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.

typo in comment s/pte/ptr/
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_nand_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 05-Jul-2021  skrll typo in comment s/pte/ptr/
 1.3 04-Jul-2021  skrll Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows

{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.28.1 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1314):

common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.4

Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows
{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.

typo in comment s/pte/ptr/
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_nand_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.6 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5 13-Oct-2020  skrll Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.
 1.4 07-Oct-2020  skrll Comment nit
 1.3 08-Feb-2019  ryo branches: 1.3.2;
- atomic_*_{8,16}_nv() must return a new value, not an old value.
- use "dmb sy" for atomic_*{8,16}_nv() in the same way as atomic_*{32,64}_nv().
 1.2 06-Feb-2019  ryo fix atomic_sub_*(). it was (delta - *ptr), should be (*ptr - delta).
changing shared macro doesn't effect other atomic_ops because
(*ptr [+|^] delta) and (delta [+|^] *ptr) have same result.

atomic_sub_*() haven't used because non standard API?
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.26.2 21-Apr-2020  martin Sync with HEAD
 1.1.26.1 10-Jun-2019  christos Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_op_asm.h was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.3.2.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_or_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_or_32.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.3 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.2 15-Sep-2019  skrll __sync_or_and_fetch_8 should return new value... make it do that.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.28.1 17-Sep-2019  martin Pull up following revision(s) (requested by skrll in ticket #201):

common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.2

__sync_or_and_fetch_8 should return new value... make it do that.
 1.1.26.2 21-Apr-2020  martin Sync with HEAD
 1.1.26.1 13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_or_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_or_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_sub_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_sub_32.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_sub_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_sub_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 21-Apr-2021  skrll Do previous differently as the API is different.
 1.3 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_swap_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 21-Apr-2021  skrll Do previous differently as the API is different.
 1.3 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_swap_32.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 21-Apr-2021  skrll Do previous differently as the API is different.
 1.3 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_swap_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 21-Apr-2021  skrll Do previous differently as the API is different.
 1.3 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 12-Aug-2020  skrll Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_swap_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 21-Apr-2021  skrll Do previous differently as the API is different.
 1.1 21-Apr-2021  skrll Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_xor_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_xor_32.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_xor_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 29-Jul-2021  skrll As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file atomic_xor_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.4 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.3 09-Apr-2022  riastradh aarch64/membar_ops: Fix wrong symbol end.
 1.2 13-Oct-2020  skrll Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1331):

common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops. I don't understand why thos=
e
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file membar_ops.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file byte_swap_2.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file byte_swap_4.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 12-Oct-2019  maya Remove htonll and ntohll as symbols from aarch64 libc.

Other architectures do not define them, and so we don't provide a
function declaration in any header.

This means a package may detect it with a link-test and then fail
due to the missing declaration, like sysutils/collectd currently does.

Done this way as aarch64 has not had a release yet. Discussed with releng.
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.28.1 12-Oct-2019  martin Pull up following revision(s) (requested by maya in ticket #304):

common/lib/libc/arch/aarch64/gen/byte_swap_8.S: revision 1.2

Remove htonll and ntohll as symbols from aarch64 libc.

Other architectures do not define them, and so we don't provide a
function declaration in any header.

This means a package may detect it with a link-test and then fail
due to the missing declaration, like sysutils/collectd currently does.

Done this way as aarch64 has not had a release yet. Discussed with releng.
 1.1.26.2 21-Apr-2020  martin Sync with HEAD
 1.1.26.1 13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file byte_swap_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.3 03-Sep-2020  jakllsch Remove unused assembly source files
 1.2 02-Sep-2020  jakllsch Fix typo/pasteo in aarch64 clzdi2() END()
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file clzdi2.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 03-Sep-2020  jakllsch Remove unused assembly source files
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file ctzdi2.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 03-Sep-2020  jakllsch Remove unused assembly source files
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file ffsdi2.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.4 07-Feb-2024  msaitoh Remove ryo@'s mail addresses.
 1.3 23-Jul-2023  skrll port-arm/57388: Minor bug fix in bcopy.S

Use correct register to check alignment of destination in backwards copy.

Patch from Antoni Pokusinski. Thanks.
 1.2 11-Apr-2020  ryo branches: 1.2.8;
Fixed to not use the "br" instruction. Branch Target Identification (BTI) doesn't like "br".

requested by maxv@
 1.1 04-Feb-2018  skrll branches: 1.1.4;

Working / new versions from Ryo Shimizu
 1.1.4.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.4.2 21-Apr-2020  martin Sync with HEAD
 1.1.4.1 13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.2.8.1 20-Jul-2024  martin Pull up following revision(s) (requested by rin in ticket #744):

common/lib/libc/arch/aarch64/string/bcopy.S: revision 1.3

port-arm/57388: Minor bug fix in bcopy.S

Use correct register to check alignment of destination in backwards copy.
Patch from Antoni Pokusinski. Thanks.
 1.3 09-Jul-2018  ryo avoid reading from out of range that may cause access fault.
 1.2 04-Feb-2018  skrll branches: 1.2.2; 1.2.4;

Working / new versions from Ryo Shimizu
 1.1 10-Aug-2014  matt branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file memcmp.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.4.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.4.2 21-Apr-2020  martin Sync with HEAD
 1.2.4.1 10-Jun-2019  christos Sync with HEAD
 1.2.2.1 28-Jul-2018  pgoyette Sync with HEAD
 1.2 04-Feb-2018  skrll branches: 1.2.4;

Working / new versions from Ryo Shimizu
 1.1 10-Aug-2014  matt branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file memcpy.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.4.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.4.1 21-Apr-2020  martin Sync with HEAD
 1.1 04-Feb-2018  skrll branches: 1.1.4;

Working / new versions from Ryo Shimizu
 1.1.4.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.4.1 21-Apr-2020  martin Sync with HEAD
 1.3 11-Apr-2020  ryo Fixed to not use the "br" instruction. Branch Target Identification (BTI) doesn't like "br".

requested by maxv@
 1.2 29-Aug-2017  ryo branches: 1.2.4;
* aarch64/memset.S didn't work! fixed some bugs.
* maximum size of DCZID_EL0:BS (2048) supported.
 1.1 10-Aug-2014  matt branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file memset.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.4.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.4.2 21-Apr-2020  martin Sync with HEAD
 1.2.4.1 13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file strcat.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 09-Sep-2020  jakllsch Re-do previous aarch64eb strlen fix more simply and correctly.
 1.4 05-Sep-2020  jakllsch Fix a broken corner case of strlen()/strnlen() on aarch64eb

Previously a string such as "\x1\x1\x1\x1\x1\x1\x1" would count as
0 instead of 7 on BE.
 1.3 01-Aug-2018  ryo strnlen(s, (size_t)-1) returned -1. it must return the length of s.
 1.2 22-Aug-2017  ryo branches: 1.2.2; 1.2.4;
aarch64/strlen.S didn't work. fixed some bugs.
 1.1 10-Aug-2014  matt branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file strlen.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.4.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.4.2 21-Apr-2020  martin Sync with HEAD
 1.2.4.1 10-Jun-2019  christos Sync with HEAD
 1.2.2.1 06-Sep-2018  pgoyette Sync with HEAD

Resolve a couple of conflicts (result of the uimin/uimax changes)
 1.1 10-Aug-2014  matt branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.
 1.1.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.26.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 10-Aug-2014  tls file strnlen.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 04-Jan-2009  pooka branches: 1.5.50;
allow inclusion of atomic ops in librump
 1.4 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.3 11-Feb-2008  ad Only build atomic ops for libkern/libc.
 1.2 28-Nov-2007  ad branches: 1.2.4;
Pull in alpha atomic ops from the thorpej-atomic branch.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.
 1.1.2.1 18-Apr-2007  thorpej Atomic op implementation for Alpha.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:20:47 +0000
 1.5.50.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.50.1 21-Apr-2020  martin Sync with HEAD
 1.5 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.4 25-May-2008  chs branches: 1.4.62;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in alpha atomic ops from the thorpej-atomic branch.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file atomic_add.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 18-Apr-2007  thorpej Atomic op implementation for Alpha.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_add.S was added on branch matt-armv6 on 2008-01-09 01:20:47 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.62.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.62.1 21-Apr-2020  martin Sync with HEAD
 1.5 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.4 25-May-2008  chs branches: 1.4.62;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in alpha atomic ops from the thorpej-atomic branch.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file atomic_and.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 18-Apr-2007  thorpej Atomic op implementation for Alpha.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_and.S was added on branch matt-armv6 on 2008-01-09 01:20:48 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.62.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.62.1 21-Apr-2020  martin Sync with HEAD
 1.6 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.5 25-May-2008  chs branches: 1.5.62;
enable profiling of assembly functions.
 1.4 28-Apr-2008  martin branches: 1.4.2;
Remove clause 3 and 4 from TNF licenses
 1.3 10-Feb-2008  ad branches: 1.3.4;
Add atomic_cas_foo_ni().
 1.2 28-Nov-2007  ad branches: 1.2.4;
Pull in alpha atomic ops from the thorpej-atomic branch.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file atomic_cas.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 18-Apr-2007  thorpej Atomic op implementation for Alpha.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_cas.S was added on branch matt-armv6 on 2008-01-09 01:20:48 +0000
 1.3.4.2 04-Jun-2008  yamt sync with head
 1.3.4.1 18-May-2008  yamt sync with head.
 1.4.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.5.62.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.62.1 21-Apr-2020  martin Sync with HEAD
 1.5 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.4 25-May-2008  chs branches: 1.4.62;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in alpha atomic ops from the thorpej-atomic branch.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file atomic_dec.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 18-Apr-2007  thorpej Atomic op implementation for Alpha.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_dec.S was added on branch matt-armv6 on 2008-01-09 01:20:48 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.62.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.62.1 21-Apr-2020  martin Sync with HEAD
 1.5 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.4 25-May-2008  chs branches: 1.4.62;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in alpha atomic ops from the thorpej-atomic branch.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file atomic_inc.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 18-Apr-2007  thorpej Atomic op implementation for Alpha.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_inc.S was added on branch matt-armv6 on 2008-01-09 01:20:49 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.62.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.62.1 21-Apr-2020  martin Sync with HEAD
 1.4 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.3 28-Apr-2008  martin branches: 1.3.64;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in alpha atomic ops from the thorpej-atomic branch.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.
 1.1.2.1 18-Apr-2007  thorpej Atomic op implementation for Alpha.
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_op_asm.h was added on branch matt-armv6 on 2008-01-09 01:20:49 +0000
 1.3.64.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.64.1 21-Apr-2020  martin Sync with HEAD
 1.5 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.4 25-May-2008  chs branches: 1.4.62;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in alpha atomic ops from the thorpej-atomic branch.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file atomic_or.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 18-Apr-2007  thorpej Atomic op implementation for Alpha.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_or.S was added on branch matt-armv6 on 2008-01-09 01:20:50 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.62.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.62.1 21-Apr-2020  martin Sync with HEAD
 1.5 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.4 25-May-2008  chs branches: 1.4.62;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in alpha atomic ops from the thorpej-atomic branch.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file atomic_swap.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 18-Apr-2007  thorpej Atomic op implementation for Alpha.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_swap.S was added on branch matt-armv6 on 2008-01-09 01:20:50 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.62.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.62.1 21-Apr-2020  martin Sync with HEAD
 1.9 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 08-Jan-2015  riastradh branches: 1.7.16;
Introduce membar_datadep_consumer.

Discussed briefly on tech-kern without objection:

https://mail-index.netbsd.org/tech-kern/2014/11/20/msg018054.html
https://mail-index.netbsd.org/tech-kern/2015/01/07/msg018326.html
 1.6 25-May-2008  chs branches: 1.6.40;
enable profiling of assembly functions.
 1.5 28-Apr-2008  martin branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses
 1.4 12-Feb-2008  ad branches: 1.4.4;
Always enable memory barriers in userspace, otherwise libpthread won't
work correctly on MP systems.
 1.3 12-Feb-2008  matt Fix pthread problem. (export membar* to userland)
 1.2 28-Nov-2007  ad branches: 1.2.4;
Pull in alpha atomic ops from the thorpej-atomic branch.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file membar_ops.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 18-Apr-2007  thorpej Atomic op implementation for Alpha.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file membar_ops.S was added on branch matt-armv6 on 2008-01-09 01:20:51 +0000
 1.4.4.2 04-Jun-2008  yamt sync with head
 1.4.4.1 18-May-2008  yamt sync with head.
 1.5.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.6.40.1 18-Mar-2015  snj Pull up following revision(s) (requested by riastradh in ticket #600):
common/lib/libc/arch/alpha/atomic/membar_ops.S: revision 1.7
lib/libc/atomic/membar_ops.3: revision 1.4
sys/arch/alpha/include/types.h: revision 1.50
sys/sys/atomic.h: revision 1.13
Introduce membar_datadep_consumer.
Discussed briefly on tech-kern without objection:
https://mail-index.netbsd.org/tech-kern/2014/11/20/msg018054.html
https://mail-index.netbsd.org/tech-kern/2015/01/07/msg018326.html
 1.7.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.16.1 21-Apr-2020  martin Sync with HEAD
 1.3 16-Feb-2008  apb branches: 1.3.70;
For all files under src/common, as well as related man pages under
src/lib/libc, or related headers under src/sys/sys: change u_intNN_t to
uintNN_t.
 1.2 27-Dec-2005  yamt branches: 1.2.6; 1.2.18;
make alpha kernel buildable again.
don't add __ prefix to bswap{16,32}
if defined(_KERNEL) || defined(_STANDALONE).
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.18.1 23-Mar-2008  matt sync with HEAD
 1.2.6.1 18-Nov-2008  snj Pull up following revision(s) (requested by tsutsui in ticket #1224):
common/lib/libc/gen/bswap64.c: revision 1.2
common/lib/libprop/prop_data.c: revision 1.10
common/lib/libc/inet/inet_addr.c: revision 1.2
common/lib/libc/hash/rmd160/rmd160.c: revision 1.4
common/lib/libc/gen/bswap32.c: revision 1.2
common/lib/libc/gen/bswap16.c: revision 1.2
common/lib/libc/arch/alpha/gen/byte_swap_4.S: revision 1.3
common/lib/libc/arch/sh3/string/memset.S: revision 1.2
common/lib/libc/arch/alpha/gen/byte_swap_2.S: revision 1.3
common/lib/libc/md/md5c.c: revision 1.3
common/lib/libc/hash/sha1/sha1.c: revision 1.3
common/lib/libc/md/md4c.c: revision 1.3
sys/sys/rmd160.h: revision 1.2
sys/sys/sha2.h: revision 1.2
For all files under src/common, as well as related man pages under
src/lib/libc, or related headers under src/sys/sys: change u_intNN_t to
uintNN_t.
 1.3.70.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.70.1 21-Apr-2020  martin Sync with HEAD
 1.3 16-Feb-2008  apb branches: 1.3.70;
For all files under src/common, as well as related man pages under
src/lib/libc, or related headers under src/sys/sys: change u_intNN_t to
uintNN_t.
 1.2 27-Dec-2005  yamt branches: 1.2.6; 1.2.18;
make alpha kernel buildable again.
don't add __ prefix to bswap{16,32}
if defined(_KERNEL) || defined(_STANDALONE).
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.18.1 23-Mar-2008  matt sync with HEAD
 1.2.6.1 18-Nov-2008  snj Pull up following revision(s) (requested by tsutsui in ticket #1224):
common/lib/libc/gen/bswap64.c: revision 1.2
common/lib/libprop/prop_data.c: revision 1.10
common/lib/libc/inet/inet_addr.c: revision 1.2
common/lib/libc/hash/rmd160/rmd160.c: revision 1.4
common/lib/libc/gen/bswap32.c: revision 1.2
common/lib/libc/gen/bswap16.c: revision 1.2
common/lib/libc/arch/alpha/gen/byte_swap_4.S: revision 1.3
common/lib/libc/arch/sh3/string/memset.S: revision 1.2
common/lib/libc/arch/alpha/gen/byte_swap_2.S: revision 1.3
common/lib/libc/md/md5c.c: revision 1.3
common/lib/libc/hash/sha1/sha1.c: revision 1.3
common/lib/libc/md/md4c.c: revision 1.3
sys/sys/rmd160.h: revision 1.2
sys/sys/sha2.h: revision 1.2
For all files under src/common, as well as related man pages under
src/lib/libc, or related headers under src/sys/sys: change u_intNN_t to
uintNN_t.
 1.3.70.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.70.1 21-Apr-2020  martin Sync with HEAD
 1.2 21-Dec-2005  christos branches: 1.2.94;
Only build this if GPROF && !_STANDALONE
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.3 11-Oct-2020  skrll Define _ARM_ARCH_8 when __ARM_ARCH_8A (no trailing double underscore) as
it is defined by gcc.

__ARM_ARCH_8A__ (with trailing double underscore) seems to be a typo (or
maybe historical)
 1.2 02-Aug-2019  joerg ARMv6KZ has been misspelled by GCC since forever, but clang only
provides the correct name. Support both.
 1.1 27-Feb-2014  matt branches: 1.1.4; 1.1.8; 1.1.30;
Add a method to test the compiler for things like LDREX availability,
LDRD availability, THUMB2, EABI.
 1.1.30.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.30.2 21-Apr-2020  martin Sync with HEAD
 1.1.30.1 13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.1.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.8.1 27-Feb-2014  tls file features.c was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.1.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.4.1 27-Feb-2014  yamt file features.c was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2 07-Jun-2015  matt branches: 1.2.16;
Use ${COPT} so it properly emits the right features doing MKCOMPAT
 1.1 27-Feb-2014  matt branches: 1.1.4; 1.1.6; 1.1.8;
Add a method to test the compiler for things like LDREX availability,
LDRD availability, THUMB2, EABI.
 1.1.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.8.1 27-Feb-2014  tls file features.mk was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.1.6.1 10-Jun-2015  snj Pull up following revision(s) (requested by martin in ticket #837):
common/lib/libc/arch/arm/features.mk: revision 1.2
lib/libarch/Makefile: revision 1.29
lib/libarch/i386/Makefile.inc: revision 1.15
lib/libarch/x86_64/Makefile.inc: revision 1.5
share/mk/bsd.gcc.mk: revision 1.11
share/mk/bsd.prog.mk: revision 1.292
Make LIBCRT* depend on the right files if MLIBDIR from MKCOMPAT is defined.
Use ${COPT} so it properly emits the right features doing MKCOMPAT
Avoid defining MLIBDIR. Use ${MLIBDIR:Unone} instead.
 1.1.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.4.1 27-Feb-2014  yamt file features.mk was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.29 07-Jun-2015  matt branches: 1.29.16;
Back out last change; fixed in the correct place.
 1.28 07-Jun-2015  joerg Require ARMv5TE to assemble.
 1.27 17-May-2015  justin Move arm sync_* changes to Makefile.inc
 1.26 14-Oct-2014  martin Provide C++ 2011 <atomic> support functions for hppa and arm.
 1.25 13-Oct-2014  martin Move the and_{16,8}_nv sources into the right (libc only) block.
 1.24 13-Oct-2014  martin Provide __sync_and_and_fetch_2 and __sync_and_and_fetch_1 for pre-ARMv6,
they are needed for the C++ 2011 <atomic> stuff.
 1.23 05-Jul-2014  joerg branches: 1.23.2;
Provide a basic implementation of __atomic_load_* and __atomic_store_*,
used by GCC and LLVM as backing for C11/C++11 atomics, if the hardware
is not known to have corresponding features. Include it on ARM as LLVM
and libc++ hit it when compiled for ARMv4.
 1.22 04-Mar-2014  matt branches: 1.22.2;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.21 04-Mar-2014  matt fix typo.
 1.20 04-Mar-2014  matt Add atomic_sub_64.S
 1.19 27-Feb-2014  matt Add atomic_cas_64 support for ARM EABI on V5TE and V5TEJ cpus.
(strd is atomic).
 1.18 22-Feb-2014  martin Missed one __sync_* op (or gcc4.8 does inline it, while 4.5 does not?)
 1.17 22-Feb-2014  martin Move the __sync_* ops added in the previous change to a libc-only section
 1.16 21-Feb-2014  martin Provide the missing __sync_* ops for earlier arm versions
 1.15 27-Jan-2014  matt Add _atomic_cas_16_up and _atomic_cas_8_up
 1.14 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.13 19-Aug-2013  matt Rework to allow thumb armv7 compilation.
Add atomic_simplelock.c for thumb
 1.12 06-Aug-2013  matt Only use MACHINE_ARCH if !_STANDALONE and CPPFLAGS/CFLAGS/CPUFLAGS don't
contain -mcpu or -march
 1.11 06-Aug-2013  matt Select ldrex/strex if ${MACHINE_ARCH:Mearmv[67]*}
 1.10 11-Sep-2012  matt branches: 1.10.2;
Add 64bit atomic ops for ARMv6+ (using ldrexd/strexd).
 1.9 16-Aug-2012  matt Actually use the assembly version of the atomic function if compiling
for armv6 or armv7 cpus. Use atomic_cas_ptr instead of _lock_cas so
we pick up the assembly version when it's used.
 1.8 04-Jan-2009  pooka branches: 1.8.8; 1.8.12;
allow inclusion of atomic ops in librump
 1.7 29-Sep-2008  ad branches: 1.7.8;
Allow atomic ops to be built as part of libpthread.
 1.6 29-Apr-2008  scw Implement _atomic_cas_up() in assembly code as the compiler cannot be
trusted to generate fully restartable code sequences.

Addresses lib/38482 for ARM and m68000.
 1.5 11-Feb-2008  ad branches: 1.5.4;
Only build atomic ops for libkern/libc.
 1.4 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.3 29-Nov-2007  ad branches: 1.3.4;
Use the CAS-based inc/dec variants, since these CPUs don't have atomic
add in hardware (does arm?).
 1.2 29-Nov-2007  ad Atomic ops for arm.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.
 1.1.2.2 26-Aug-2007  matt Add armv6 variants of atomic ops which use ldrex/strex.
Add atomic_swap (all arm architectures) which uses swp and swpb.
Add membar_ops which uses armv6 data {sync, memory} barrier.
XXX let thorpej figure out the Makefile magic.
 1.1.2.1 18-Apr-2007  thorpej Atomic ops implementation for ARM. Everything is built on top of
atomic_cas_32(), which is itself a restartable atomic sequence.
 1.3.4.3 23-Mar-2008  matt sync with HEAD
 1.3.4.2 09-Jan-2008  matt sync with HEAD
 1.3.4.1 29-Nov-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:20:51 +0000
 1.5.4.1 18-May-2008  yamt sync with head.
 1.7.8.2 15-Feb-2014  matt Add 64-bit atomic ops if ARMv6+
 1.7.8.1 19-Dec-2013  matt Enable ldrex/strex based atomic ops on armv6/armv7
 1.8.12.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.8.8.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.8.8.1 30-Oct-2012  yamt sync with head
 1.10.2.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.22.2.1 10-Aug-2014  tls Rebase.
 1.23.2.1 12-Nov-2014  snj Pull up following revision(s) (requested by martin in ticket #218):
common/lib/libc/arch/arm/atomic/Makefile.inc: revision 1.24-1.26
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.7
common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.18
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.7
common/lib/libc/atomic/atomic_and_16_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_and_8_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_16.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_32.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_8.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_cas_by_cas32.c: revision 1.4
common/lib/libc/atomic/atomic_op_namespace.h: revision 1.7
Add __sync_val_compare_and_swap_{1,2} aliases for _atomic_cas_{8,16}
--
Provide __atomic_compare_exchange_N (as needed for the C11 2011 <atomic>
ops) via the corresponding CAS.
--
Hook __atomic_compare_exchange_N into vax libc.
--
Provide __sync_and_and_fetch_2 and __sync_and_and_fetch_1 for pre-ARMv6,
they are needed for the C++ 2011 <atomic> stuff.
--
Add C++ 2011 <atomic> support functions.
--
Move the and_{16,8}_nv sources into the right (libc only) block.
--
Provide <atomic> C++ 2011 support functions for mips and sh3.
--
Provide C++ 2011 <atomic> support functions for hppa and arm.
--
Provide prototypes to fix build with clang.
 1.29.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.29.16.1 21-Apr-2020  martin Sync with HEAD
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26; 1.3.28;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 04-Mar-2014  matt branches: 1.2.2; 1.2.4;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_add_16.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_add_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.10 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.9 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.8 23-Jun-2014  joerg branches: 1.8.24; 1.8.26;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.7 22-Feb-2014  martin branches: 1.7.2;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.6 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.5 11-Aug-2013  matt Use foo{s} since it reduces the ifdefs for thumb
 1.4 10-Aug-2013  matt Make these under Thumb2
 1.3 31-Aug-2012  matt branches: 1.3.2;
Add dmb/dsb instructions as required by the armv7 arch man.
 1.2 16-Aug-2008  matt branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex
 1.1 26-Aug-2007  matt branches: 1.1.2;
file atomic_add_32.S was initially added on branch thorpej-atomic.
 1.1.2.2 27-Aug-2007  matt Don't use r4 (since it's caller saved). Use ip (it's available since we
are leaf functions).
 1.1.2.1 26-Aug-2007  matt Add armv6 variants of atomic ops which use ldrex/strex.
Add atomic_swap (all arm architectures) which uses swp and swpb.
Add membar_ops which uses armv6 data {sync, memory} barrier.
XXX let thorpej figure out the Makefile magic.
 1.2.24.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Oct-2012  yamt sync with head
 1.2.2.2 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.2.2.1 16-Aug-2008  wrstuden file atomic_add_32.S was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000
 1.3.2.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.7.2.1 10-Aug-2014  tls Rebase.
 1.8.26.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.8.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.24.1 21-Apr-2020  martin Sync with HEAD
 1.14 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.13 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.12 28-Jun-2021  skrll Whitespace
 1.11 04-Mar-2014  matt branches: 1.11.26; 1.11.28;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.10 30-Nov-2013  joerg branches: 1.10.2;
Explicitly name the register pairs.
 1.9 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.8 20-Aug-2013  matt Push two registers to keep stack aligned.
 1.7 11-Aug-2013  matt Use foo{s} since it reduces the ifdefs for thumb
 1.6 10-Aug-2013  matt Make these under Thumb2
 1.5 10-Aug-2013  matt Never modify NLO/NHI (r2,r3) only LO/HI (r0,r1) so subsequent loops will work.
 1.4 10-Aug-2013  matt use push/pop
 1.3 13-Sep-2012  matt branches: 1.3.2; 1.3.4;
Correct copyright/fix comments.
 1.2 12-Sep-2012  matt Fix bas code, use ldr
 1.1 11-Sep-2012  matt branches: 1.1.2;
Add 64bit atomic ops for ARMv6+ (using ldrexd/strexd).
 1.1.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.3.4.2 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.3.4.1 13-Sep-2012  matt file atomic_add_64.S was added on branch matt-nb6-plus on 2012-11-27 23:42:35 +0000
 1.3.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.2.2 30-Oct-2012  yamt sync with head
 1.3.2.1 13-Sep-2012  yamt file atomic_add_64.S was added on branch yamt-pagecache on 2012-10-30 18:46:10 +0000
 1.10.2.2 15-Feb-2014  matt Add 64-bit atomic ops if ARMv6+
 1.10.2.1 30-Nov-2013  matt file atomic_add_64.S was added on branch matt-nb5-mips64 on 2014-02-15 10:27:44 +0000
 1.11.28.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.11.28.1 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1313):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.12

Whitespace
 1.11.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.11.26.1 21-Apr-2020  martin Sync with HEAD
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26; 1.3.28;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 04-Mar-2014  matt branches: 1.2.2; 1.2.4;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_add_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_add_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26; 1.3.28;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 04-Mar-2014  matt branches: 1.2.2; 1.2.4;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_and_16.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_and_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.10 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.9 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.8 23-Jun-2014  joerg branches: 1.8.24; 1.8.26;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.7 22-Feb-2014  martin branches: 1.7.2;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.6 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.5 11-Aug-2013  matt Use foo{s} since it reduces the ifdefs for thumb
 1.4 10-Aug-2013  matt Make these under Thumb2
 1.3 31-Aug-2012  matt branches: 1.3.2;
Add dmb/dsb instructions as required by the armv7 arch man.
 1.2 16-Aug-2008  matt branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex
 1.1 26-Aug-2007  matt branches: 1.1.2;
file atomic_and_32.S was initially added on branch thorpej-atomic.
 1.1.2.2 27-Aug-2007  matt Don't use r4 (since it's caller saved). Use ip (it's available since we
are leaf functions).
 1.1.2.1 26-Aug-2007  matt Add armv6 variants of atomic ops which use ldrex/strex.
Add atomic_swap (all arm architectures) which uses swp and swpb.
Add membar_ops which uses armv6 data {sync, memory} barrier.
XXX let thorpej figure out the Makefile magic.
 1.2.24.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Oct-2012  yamt sync with head
 1.2.2.2 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.2.2.1 16-Aug-2008  wrstuden file atomic_and_32.S was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000
 1.3.2.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.7.2.1 10-Aug-2014  tls Rebase.
 1.8.26.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.8.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.24.1 21-Apr-2020  martin Sync with HEAD
 1.13 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.12 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.11 28-Jun-2021  skrll Whitespace
 1.10 04-Mar-2014  matt branches: 1.10.26; 1.10.28;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.9 30-Nov-2013  joerg branches: 1.9.2;
Use explicit form of register pair operations by specifying both.
 1.8 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.7 20-Aug-2013  matt Push two registers to keep stack aligned.
 1.6 11-Aug-2013  matt Use foo{s} since it reduces the ifdefs for thumb
 1.5 10-Aug-2013  matt Make these under Thumb2
 1.4 10-Aug-2013  matt Never modify NLO/NHI (r2,r3) only LO/HI (r0,r1) so subsequent loops will work.
 1.3 10-Aug-2013  matt Use push/pop
 1.2 13-Sep-2012  matt branches: 1.2.2; 1.2.4;
Correct copyright/fix comments.
 1.1 11-Sep-2012  matt branches: 1.1.2;
Add 64bit atomic ops for ARMv6+ (using ldrexd/strexd).
 1.1.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.2.4.2 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.1 13-Sep-2012  matt file atomic_and_64.S was added on branch matt-nb6-plus on 2012-11-27 23:42:35 +0000
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 30-Oct-2012  yamt sync with head
 1.2.2.1 13-Sep-2012  yamt file atomic_and_64.S was added on branch yamt-pagecache on 2012-10-30 18:46:10 +0000
 1.9.2.2 15-Feb-2014  matt Add 64-bit atomic ops if ARMv6+
 1.9.2.1 30-Nov-2013  matt file atomic_and_64.S was added on branch matt-nb5-mips64 on 2014-02-15 10:27:44 +0000
 1.10.28.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.10.28.1 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1313):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.12

Whitespace
 1.10.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.10.26.1 21-Apr-2020  martin Sync with HEAD
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26; 1.3.28;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 04-Mar-2014  matt branches: 1.2.2; 1.2.4;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_and_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_and_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.3 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.2 04-Mar-2014  matt branches: 1.2.4; 1.2.8; 1.2.30; 1.2.32;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.32.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.2.30.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.30.1 21-Apr-2020  martin Sync with HEAD
 1.2.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.1 04-Mar-2014  tls file atomic_cas_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_cas_16.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.8 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.7 04-Mar-2014  matt branches: 1.7.26; 1.7.28;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.6 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.5 10-Aug-2013  matt Make these under Thumb2
 1.4 29-Oct-2012  chs _ARCH_ARM_6 -> _ARM_ARCH_6
 1.3 31-Aug-2012  matt branches: 1.3.2;
Add dmb/dsb instructions as required by the armv7 arch man.
 1.2 16-Aug-2008  matt branches: 1.2.2; 1.2.4; 1.2.12; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file atomic_cas_32.S was initially added on branch thorpej-atomic.
 1.1.2.5 29-Aug-2007  matt Typo. cmpeq should just be cmp
 1.1.2.4 27-Aug-2007  matt Don't use r4 (since it's caller saved). Use ip (it's available since we
are leaf functions).
 1.1.2.3 26-Aug-2007  matt Add armv6 variants of atomic ops which use ldrex/strex.
Add atomic_swap (all arm architectures) which uses swp and swpb.
Add membar_ops which uses armv6 data {sync, memory} barrier.
XXX let thorpej figure out the Makefile magic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 18-Apr-2007  thorpej Atomic ops implementation for ARM. Everything is built on top of
atomic_cas_32(), which is itself a restartable atomic sequence.
 1.2.24.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.12.1 19-Dec-2013  matt Enable ldrex/strex based atomic ops on armv6/armv7
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Oct-2012  yamt sync with head
 1.2.2.2 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.2.2.1 16-Aug-2008  wrstuden file atomic_cas_32.S was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000
 1.3.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.7.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.12 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.11 18-Feb-2019  martin branches: 1.11.2;
Add some atomic_cas_64_ni aliases
 1.10 05-Mar-2014  matt branches: 1.10.26;
apcs-gnu only passes one register on the stack.
ldrd always loads little endian (low address, low register).
 1.9 04-Mar-2014  matt Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.8 04-Mar-2014  matt Fix #if/#endif nesting
 1.7 04-Mar-2014  matt Fix non-EABI loading of argument. Deal with endian issues.
Fixes PR/48635
 1.6 04-Mar-2014  matt Fetch value from correct stack location. Push an even number of registers
so ldrd won't fail.
 1.5 30-Nov-2013  joerg branches: 1.5.2;
Use explicit form of register pair operations by specifying both.
 1.4 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.3 10-Aug-2013  matt Make these under Thumb2
 1.2 29-Oct-2012  chs branches: 1.2.2; 1.2.4;
_ARCH_ARM_6 -> _ARM_ARCH_6
 1.1 11-Sep-2012  matt branches: 1.1.2;
Add 64bit atomic ops for ARMv6+ (using ldrexd/strexd).
 1.1.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.2.4.2 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.1 29-Oct-2012  matt file atomic_cas_64.S was added on branch matt-nb6-plus on 2012-11-27 23:42:35 +0000
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 30-Oct-2012  yamt sync with head
 1.2.2.1 29-Oct-2012  yamt file atomic_cas_64.S was added on branch yamt-pagecache on 2012-10-30 18:46:10 +0000
 1.5.2.2 15-Feb-2014  matt Add 64-bit atomic ops if ARMv6+
 1.5.2.1 30-Nov-2013  matt file atomic_cas_64.S was added on branch matt-nb5-mips64 on 2014-02-15 10:27:44 +0000
 1.10.26.2 21-Apr-2020  martin Sync with HEAD
 1.10.26.1 10-Jun-2019  christos Sync with HEAD
 1.11.2.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.9 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.8 29-Jun-2021  skrll Whitespace
 1.7 04-Mar-2014  matt branches: 1.7.26; 1.7.28;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.6 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.5 10-Aug-2013  matt Make these under Thumb2
 1.4 29-Oct-2012  chs _ARCH_ARM_6 -> _ARM_ARCH_6
 1.3 31-Aug-2012  matt branches: 1.3.2;
Add dmb/dsb instructions as required by the armv7 arch man.
 1.2 16-Aug-2012  matt Actually use the assembly version of the atomic function if compiling
for armv6 or armv7 cpus. Use atomic_cas_ptr instead of _lock_cas so
we pick up the assembly version when it's used.
 1.1 18-Nov-2008  matt branches: 1.1.8; 1.1.12;
Add an atomic_cas_8 which uses ldrex/strex.
 1.1.12.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.1.8.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.8.1 30-Oct-2012  yamt sync with head
 1.3.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.7.28.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.7.28.1 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1313):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.12

Whitespace
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.8 09-Mar-2020  skrll Give the thumb atomic ops a chance of working
 1.7 04-Mar-2014  matt branches: 1.7.26; 1.7.28;
Load new value from correct stack location in _atomic_cas_64_up
 1.6 27-Feb-2014  matt Add atomic_cas_64 support for ARM EABI on V5TE and V5TEJ cpus.
(strd is atomic).
 1.5 27-Jan-2014  matt Add _atomic_cas_16_up and _atomic_cas_8_up
 1.4 19-Aug-2013  matt Thumbify
 1.3 10-Aug-2013  matt Make these under Thumb2
 1.2 25-May-2008  chs branches: 1.2.2; 1.2.24;
enable profiling of assembly functions.
 1.1 29-Apr-2008  scw branches: 1.1.2; 1.1.4;
Implement _atomic_cas_up() in assembly code as the compiler cannot be
trusted to generate fully restartable code sequences.

Addresses lib/38482 for ARM and m68000.
 1.1.4.3 04-Jun-2008  yamt sync with head
 1.1.4.2 18-May-2008  yamt sync with head.
 1.1.4.1 29-Apr-2008  yamt file atomic_cas_up.S was added on branch yamt-pf42 on 2008-05-18 12:28:44 +0000
 1.1.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.2.24.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.7.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.7.26.2 21-Apr-2020  martin Sync with HEAD
 1.7.26.1 08-Apr-2020  martin Merge changes from current as of 20200406
 1.7 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.6 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.5 11-Aug-2013  matt branches: 1.5.26; 1.5.28;
Use foo{s} since it reduces the ifdefs for thumb
 1.4 10-Aug-2013  matt Make these under Thumb2
 1.3 31-Aug-2012  matt branches: 1.3.2;
Add dmb/dsb instructions as required by the armv7 arch man.
 1.2 16-Aug-2008  matt branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex
 1.1 26-Aug-2007  matt branches: 1.1.2;
file atomic_dec_32.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Aug-2007  matt Add armv6 variants of atomic ops which use ldrex/strex.
Add atomic_swap (all arm architectures) which uses swp and swpb.
Add membar_ops which uses armv6 data {sync, memory} barrier.
XXX let thorpej figure out the Makefile magic.
 1.2.24.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Oct-2012  yamt sync with head
 1.2.2.2 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.2.2.1 16-Aug-2008  wrstuden file atomic_dec_32.S was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000
 1.3.2.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.9 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.8 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.7 30-Nov-2013  joerg branches: 1.7.2; 1.7.28; 1.7.30;
Use explicit form of register pair operations by specifying both.
 1.6 11-Aug-2013  matt Use foo{s} since it reduces the ifdefs for thumb
 1.5 10-Aug-2013  matt Make these under Thumb2
 1.4 10-Aug-2013  matt Use r2 instead of ip
 1.3 10-Aug-2013  matt Use subs/sbc
Use push/pop
Don't adjust return for atomic_dec_64 since it's void and returns nothing
 1.2 13-Sep-2012  matt branches: 1.2.2; 1.2.4;
Correct copyright/fix comments.
 1.1 11-Sep-2012  matt branches: 1.1.2;
Add 64bit atomic ops for ARMv6+ (using ldrexd/strexd).
 1.1.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.2.4.2 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.1 13-Sep-2012  matt file atomic_dec_64.S was added on branch matt-nb6-plus on 2012-11-27 23:42:35 +0000
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 30-Oct-2012  yamt sync with head
 1.2.2.1 13-Sep-2012  yamt file atomic_dec_64.S was added on branch yamt-pagecache on 2012-10-30 18:46:11 +0000
 1.7.30.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.7.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.28.1 21-Apr-2020  martin Sync with HEAD
 1.7.2.2 15-Feb-2014  matt Add 64-bit atomic ops if ARMv6+
 1.7.2.1 30-Nov-2013  matt file atomic_dec_64.S was added on branch matt-nb5-mips64 on 2014-02-15 10:27:44 +0000
 1.9 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.8 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.7 08-Nov-2013  matt branches: 1.7.26; 1.7.28;
Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.6 11-Aug-2013  matt Use foo{s} since it reduces the ifdefs for thumb
 1.5 10-Aug-2013  matt Make these under Thumb2
 1.4 29-Oct-2012  chs _ARCH_ARM_6 -> _ARM_ARCH_6
 1.3 31-Aug-2012  matt branches: 1.3.2;
Add dmb/dsb instructions as required by the armv7 arch man.
 1.2 16-Aug-2008  matt branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex
 1.1 26-Aug-2007  matt branches: 1.1.2;
file atomic_inc_32.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Aug-2007  matt Add armv6 variants of atomic ops which use ldrex/strex.
Add atomic_swap (all arm architectures) which uses swp and swpb.
Add membar_ops which uses armv6 data {sync, memory} barrier.
XXX let thorpej figure out the Makefile magic.
 1.2.24.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Oct-2012  yamt sync with head
 1.2.2.2 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.2.2.1 16-Aug-2008  wrstuden file atomic_inc_32.S was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000
 1.3.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.7.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.11 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.10 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.9 30-Nov-2013  joerg branches: 1.9.2; 1.9.28; 1.9.30;
Use explicit form of register pair operations by specifying both.
 1.8 11-Aug-2013  matt Use foo{s} since it reduces the ifdefs for thumb
 1.7 10-Aug-2013  matt Make these under Thumb2
 1.6 10-Aug-2013  matt Use r2 instead of ip
 1.5 10-Aug-2013  matt Don't adjust return of atomic_inc_64 since it's void and doesn't return
anything.
 1.4 10-Aug-2013  matt Fix add -> adds
 1.3 29-Oct-2012  chs branches: 1.3.2; 1.3.4;
_ARCH_ARM_6 -> _ARM_ARCH_6
 1.2 13-Sep-2012  matt Correct copyright/fix comments.
 1.1 11-Sep-2012  matt branches: 1.1.2;
Add 64bit atomic ops for ARMv6+ (using ldrexd/strexd).
 1.1.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.3.4.2 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.3.4.1 29-Oct-2012  matt file atomic_inc_64.S was added on branch matt-nb6-plus on 2012-11-27 23:42:35 +0000
 1.3.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.2.2 30-Oct-2012  yamt sync with head
 1.3.2.1 29-Oct-2012  yamt file atomic_inc_64.S was added on branch yamt-pagecache on 2012-10-30 18:46:11 +0000
 1.9.30.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.9.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.9.28.1 21-Apr-2020  martin Sync with HEAD
 1.9.2.2 15-Feb-2014  matt Add 64-bit atomic ops if ARMv6+
 1.9.2.1 30-Nov-2013  matt file atomic_inc_64.S was added on branch matt-nb5-mips64 on 2014-02-15 10:27:44 +0000
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 11-Dec-2015  skrll branches: 1.3.16; 1.3.18;
Use gcc 4.4 and later operation for nand, i.e.
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value

There was also another bug in sync_fetch_and_nand_8 which I've also fixed.

PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
 1.2 04-Mar-2014  matt branches: 1.2.4; 1.2.6; 1.2.8;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.1 04-Mar-2014  tls file atomic_nand_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.6.1 26-Feb-2016  snj Pull up following revision(s) (requested by skrll in ticket #1105):
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.5
Use gcc 4.4 and later operation for nand, i.e.
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value
There was also another bug in sync_fetch_and_nand_8 which I've also fixed.
PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_nand_16.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.3.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.16.1 21-Apr-2020  martin Sync with HEAD
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 11-Dec-2015  skrll branches: 1.3.16; 1.3.18;
Use gcc 4.4 and later operation for nand, i.e.
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value

There was also another bug in sync_fetch_and_nand_8 which I've also fixed.

PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
 1.2 04-Mar-2014  matt branches: 1.2.4; 1.2.6; 1.2.8;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.1 04-Mar-2014  tls file atomic_nand_32.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.6.1 26-Feb-2016  snj Pull up following revision(s) (requested by skrll in ticket #1105):
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.5
Use gcc 4.4 and later operation for nand, i.e.
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value
There was also another bug in sync_fetch_and_nand_8 which I've also fixed.
PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_nand_32.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.3.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.16.1 21-Apr-2020  martin Sync with HEAD
 1.7 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.6 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.5 28-Jun-2021  skrll Whitespace
 1.4 11-Dec-2015  skrll branches: 1.4.16; 1.4.18;
Use gcc 4.4 and later operation for nand, i.e.
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value

There was also another bug in sync_fetch_and_nand_8 which I've also fixed.

PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
 1.3 04-Mar-2014  matt branches: 1.3.4; 1.3.6; 1.3.8;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.2 30-Nov-2013  joerg Use explicit form of register pair operations by specifying both.
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.3.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.8.1 04-Mar-2014  tls file atomic_nand_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.3.6.1 26-Feb-2016  snj Pull up following revision(s) (requested by skrll in ticket #1105):
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.5
Use gcc 4.4 and later operation for nand, i.e.
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value
There was also another bug in sync_fetch_and_nand_8 which I've also fixed.
PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
 1.3.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.4.1 04-Mar-2014  yamt file atomic_nand_64.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.4.18.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4.18.1 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1313):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.12

Whitespace
 1.4.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.16.1 21-Apr-2020  martin Sync with HEAD
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 11-Dec-2015  skrll branches: 1.3.16; 1.3.18;
Use gcc 4.4 and later operation for nand, i.e.
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value

There was also another bug in sync_fetch_and_nand_8 which I've also fixed.

PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
 1.2 04-Mar-2014  matt branches: 1.2.4; 1.2.6; 1.2.8;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.1 04-Mar-2014  tls file atomic_nand_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.6.1 26-Feb-2016  snj Pull up following revision(s) (requested by skrll in ticket #1105):
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.5
Use gcc 4.4 and later operation for nand, i.e.
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value
There was also another bug in sync_fetch_and_nand_8 which I've also fixed.
PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_nand_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.3.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.16.1 21-Apr-2020  martin Sync with HEAD
 1.10 28-Jul-2021  simonb #define<tab> consistency.
 1.9 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.8 16-Sep-2019  skrll Traiing whitespace.
 1.7 17-May-2015  justin branches: 1.7.16; 1.7.18;
Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.6 04-Mar-2014  matt Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.5 22-Feb-2014  martin Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.4 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.3 11-Sep-2012  matt branches: 1.3.2;
Add 64bit atomic ops for ARMv6+ (using ldrexd/strexd).
 1.2 16-Aug-2008  matt branches: 1.2.2; 1.2.4; 1.2.12; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.
 1.1.2.1 18-Apr-2007  thorpej Atomic ops implementation for ARM. Everything is built on top of
atomic_cas_32(), which is itself a restartable atomic sequence.
 1.2.24.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.12.1 15-Feb-2014  matt Add 64-bit atomic ops if ARMv6+
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Oct-2012  yamt sync with head
 1.2.2.2 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.2.2.1 16-Aug-2008  wrstuden file atomic_op_asm.h was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000
 1.3.2.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.7.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.7.16.2 21-Apr-2020  martin Sync with HEAD
 1.7.16.1 13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26; 1.3.28;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 04-Mar-2014  matt branches: 1.2.2; 1.2.4;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_or_16.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_or_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.10 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.9 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.8 23-Jun-2014  joerg branches: 1.8.24; 1.8.26;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.7 22-Feb-2014  martin branches: 1.7.2;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.6 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.5 11-Aug-2013  matt Use foo{s} since it reduces the ifdefs for thumb
 1.4 10-Aug-2013  matt Make these under Thumb2
 1.3 31-Aug-2012  matt branches: 1.3.2;
Add dmb/dsb instructions as required by the armv7 arch man.
 1.2 16-Aug-2008  matt branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex
 1.1 26-Aug-2007  matt branches: 1.1.2;
file atomic_or_32.S was initially added on branch thorpej-atomic.
 1.1.2.2 27-Aug-2007  matt Don't use r4 (since it's caller saved). Use ip (it's available since we
are leaf functions).
 1.1.2.1 26-Aug-2007  matt Add armv6 variants of atomic ops which use ldrex/strex.
Add atomic_swap (all arm architectures) which uses swp and swpb.
Add membar_ops which uses armv6 data {sync, memory} barrier.
XXX let thorpej figure out the Makefile magic.
 1.2.24.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Oct-2012  yamt sync with head
 1.2.2.2 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.2.2.1 16-Aug-2008  wrstuden file atomic_or_32.S was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000
 1.3.2.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.7.2.1 10-Aug-2014  tls Rebase.
 1.8.26.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.8.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.24.1 21-Apr-2020  martin Sync with HEAD
 1.14 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.13 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.12 28-Jun-2021  skrll Whitespace
 1.11 15-Sep-2019  skrll __sync_{,x}or_and_fetch_8 should return new value... make it so.
 1.10 04-Mar-2014  matt branches: 1.10.18; 1.10.26; 1.10.28;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.9 30-Nov-2013  joerg branches: 1.9.2;
Use explicit form of register pair operations by specifying both.
 1.8 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.7 20-Aug-2013  matt Push two registers to keep stack aligned.
 1.6 11-Aug-2013  matt Use foo{s} since it reduces the ifdefs for thumb
 1.5 10-Aug-2013  matt Make these under Thumb2
 1.4 10-Aug-2013  matt Never modify NLO/NHI (r2,r3) only LO/HI (r0,r1) so subsequent loops will work.
 1.3 10-Aug-2013  matt use push/pop
 1.2 13-Sep-2012  matt branches: 1.2.2; 1.2.4;
Correct copyright/fix comments.
 1.1 11-Sep-2012  matt branches: 1.1.2;
Add 64bit atomic ops for ARMv6+ (using ldrexd/strexd).
 1.1.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.2.4.2 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.1 13-Sep-2012  matt file atomic_or_64.S was added on branch matt-nb6-plus on 2012-11-27 23:42:35 +0000
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 30-Oct-2012  yamt sync with head
 1.2.2.1 13-Sep-2012  yamt file atomic_or_64.S was added on branch yamt-pagecache on 2012-10-30 18:46:11 +0000
 1.9.2.2 15-Feb-2014  matt Add 64-bit atomic ops if ARMv6+
 1.9.2.1 30-Nov-2013  matt file atomic_or_64.S was added on branch matt-nb5-mips64 on 2014-02-15 10:27:44 +0000
 1.10.28.3 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.10.28.2 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1313):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.12

Whitespace
 1.10.28.1 17-Sep-2019  martin Pull up following revision(s) (requested by skrll in ticket #202):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.11

__sync_{,x}or_and_fetch_8 should return new value... make it so.
 1.10.26.2 21-Apr-2020  martin Sync with HEAD
 1.10.26.1 13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.10.18.1 17-Sep-2019  martin Pull up following revision(s) (requested by skrll in ticket #1375):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.11

__sync_{,x}or_and_fetch_8 should return new value... make it so.
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26; 1.3.28;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 04-Mar-2014  matt branches: 1.2.2; 1.2.4;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_or_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_or_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2 16-Aug-2013  matt branches: 1.2.4; 1.2.8; 1.2.30;
Add a hidden version for libpthread.
 1.1 15-Aug-2013  matt When compiling for Thumb1, the swp instruction is not availble. So this
stub is added to provide __cpu_simple_lock and __cpu_simple_lock_try via
thumb interwork support. It is compiled with -mno-thumb.
 1.2.30.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.30.1 21-Apr-2020  martin Sync with HEAD
 1.2.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.1 16-Aug-2013  tls file atomic_simplelock.c was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 16-Aug-2013  yamt file atomic_simplelock.c was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 28-Jun-2021  skrll Whitespace
 1.2 04-Mar-2014  matt branches: 1.2.4; 1.2.8; 1.2.30; 1.2.32;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.32.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.2.32.1 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1313):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.12

Whitespace
 1.2.30.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.30.1 21-Apr-2020  martin Sync with HEAD
 1.2.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.1 04-Mar-2014  tls file atomic_sub_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_sub_64.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.19 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.18 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.17 26-Apr-2021  skrll Add the appropriate memory barrier before the lock is cleared in
__sync_lock_release_{1,2,4,8}. That is, all reads and write for in inner
shareability domain before the lock clear store.
 1.16 24-Apr-2021  skrll Trailing whitespace
 1.15 24-Apr-2021  skrll Fix __sync_lock_release_4 to actually zeroise the whole 4bytes/32bits.
 1.14 17-May-2015  justin branches: 1.14.8; 1.14.16; 1.14.18;
Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.13 17-Apr-2015  skrll Use the right register in previous. Spotted by matt@
 1.12 17-Apr-2015  skrll ARM ARM D7.3.2 - ensure all previous accesses are observed before
the lock is cleared
 1.11 28-Jun-2014  joerg Add aliases for the C11/C++11 spelling of the CAS primitives.
 1.10 04-Mar-2014  matt branches: 1.10.2;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.9 30-Jan-2014  matt switch to unified syntax
 1.8 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.7 11-Aug-2013  matt Use foo{s} since it reduces the ifdefs for thumb
 1.6 10-Aug-2013  matt Make these under Thumb2
 1.5 28-Jan-2013  matt Change movsne to movnes for clang.
 1.4 31-Aug-2012  matt branches: 1.4.2;
Add dmb/dsb instructions as required by the armv7 arch man.
 1.3 16-Aug-2012  matt Actually use the assembly version of the atomic function if compiling
for armv6 or armv7 cpus. Use atomic_cas_ptr instead of _lock_cas so
we pick up the assembly version when it's used.
 1.2 16-Aug-2008  matt branches: 1.2.2; 1.2.4; 1.2.12; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex
 1.1 26-Aug-2007  matt branches: 1.1.2;
file atomic_swap.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Aug-2007  matt Add armv6 variants of atomic ops which use ldrex/strex.
Add atomic_swap (all arm architectures) which uses swp and swpb.
Add membar_ops which uses armv6 data {sync, memory} barrier.
XXX let thorpej figure out the Makefile magic.
 1.2.24.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.12.1 24-Mar-2014  matt Use ldrex/strex instead of swp when possible.
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Oct-2012  yamt sync with head
 1.2.2.2 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.2.2.1 16-Aug-2008  wrstuden file atomic_swap.S was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000
 1.4.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.2.1 25-Feb-2013  tls resync with head
 1.10.2.1 10-Aug-2014  tls Rebase.
 1.14.18.3 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.14.18.2 30-Apr-2021  martin Pull up following revision(s) (requested by skrll in ticket #1261):

sys/arch/arm/include/lock.h: revision 1.38
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.17

Add the appropriate memory barrier before the lock is cleared in
__sync_lock_release_{1,2,4,8}. That is, all reads and write for the
inner shareability domain before the lock clear store.

Improve the membar_ops barriers - no need to use dsb and wait for
completion. Also, we only to act on the inner shareability domain.

Fix the barrier confusion. From Riastradh - thanks!.
 1.14.18.1 26-Apr-2021  martin Pull up following revision(s) (requested by skrll in ticket #1254):

sys/arch/arm/include/lock.h: revision 1.37
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.15
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.16
sys/arch/arm/include/lock.h: revision 1.36

Fix ARMv8 instructions

Fix __sync_lock_release_4 to actually zeroise the whole 4bytes/32bits.

Trailing whitespace

Change #ifdef FOO to #if defined(FOO). NFCI.
 1.14.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.14.16.1 21-Apr-2020  martin Sync with HEAD
 1.14.8.1 26-Apr-2021  martin Pull up following revision(s) (requested by skrll in ticket #1672):

sys/arch/arm/include/lock.h: revision 1.37
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.15
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.16
sys/arch/arm/include/lock.h: revision 1.36

Fix ARMv8 instructions

Fix __sync_lock_release_4 to actually zeroise the whole 4bytes/32bits.

Trailing whitespace

Change #ifdef FOO to #if defined(FOO). NFCI.
 1.7 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.6 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.5 26-Apr-2021  skrll Add the appropriate memory barrier before the lock is cleared in
__sync_lock_release_{1,2,4,8}. That is, all reads and write for in inner
shareability domain before the lock clear store.
 1.4 17-May-2015  justin branches: 1.4.16; 1.4.18;
Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.3 28-Jun-2014  joerg branches: 1.3.4;
Add aliases for the C11/C++11 spelling of the CAS primitives.
 1.2 04-Mar-2014  matt branches: 1.2.2; 1.2.4;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_swap_16.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 28-Jun-2014  tls file atomic_swap_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.4.18.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4.18.1 30-Apr-2021  martin Pull up following revision(s) (requested by skrll in ticket #1261):

sys/arch/arm/include/lock.h: revision 1.38
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.17

Add the appropriate memory barrier before the lock is cleared in
__sync_lock_release_{1,2,4,8}. That is, all reads and write for the
inner shareability domain before the lock clear store.

Improve the membar_ops barriers - no need to use dsb and wait for
completion. Also, we only to act on the inner shareability domain.

Fix the barrier confusion. From Riastradh - thanks!.
 1.4.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.16.1 21-Apr-2020  martin Sync with HEAD
 1.15 01-Aug-2021  andvar s/overwriten/overwritten/ in comments.
 1.14 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.13 28-Jun-2021  skrll Whitespace
 1.12 26-Apr-2021  skrll Add the appropriate memory barrier before the lock is cleared in
__sync_lock_release_{1,2,4,8}. That is, all reads and write for in inner
shareability domain before the lock clear store.
 1.11 24-Apr-2021  skrll Trailing whitespace
 1.10 17-May-2015  justin branches: 1.10.8; 1.10.16; 1.10.18;
Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.9 28-Jun-2014  joerg Add aliases for the C11/C++11 spelling of the CAS primitives.
 1.8 04-Mar-2014  matt branches: 1.8.2;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.7 30-Nov-2013  joerg Use explicit form of register pair operations by specifying both.
 1.6 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.5 20-Aug-2013  matt Push two registers to keep stack aligned.
 1.4 10-Aug-2013  matt Make these under Thumb2
 1.3 10-Aug-2013  matt cmpne -> cmp
Use push/pop
 1.2 13-Sep-2012  matt branches: 1.2.2; 1.2.4;
Correct copyright/fix comments.
 1.1 11-Sep-2012  matt branches: 1.1.2;
Add 64bit atomic ops for ARMv6+ (using ldrexd/strexd).
 1.1.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.2.4.2 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.1 13-Sep-2012  matt file atomic_swap_64.S was added on branch matt-nb6-plus on 2012-11-27 23:42:35 +0000
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 30-Oct-2012  yamt sync with head
 1.2.2.1 13-Sep-2012  yamt file atomic_swap_64.S was added on branch yamt-pagecache on 2012-10-30 18:46:12 +0000
 1.8.2.1 10-Aug-2014  tls Rebase.
 1.10.18.4 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.10.18.3 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1313):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.12

Whitespace
 1.10.18.2 30-Apr-2021  martin Pull up following revision(s) (requested by skrll in ticket #1261):

sys/arch/arm/include/lock.h: revision 1.38
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.17

Add the appropriate memory barrier before the lock is cleared in
__sync_lock_release_{1,2,4,8}. That is, all reads and write for the
inner shareability domain before the lock clear store.

Improve the membar_ops barriers - no need to use dsb and wait for
completion. Also, we only to act on the inner shareability domain.

Fix the barrier confusion. From Riastradh - thanks!.
 1.10.18.1 26-Apr-2021  martin Pull up following revision(s) (requested by skrll in ticket #1254):

sys/arch/arm/include/lock.h: revision 1.37
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.15
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.16
sys/arch/arm/include/lock.h: revision 1.36

Fix ARMv8 instructions

Fix __sync_lock_release_4 to actually zeroise the whole 4bytes/32bits.

Trailing whitespace

Change #ifdef FOO to #if defined(FOO). NFCI.
 1.10.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.10.16.1 21-Apr-2020  martin Sync with HEAD
 1.10.8.1 26-Apr-2021  martin Pull up following revision(s) (requested by skrll in ticket #1672):

sys/arch/arm/include/lock.h: revision 1.37
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.15
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.16
sys/arch/arm/include/lock.h: revision 1.36

Fix ARMv8 instructions

Fix __sync_lock_release_4 to actually zeroise the whole 4bytes/32bits.

Trailing whitespace

Change #ifdef FOO to #if defined(FOO). NFCI.
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26; 1.3.28;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 04-Mar-2014  matt branches: 1.2.2; 1.2.4;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_xor_16.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_xor_16.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26; 1.3.28;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 04-Mar-2014  matt branches: 1.2.2; 1.2.4;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_xor_32.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_xor_32.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.7 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.6 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.5 28-Jun-2021  skrll Whitespace
 1.4 15-Sep-2019  skrll __sync_{,x}or_and_fetch_8 should return new value... make it so.
 1.3 04-Mar-2014  matt branches: 1.3.4; 1.3.8; 1.3.22; 1.3.30; 1.3.32;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.2 30-Nov-2013  joerg Use explicit form of register pair operations by specifying both.
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.3.32.3 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.32.2 06-Jul-2021  martin Pull up following revision(s) (requested by skrll in ticket #1313):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.12

Whitespace
 1.3.32.1 17-Sep-2019  martin Pull up following revision(s) (requested by skrll in ticket #202):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.11

__sync_{,x}or_and_fetch_8 should return new value... make it so.
 1.3.30.2 21-Apr-2020  martin Sync with HEAD
 1.3.30.1 13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.3.22.1 17-Sep-2019  martin Pull up following revision(s) (requested by skrll in ticket #1375):

common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.11

__sync_{,x}or_and_fetch_8 should return new value... make it so.
 1.3.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.8.1 04-Mar-2014  tls file atomic_xor_64.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.3.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.4.1 04-Mar-2014  yamt file atomic_xor_64.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26; 1.3.28;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 04-Mar-2014  matt branches: 1.2.2; 1.2.4;
Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 04-Mar-2014  yamt file atomic_xor_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.28.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_xor_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.10 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.9 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.8 10-Jul-2021  skrll s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.
 1.7 27-Apr-2021  skrll Improve the membar_ops barriers - no need to use dsb and wait for
completion. Also, we only to act on the inner shareability domain.
 1.6 28-Mar-2014  skrll branches: 1.6.26; 1.6.28;
Ensure SBZ register is zero
 1.5 04-Mar-2014  matt Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)
 1.4 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.3 16-Aug-2012  matt branches: 1.3.2;
Actually use the assembly version of the atomic function if compiling
for armv6 or armv7 cpus. Use atomic_cas_ptr instead of _lock_cas so
we pick up the assembly version when it's used.
 1.2 16-Aug-2008  matt branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex
 1.1 26-Aug-2007  matt branches: 1.1.2;
file membar_ops.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Aug-2007  matt Add armv6 variants of atomic ops which use ldrex/strex.
Add atomic_swap (all arm architectures) which uses swp and swpb.
Add membar_ops which uses armv6 data {sync, memory} barrier.
XXX let thorpej figure out the Makefile magic.
 1.2.24.1 27-Nov-2012  matt Pull atomic ops from HEAD.
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Oct-2012  yamt sync with head
 1.2.2.2 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.2.2.1 16-Aug-2008  wrstuden file membar_ops.S was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000
 1.3.2.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.28.2 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.6.28.1 30-Apr-2021  martin Pull up following revision(s) (requested by skrll in ticket #1261):

sys/arch/arm/include/lock.h: revision 1.38
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.17

Add the appropriate memory barrier before the lock is cleared in
__sync_lock_release_{1,2,4,8}. That is, all reads and write for the
inner shareability domain before the lock clear store.

Improve the membar_ops barriers - no need to use dsb and wait for
completion. Also, we only to act on the inner shareability domain.

Fix the barrier confusion. From Riastradh - thanks!.
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3 17-May-2015  justin branches: 1.3.16; 1.3.18;
Move arm sync_* changes to Makefile.inc
 1.2 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.1 08-Nov-2013  matt branches: 1.1.4; 1.1.8;
Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.1.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.8.1 08-Nov-2013  tls file sync_bool_compare_and_swap_1.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.1.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.4.1 08-Nov-2013  yamt file sync_bool_compare_and_swap_1.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.3.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.16.1 21-Apr-2020  martin Sync with HEAD
 1.4 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3 17-May-2015  justin branches: 1.3.16; 1.3.18;
Move arm sync_* changes to Makefile.inc
 1.2 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.1 08-Nov-2013  matt branches: 1.1.4; 1.1.8;
Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.1.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.8.1 08-Nov-2013  tls file sync_bool_compare_and_swap_2.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.1.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.4.1 08-Nov-2013  yamt file sync_bool_compare_and_swap_2.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.3.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.16.1 21-Apr-2020  martin Sync with HEAD
 1.4 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3 17-May-2015  justin branches: 1.3.16; 1.3.18;
Move arm sync_* changes to Makefile.inc
 1.2 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.1 08-Nov-2013  matt branches: 1.1.4; 1.1.8;
Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.1.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.8.1 08-Nov-2013  tls file sync_bool_compare_and_swap_4.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.1.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.4.1 08-Nov-2013  yamt file sync_bool_compare_and_swap_4.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.3.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.3.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.16.1 21-Apr-2020  martin Sync with HEAD
 1.5 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4 17-May-2015  justin branches: 1.4.16; 1.4.18;
Move arm sync_* changes to Makefile.inc
 1.3 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.2 30-Nov-2013  joerg branches: 1.2.4; 1.2.8;
Use explicit form of register pair operations by specifying both.
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.1 30-Nov-2013  tls file sync_bool_compare_and_swap_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Nov-2013  yamt file sync_bool_compare_and_swap_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.4.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.4.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.16.1 21-Apr-2020  martin Sync with HEAD
 1.6 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5 17-May-2015  justin branches: 1.5.16; 1.5.18;
Move arm sync_* changes to Makefile.inc
 1.4 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.3 23-Jun-2014  joerg branches: 1.3.4;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 30-Nov-2013  joerg branches: 1.2.2; 1.2.4;
Use explicit form of register pair operations by specifying both.
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Nov-2013  yamt file sync_fetch_and_add_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file sync_fetch_and_add_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.6 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5 17-May-2015  justin branches: 1.5.16; 1.5.18;
Move arm sync_* changes to Makefile.inc
 1.4 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.3 23-Jun-2014  joerg branches: 1.3.4;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 30-Nov-2013  joerg branches: 1.2.2; 1.2.4;
Use explicit form of register pair operations by specifying both.
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Nov-2013  yamt file sync_fetch_and_and_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file sync_fetch_and_and_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.6 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5 11-Dec-2015  skrll branches: 1.5.16; 1.5.18;
Use gcc 4.4 and later operation for nand, i.e.
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value

There was also another bug in sync_fetch_and_nand_8 which I've also fixed.

PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
 1.4 17-May-2015  justin Move arm sync_* changes to Makefile.inc
 1.3 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.2 30-Nov-2013  joerg branches: 1.2.4; 1.2.6; 1.2.8;
Use explicit form of register pair operations by specifying both.
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.1 30-Nov-2013  tls file sync_fetch_and_nand_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.2.6.1 26-Feb-2016  snj Pull up following revision(s) (requested by skrll in ticket #1105):
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.3
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.5
Use gcc 4.4 and later operation for nand, i.e.
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value
There was also another bug in sync_fetch_and_nand_8 which I've also fixed.
PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Nov-2013  yamt file sync_fetch_and_nand_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.5.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.6 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5 17-May-2015  justin branches: 1.5.16; 1.5.18;
Move arm sync_* changes to Makefile.inc
 1.4 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.3 23-Jun-2014  joerg branches: 1.3.4;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 30-Nov-2013  joerg branches: 1.2.2; 1.2.4;
Use explicit form of register pair operations by specifying both.
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Nov-2013  yamt file sync_fetch_and_or_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file sync_fetch_and_or_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.6 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5 17-May-2015  justin branches: 1.5.16; 1.5.18;
Move arm sync_* changes to Makefile.inc
 1.4 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.3 23-Jun-2014  joerg branches: 1.3.4;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 30-Nov-2013  joerg branches: 1.2.2; 1.2.4;
Use explicit form of register pair operations by specifying both.
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Nov-2013  yamt file sync_fetch_and_sub_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file sync_fetch_and_sub_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.6 28-Jul-2021  skrll Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5 17-May-2015  justin branches: 1.5.16; 1.5.18;
Move arm sync_* changes to Makefile.inc
 1.4 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.3 23-Jun-2014  joerg branches: 1.3.4;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 30-Nov-2013  joerg branches: 1.2.2; 1.2.4;
Use explicit form of register pair operations by specifying both.
 1.1 08-Nov-2013  matt Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 30-Nov-2013  yamt file sync_fetch_and_xor_8.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file sync_fetch_and_xor_8.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.5.18.1 11-Aug-2021  martin Pull up following revision(s) (requested by skrll in ticket #1330):

common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.18
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap.S: revision 1.19
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_inc_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_cas_up.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.9
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.8
common/lib/libc/arch/arm/atomic/membar_ops.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_op_asm.h: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_cas_16.S: revision 1.3
common/lib/libc/arch/arm/atomic/sync_fetch_and_and_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_xor_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_64.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_cas_32.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_xor_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_swap_16.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/sync_fetch_and_add_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.8
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/sync_fetch_and_sub_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_64.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_add_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_xor_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_add_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_or_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_nand_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_16.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_or_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_nand_32.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_cas_8.S: revision 1.9
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_inc_64.S: revision 1.11
common/lib/libc/arch/arm/atomic/atomic_cas_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.6
common/lib/libc/arch/arm/atomic/atomic_or_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_dec_32.S: revision 1.7
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_sub_64.S: revision 1.5
common/lib/libc/arch/arm/atomic/sync_fetch_and_xor_8.S: revision 1.6
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_4.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.12
common/lib/libc/arch/arm/atomic/atomic_or_32.S: revision 1.10
common/lib/libc/arch/arm/atomic/atomic_and_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_2.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_swap_64.S: revision 1.14
common/lib/libc/arch/arm/atomic/sync_bool_compare_and_swap_1.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.13
common/lib/libc/arch/arm/atomic/atomic_nand_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_8.S: revision 1.5
common/lib/libc/arch/arm/atomic/atomic_and_16.S: revision 1.4
common/lib/libc/arch/arm/atomic/atomic_add_64.S: revision 1.14

Give the thumb atomic ops a chance of working

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.

That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)
This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.3 29-Jan-2014  joerg branches: 1.3.4; 1.3.28;
Kernel and standalone code uses panic() for broken code.
 1.2 08-May-2013  matt When using EABI, call __aeabi_{i,l}div0 when diving by 0.
 1.1 23-Jan-2013  matt branches: 1.1.4;
Add EABI support routines to raise SIGFPE on integer divide by 0.
 1.1.4.4 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.3 23-Jun-2013  tls resync from head
 1.1.4.2 25-Feb-2013  tls resync with head
 1.1.4.1 23-Jan-2013  tls file __aeabi_idiv0.c was added on branch tls-maxphys on 2013-02-25 00:23:55 +0000
 1.3.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.28.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.4.1 29-Jan-2014  yamt file __aeabi_idiv0.c was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.3 29-Jan-2014  joerg branches: 1.3.4; 1.3.28;
Kernel and standalone code uses panic() for broken code.
 1.2 08-May-2013  matt When using EABI, call __aeabi_{i,l}div0 when diving by 0.
 1.1 23-Jan-2013  matt branches: 1.1.4;
Add EABI support routines to raise SIGFPE on integer divide by 0.
 1.1.4.4 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.3 23-Jun-2013  tls resync from head
 1.1.4.2 25-Feb-2013  tls resync with head
 1.1.4.1 23-Jan-2013  tls file __aeabi_ldiv0.c was added on branch tls-maxphys on 2013-02-25 00:23:55 +0000
 1.3.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.28.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.4.1 29-Jan-2014  yamt file __aeabi_ldiv0.c was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.6 19-Aug-2013  matt branches: 1.6.26;
Use STRONG_ALIAS
Add thumb variation
 1.5 27-Nov-2012  matt Use the armv6 rev/rev16 if armv6 or later
 1.4 28-Apr-2008  martin branches: 1.4.4; 1.4.24; 1.4.26;
Remove clause 3 and 4 from TNF licenses
 1.3 04-Feb-2006  uwe branches: 1.3.18; 1.3.26;
Define htonl &co as alternative names to bswap only if little endian.
I'm not sure if idempotent versions need to be defined for big endian,
as the man page says in that case they are defined idempotent macros.

PR port-arm/32734
 1.2 04-Feb-2006  uwe libc wants __bswapNN, kernel wants bswapNN. That was not accounted
for during the merge of kernel and libc versions. Fix to match
e.g. i386 code.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.3.26.1 18-May-2008  yamt sync with head.
 1.3.18.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.4.26.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.26.1 25-Feb-2013  tls resync with head
 1.4.24.1 28-Nov-2012  matt Use rev/rev16 on armv6 and later.
 1.4.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.4.1 16-Jan-2013  yamt sync with (a bit old) head
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.9 11-Dec-2020  dholland arm bswap32: fix fatal typo in thumb code (PR 55854)
 1.8 09-Dec-2020  dholland arm bswap32: Improve the comments showing the byte flow.

It's confusing to use 1-4 for bytes 1-4 and then 0 for literal zero,
so use a-d for bytes 1-4.
 1.7 19-Aug-2013  matt branches: 1.7.26;
Add thumb version
Use STRONG_ALIAS
 1.6 16-Jan-2013  matt Add some comments to illustrate what is actually happening.
 1.5 27-Nov-2012  matt Use the armv6 rev/rev16 if armv6 or later
 1.4 28-Apr-2008  martin branches: 1.4.4; 1.4.24; 1.4.26;
Remove clause 3 and 4 from TNF licenses
 1.3 04-Feb-2006  uwe branches: 1.3.18; 1.3.26;
Define htonl &co as alternative names to bswap only if little endian.
I'm not sure if idempotent versions need to be defined for big endian,
as the man page says in that case they are defined idempotent macros.

PR port-arm/32734
 1.2 04-Feb-2006  uwe libc wants __bswapNN, kernel wants bswapNN. That was not accounted
for during the merge of kernel and libc versions. Fix to match
e.g. i386 code.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.3.26.1 18-May-2008  yamt sync with head.
 1.3.18.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.4.26.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.26.1 25-Feb-2013  tls resync with head
 1.4.24.1 28-Nov-2012  matt Use rev/rev16 on armv6 and later.
 1.4.4.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.4.2 23-Jan-2013  yamt sync with head
 1.4.4.1 16-Jan-2013  yamt sync with (a bit old) head
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.5 29-Jan-2014  joerg Include compiler-rt in libc, libm and libkern.
 1.4 19-Aug-2013  matt This is ARM only
 1.3 20-Jun-2013  matt Add support for __ARM_ARCH_EXT_IDIV__ which is set for those cores
with hardware divide instructions. Note that gcc 4.5.x doesn't support
this so this is just latent. gcc 4.7.x does.
 1.2 08-May-2013  matt When using EABI, call __aeabi_{i,l}div0 when diving by 0.
 1.1 30-Oct-2012  christos branches: 1.1.2; 1.1.4; 1.1.6; 1.1.8;
split udivsi3 and divsi3 to fix static linking. This could be done better.
 1.1.8.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.8.2 16-Jan-2013  yamt sync with (a bit old) head
 1.1.8.1 30-Oct-2012  yamt file divide.S was added on branch yamt-pagecache on 2013-01-16 05:25:53 +0000
 1.1.6.2 28-Nov-2012  matt Merge from HEAD.
split udivsi3 and divsi3 to fix static linking.
 1.1.6.1 30-Oct-2012  matt file divide.S was added on branch matt-nb6-plus on 2012-11-28 01:53:42 +0000
 1.1.4.4 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.3 23-Jun-2013  tls resync from head
 1.1.4.2 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.1.4.1 30-Oct-2012  tls file divide.S was added on branch tls-maxphys on 2012-11-20 02:57:29 +0000
 1.1.2.2 19-Nov-2012  riz Pull up following revision(s) (requested by christos in ticket #654):
common/lib/libc/arch/arm/gen/modsi3.S: revision 1.1
common/lib/libc/arch/arm/gen/umodsi3.S: revision 1.1
sys/lib/libkern/arch/arm/Makefile.inc: revision 1.11
sys/lib/libkern/arch/arm/Makefile.inc: revision 1.12
common/lib/libc/arch/arm/gen/divide.S: revision 1.1
lib/libc/arch/arm/gen/Makefile.inc: revision 1.20
common/lib/libc/arch/arm/gen/divsi3.S: revision 1.3
common/lib/libc/arch/arm/gen/udivsi3.S: revision 1.1
common/lib/libc/arch/arm/gen/divsi3.S: revision 1.4
Split out modsi3 and umodsi3 from the divsi3 file. This is so that
we don't get re-defined symbols in libc from libgcc in static linking.
Example: cc -pthread -static main-calls-pthread-create.c
add split files.
add new divsi3 related files.
add new files to fix static linking
split udivsi3 and divsi3 to fix static linking. This could be done better.
Fixes PR#47139
 1.1.2.1 30-Oct-2012  riz file divide.S was added on branch netbsd-6 on 2012-11-19 18:38:02 +0000
 1.14 29-Jan-2014  joerg Include compiler-rt in libc, libm and libkern.
 1.13 12-Sep-2013  joerg Pass PICFLAGS down to cc-as-as and use __PIC__ to decide if it is small
vs big PIC mode. Retire -DPIC and -DBIGPIC.
 1.12 09-Sep-2013  matt Remove movw/movt due to linker problems.
Check for 0 divisor and __aeabi_idiv0 if needed (EABI && _LIBC only).
 1.11 09-Sep-2013  matt s/__libc/_libc/
 1.10 08-Sep-2013  matt Support using hwdiv instructions if those are available.
But only for EABI.
 1.9 05-Sep-2013  matt Use __ARM_EABI__ and new __UNWIND_TABLES__ to decide when to use .cfi ops
 1.8 22-Aug-2013  matt Don't include .cfi info if _KERNEL || _STANDALONE
 1.7 19-Aug-2013  matt Thumbify (and use .cfi ops)
 1.6 20-Jun-2013  matt Add support for __ARM_ARCH_EXT_IDIV__ which is set for those cores
with hardware divide instructions. Note that gcc 4.5.x doesn't support
this so this is just latent. gcc 4.7.x does.
 1.5 08-May-2013  matt When using EABI, call __aeabi_{i,l}div0 when diving by 0.
 1.4 30-Oct-2012  christos split udivsi3 and divsi3 to fix static linking. This could be done better.
 1.3 10-Oct-2012  christos Split out modsi3 and umodsi3 from the divsi3 file. This is so that
we don't get re-defined symbols in libc from libgcc in static linking.
Example: cc -pthread -static main-calls-pthread-create.c
 1.2 05-Aug-2012  matt branches: 1.2.2;
For __udivsi3 and __divsi3, add their EABI aliases as alternate entry
points.
 1.1 20-Dec-2005  christos branches: 1.1.18; 1.1.50; 1.1.52; 1.1.54;
Merge libkern + libc common files. As requested by core.
 1.1.54.1 28-Nov-2012  matt Merge from HEAD.
split udivsi3 and divsi3 to fix static linking.
 1.1.52.2 20-Nov-2012  msaitoh The last commit message said rev. 1.4 was included, but it wasn't in divsi3.
Apply rev. 1.4.
 1.1.52.1 19-Nov-2012  riz Pull up following revision(s) (requested by christos in ticket #654):
common/lib/libc/arch/arm/gen/modsi3.S: revision 1.1
common/lib/libc/arch/arm/gen/umodsi3.S: revision 1.1
sys/lib/libkern/arch/arm/Makefile.inc: revision 1.11
sys/lib/libkern/arch/arm/Makefile.inc: revision 1.12
common/lib/libc/arch/arm/gen/divide.S: revision 1.1
lib/libc/arch/arm/gen/Makefile.inc: revision 1.20
common/lib/libc/arch/arm/gen/divsi3.S: revision 1.3
common/lib/libc/arch/arm/gen/udivsi3.S: revision 1.1
common/lib/libc/arch/arm/gen/divsi3.S: revision 1.4
Split out modsi3 and umodsi3 from the divsi3 file. This is so that
we don't get re-defined symbols in libc from libgcc in static linking.
Example: cc -pthread -static main-calls-pthread-create.c
add split files.
add new divsi3 related files.
add new files to fix static linking
split udivsi3 and divsi3 to fix static linking. This could be done better.
Fixes PR#47139
 1.1.50.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.50.2 16-Jan-2013  yamt sync with (a bit old) head
 1.1.50.1 30-Oct-2012  yamt sync with head
 1.1.18.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.2.2.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.2.2 23-Jun-2013  tls resync from head
 1.2.2.1 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.5 29-Jan-2014  joerg Include compiler-rt in libc, libm and libkern.
 1.4 15-Aug-2013  matt Only assemble if !__ARM_EABI__
 1.3 20-Jun-2013  matt Add support for __ARM_ARCH_EXT_IDIV__ which is set for those cores
with hardware divide instructions. Note that gcc 4.5.x doesn't support
this so this is just latent. gcc 4.7.x does.
 1.2 28-Nov-2012  matt branches: 1.2.2;
Optimize.
 1.1 10-Oct-2012  christos branches: 1.1.2; 1.1.4; 1.1.6;
Split out modsi3 and umodsi3 from the divsi3 file. This is so that
we don't get re-defined symbols in libc from libgcc in static linking.
Example: cc -pthread -static main-calls-pthread-create.c
 1.1.6.5 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.6.4 23-Jun-2013  tls resync from head
 1.1.6.3 25-Feb-2013  tls resync with head
 1.1.6.2 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.1.6.1 10-Oct-2012  tls file modsi3.S was added on branch tls-maxphys on 2012-11-20 02:57:29 +0000
 1.1.4.2 19-Nov-2012  riz Pull up following revision(s) (requested by christos in ticket #654):
common/lib/libc/arch/arm/gen/modsi3.S: revision 1.1
common/lib/libc/arch/arm/gen/umodsi3.S: revision 1.1
sys/lib/libkern/arch/arm/Makefile.inc: revision 1.11
sys/lib/libkern/arch/arm/Makefile.inc: revision 1.12
common/lib/libc/arch/arm/gen/divide.S: revision 1.1
lib/libc/arch/arm/gen/Makefile.inc: revision 1.20
common/lib/libc/arch/arm/gen/divsi3.S: revision 1.3
common/lib/libc/arch/arm/gen/udivsi3.S: revision 1.1
common/lib/libc/arch/arm/gen/divsi3.S: revision 1.4
Split out modsi3 and umodsi3 from the divsi3 file. This is so that
we don't get re-defined symbols in libc from libgcc in static linking.
Example: cc -pthread -static main-calls-pthread-create.c
add split files.
add new divsi3 related files.
add new files to fix static linking
split udivsi3 and divsi3 to fix static linking. This could be done better.
Fixes PR#47139
 1.1.4.1 10-Oct-2012  riz file modsi3.S was added on branch netbsd-6 on 2012-11-19 18:38:01 +0000
 1.1.2.4 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.2.3 16-Jan-2013  yamt sync with (a bit old) head
 1.1.2.2 30-Oct-2012  yamt sync with head
 1.1.2.1 10-Oct-2012  yamt file modsi3.S was added on branch yamt-pagecache on 2012-10-30 18:46:12 +0000
 1.2.2.2 28-Nov-2012  matt Merge from HEAD.
split udivsi3 and divsi3 to fix static linking.
 1.2.2.1 28-Nov-2012  matt file modsi3.S was added on branch matt-nb6-plus on 2012-11-28 01:53:42 +0000
 1.2 18-Dec-2012  matt branches: 1.2.2; 1.2.6; 1.2.36;
Don't need to include assym.h
Add a missing comma.
 1.1 17-Dec-2012  matt Add a routine to create an up to an 128 bitmask returned
in VFP/NEON q0 starting at the rightmost bit (bit 0).
 1.2.36.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.36.1 21-Apr-2020  martin Sync with HEAD
 1.2.6.2 25-Feb-2013  tls resync with head
 1.2.6.1 18-Dec-2012  tls file neon_mask.S was added on branch tls-maxphys on 2013-02-25 00:23:55 +0000
 1.2.2.2 23-Jan-2013  yamt sync with head
 1.2.2.1 18-Dec-2012  yamt file neon_mask.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.10 29-Jan-2014  joerg Include compiler-rt in libc, libm and libkern.
 1.9 12-Sep-2013  joerg Pass PICFLAGS down to cc-as-as and use __PIC__ to decide if it is small
vs big PIC mode. Retire -DPIC and -DBIGPIC.
 1.8 09-Sep-2013  matt Remove movw/movt due to linker problems.
Check for 0 divisor and __aeabi_idiv0 if needed (EABI && _LIBC only).
 1.7 09-Sep-2013  matt s/__libc/_libc/
 1.6 08-Sep-2013  matt Support using hwdiv instructions if those are available.
But only for EABI.
 1.5 05-Sep-2013  matt Use __ARM_EABI__ and new __UNWIND_TABLES__ to decide when to use .cfi ops
 1.4 22-Aug-2013  matt Don't include .cfi info if _KERNEL || _STANDALONE
 1.3 19-Aug-2013  matt thumbify
add .cfi ops (for thumb)
 1.2 20-Jun-2013  matt Add support for __ARM_ARCH_EXT_IDIV__ which is set for those cores
with hardware divide instructions. Note that gcc 4.5.x doesn't support
this so this is just latent. gcc 4.7.x does.
 1.1 30-Oct-2012  christos branches: 1.1.2; 1.1.4; 1.1.6; 1.1.8;
split udivsi3 and divsi3 to fix static linking. This could be done better.
 1.1.8.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.8.2 16-Jan-2013  yamt sync with (a bit old) head
 1.1.8.1 30-Oct-2012  yamt file udivsi3.S was added on branch yamt-pagecache on 2013-01-16 05:25:53 +0000
 1.1.6.2 28-Nov-2012  matt Merge from HEAD.
split udivsi3 and divsi3 to fix static linking.
 1.1.6.1 30-Oct-2012  matt file udivsi3.S was added on branch matt-nb6-plus on 2012-11-28 01:53:42 +0000
 1.1.4.4 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.3 23-Jun-2013  tls resync from head
 1.1.4.2 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.1.4.1 30-Oct-2012  tls file udivsi3.S was added on branch tls-maxphys on 2012-11-20 02:57:29 +0000
 1.1.2.2 19-Nov-2012  riz Pull up following revision(s) (requested by christos in ticket #654):
common/lib/libc/arch/arm/gen/modsi3.S: revision 1.1
common/lib/libc/arch/arm/gen/umodsi3.S: revision 1.1
sys/lib/libkern/arch/arm/Makefile.inc: revision 1.11
sys/lib/libkern/arch/arm/Makefile.inc: revision 1.12
common/lib/libc/arch/arm/gen/divide.S: revision 1.1
lib/libc/arch/arm/gen/Makefile.inc: revision 1.20
common/lib/libc/arch/arm/gen/divsi3.S: revision 1.3
common/lib/libc/arch/arm/gen/udivsi3.S: revision 1.1
common/lib/libc/arch/arm/gen/divsi3.S: revision 1.4
Split out modsi3 and umodsi3 from the divsi3 file. This is so that
we don't get re-defined symbols in libc from libgcc in static linking.
Example: cc -pthread -static main-calls-pthread-create.c
add split files.
add new divsi3 related files.
add new files to fix static linking
split udivsi3 and divsi3 to fix static linking. This could be done better.
Fixes PR#47139
 1.1.2.1 30-Oct-2012  riz file udivsi3.S was added on branch netbsd-6 on 2012-11-19 18:38:02 +0000
 1.5 29-Jan-2014  joerg Include compiler-rt in libc, libm and libkern.
 1.4 15-Aug-2013  matt Only assemble if !__ARM_EABI__
 1.3 20-Jun-2013  matt Add support for __ARM_ARCH_EXT_IDIV__ which is set for those cores
with hardware divide instructions. Note that gcc 4.5.x doesn't support
this so this is just latent. gcc 4.7.x does.
 1.2 28-Nov-2012  matt branches: 1.2.2;
Optimize.
 1.1 10-Oct-2012  christos branches: 1.1.2; 1.1.4; 1.1.6;
Split out modsi3 and umodsi3 from the divsi3 file. This is so that
we don't get re-defined symbols in libc from libgcc in static linking.
Example: cc -pthread -static main-calls-pthread-create.c
 1.1.6.5 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.6.4 23-Jun-2013  tls resync from head
 1.1.6.3 25-Feb-2013  tls resync with head
 1.1.6.2 20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.1.6.1 10-Oct-2012  tls file umodsi3.S was added on branch tls-maxphys on 2012-11-20 02:57:29 +0000
 1.1.4.2 19-Nov-2012  riz Pull up following revision(s) (requested by christos in ticket #654):
common/lib/libc/arch/arm/gen/modsi3.S: revision 1.1
common/lib/libc/arch/arm/gen/umodsi3.S: revision 1.1
sys/lib/libkern/arch/arm/Makefile.inc: revision 1.11
sys/lib/libkern/arch/arm/Makefile.inc: revision 1.12
common/lib/libc/arch/arm/gen/divide.S: revision 1.1
lib/libc/arch/arm/gen/Makefile.inc: revision 1.20
common/lib/libc/arch/arm/gen/divsi3.S: revision 1.3
common/lib/libc/arch/arm/gen/udivsi3.S: revision 1.1
common/lib/libc/arch/arm/gen/divsi3.S: revision 1.4
Split out modsi3 and umodsi3 from the divsi3 file. This is so that
we don't get re-defined symbols in libc from libgcc in static linking.
Example: cc -pthread -static main-calls-pthread-create.c
add split files.
add new divsi3 related files.
add new files to fix static linking
split udivsi3 and divsi3 to fix static linking. This could be done better.
Fixes PR#47139
 1.1.4.1 10-Oct-2012  riz file umodsi3.S was added on branch netbsd-6 on 2012-11-19 18:38:02 +0000
 1.1.2.4 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.2.3 16-Jan-2013  yamt sync with (a bit old) head
 1.1.2.2 30-Oct-2012  yamt sync with head
 1.1.2.1 10-Oct-2012  yamt file umodsi3.S was added on branch yamt-pagecache on 2012-10-30 18:46:12 +0000
 1.2.2.2 28-Nov-2012  matt Merge from HEAD.
split udivsi3 and divsi3 to fix static linking.
 1.2.2.1 28-Nov-2012  matt file umodsi3.S was added on branch matt-nb6-plus on 2012-11-28 01:53:42 +0000
 1.2 06-Aug-2012  matt branches: 1.2.4; 1.2.36;
Change include to <arm/aeabi.h>
 1.1 06-Aug-2012  matt Add long long / unsigned long long comparision routines. These differ
from cmpdi2 in they return strcmp like values (-1, 0, 1).
 1.2.36.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.36.1 21-Apr-2020  martin Sync with HEAD
 1.2.4.2 30-Oct-2012  yamt sync with head
 1.2.4.1 06-Aug-2012  yamt file __aeabi_lcmp.c was added on branch yamt-pagecache on 2012-10-30 18:46:12 +0000
 1.13 06-May-2014  joerg branches: 1.13.24;
Make EHABI optional.
 1.12 19-Aug-2013  matt branches: 1.12.2;
Add cfi ops.
Thumbify
 1.11 13-Aug-2013  matt Use mvnge AHI, #0x80000000 instead of mvnge AHI, ALO, lsr #1
 1.10 13-Aug-2013  matt 0x800000000 -> 0x80000000 (one too many zeroes)
 1.9 13-Aug-2013  matt Fix movlt
 1.8 13-Aug-2013  matt andlt -> movlt
 1.7 09-May-2013  matt Don't worry about divide by 0 for kernel or standalone
 1.6 09-May-2013  skrll Don't use old syntax - it breaks the build. hi matt.
 1.5 08-May-2013  matt When using EABI, call __aeabi_{i,l}div0 when diving by 0.
 1.4 05-May-2013  skrll Whitespace.
 1.3 19-Apr-2013  skrll Fix logic inversion.
 1.2 14-Apr-2013  matt Fix calling of __qdivrem which the 3rd arg needs to passed on the stack.
 1.1 05-Aug-2012  matt branches: 1.1.2; 1.1.4;
Add another ARM EABI runtime routine for
combined quotient / remainder for signed long long.
 1.1.4.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.4.2 30-Oct-2012  yamt sync with head
 1.1.4.1 05-Aug-2012  yamt file __aeabi_ldivmod.S was added on branch yamt-pagecache on 2012-10-30 18:46:13 +0000
 1.1.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 23-Jun-2013  tls resync from head
 1.12.2.1 10-Aug-2014  tls Rebase.
 1.13.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.13.24.1 21-Apr-2020  martin Sync with HEAD
 1.2 06-Aug-2012  matt branches: 1.2.4; 1.2.36;
Change include to <arm/aeabi.h>
 1.1 06-Aug-2012  matt Add long long / unsigned long long comparision routines. These differ
from cmpdi2 in they return strcmp like values (-1, 0, 1).
 1.2.36.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.36.1 21-Apr-2020  martin Sync with HEAD
 1.2.4.2 30-Oct-2012  yamt sync with head
 1.2.4.1 06-Aug-2012  yamt file __aeabi_ulcmp.c was added on branch yamt-pagecache on 2012-10-30 18:46:13 +0000
 1.9 06-May-2014  joerg branches: 1.9.24;
Make EHABI optional.
 1.8 12-Dec-2013  matt branches: 1.8.2;
Fix a bug in the divby0 case.
 1.7 19-Aug-2013  matt Add .cfi ops
Thumbify
 1.6 09-May-2013  matt Don't worry about divide by 0 for kernel or standalone
 1.5 08-May-2013  matt When using EABI, call __aeabi_{i,l}div0 when diving by 0.
 1.4 14-Apr-2013  skrll Pretty sure we don't want to change instruction set here. hi matt.
 1.3 14-Apr-2013  matt Fix calling of __qdivrem which the 3rd arg needs to passed on the stack.
 1.2 05-Aug-2012  matt branches: 1.2.2; 1.2.4;
Add RCSID.
Simplify.
 1.1 05-Aug-2012  matt Add a routine for __aeabi_uldivmod which is just a wrapper around __qdivrem
but returns the 64-bit dividend and remainder in r0-r3.
 1.2.4.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.2 30-Oct-2012  yamt sync with head
 1.2.4.1 05-Aug-2012  yamt file __aeabi_uldivmod.S was added on branch yamt-pagecache on 2012-10-30 18:46:13 +0000
 1.2.2.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.2.1 23-Jun-2013  tls resync from head
 1.8.2.1 10-Aug-2014  tls Rebase.
 1.9.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.9.24.1 21-Apr-2020  martin Sync with HEAD
 1.9 17-May-2015  justin branches: 1.9.16;
It is just the __ffssi2 alias we do not want in rump kernel
 1.8 17-May-2015  justin Do not build arm toolchain symbols in the rump kernel

These symbols will be provided at link time and will be
duplicate symbols in rump kernel and libc if defined.

Many have been fixed previously, but these were missed
as did not have a test.
 1.7 23-Jan-2014  martin PR port-arm/48543: do provide __ffssi2 as strong alias (in case libgcc.a gets
not linked in) for now.
OK: skrll@
 1.6 30-Sep-2013  skrll More (stylistic) whitespace.
 1.5 30-Sep-2013  skrll Appease new gas
 1.4 19-Aug-2013  matt Add END() and clarify thumb/arm
 1.3 11-Aug-2013  matt Convert some more conditional instructions to unified syntax
 1.2 03-Jul-2011  matt branches: 1.2.2; 1.2.8;
Add a weak symbol definition for __ffssi2 so that __builtin_ffs will use
this if no strong defintion of __ffssi2 is available.
 1.1 20-Dec-2005  christos branches: 1.1.18;
Merge libkern + libc common files. As requested by core.
 1.1.18.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.2.8.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.9.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.9.16.1 21-Apr-2020  martin Sync with HEAD
 1.5 26-Mar-2015  justin branches: 1.5.16;
Don't build the __eabi_ aliases in the arm rump kernel.

These are provided by libc in userspace, so leave out of kernel
 1.4 02-Dec-2013  joerg Improve EABI handling of string functions. Most importantly, fix
__aeabi_memset, which has the arguments in the wrong order.
 1.3 19-Aug-2013  matt Add END()
 1.2 11-Aug-2013  matt Switch to unified syntax
use RET/RETc
use push/pop
 1.1 20-Dec-2005  christos branches: 1.1.18; 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.18.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.8 07-Jun-2015  joerg branches: 1.8.16;
Add ARM EABI aliases for aligned arguments.
 1.7 26-Mar-2015  justin Don't build the __eabi_ aliases in the arm rump kernel.

These are provided by libc in userspace, so leave out of kernel
 1.6 02-Dec-2013  joerg branches: 1.6.4;
Improve EABI handling of string functions. Most importantly, fix
__aeabi_memset, which has the arguments in the wrong order.
 1.5 20-Aug-2013  matt Unless we are using an XSCALE, default to the normal arm version of memcpy.
 1.4 15-Feb-2013  matt #include <arm/cdefs.h> to get _ARM_ARCH_DWORD_OK
 1.3 08-Feb-2013  matt Fix typo.
 1.2 12-Dec-2012  matt Change __XSCALE__ to _ARM_ARCH_DWORD_OK so that any cpu with dword load/store
can use it.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.54; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.56.1 25-Feb-2013  tls resync with head
 1.1.54.3 15-Feb-2013  matt #include <arm/cdefs.h> to get _ARM_ARCH_DWORD_OK
 1.1.54.2 08-Feb-2013  matt Fix typo.
 1.1.54.1 07-Feb-2013  matt Sync with HEAD
 1.1.50.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.50.1 16-Jan-2013  yamt sync with (a bit old) head
 1.6.4.1 10-Jun-2015  snj Pull up following revision(s) (requested by joerg in ticket #834):
common/lib/libc/arch/arm/string/memcpy.S: revision 1.8
Add ARM EABI aliases for aligned arguments.
 1.8.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.16.1 21-Apr-2020  martin Sync with HEAD
 1.7 24-Jan-2023  mlelstv Fix two signed comparisons that were missed in the last patch.
Found be rillig@
 1.6 19-Jan-2023  mlelstv Use unsigned comparisons for pointers and size_t values.
 1.5 02-Dec-2013  joerg branches: 1.5.26; 1.5.40;
Improve EABI handling of string functions. Most importantly, fix
__aeabi_memset, which has the arguments in the wrong order.
 1.4 11-Aug-2013  matt Switch to unified syntax
use RET/RETc
use push/pop
 1.3 28-Jan-2013  matt Add aeabi strong aliases.
 1.2 28-Apr-2008  martin branches: 1.2.4; 1.2.24; 1.2.26;
Remove clause 3 and 4 from TNF licenses
 1.1 20-Dec-2005  christos branches: 1.1.18; 1.1.26;
Merge libkern + libc common files. As requested by core.
 1.1.26.1 18-May-2008  yamt sync with head.
 1.1.18.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.2.26.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.26.1 25-Feb-2013  tls resync with head
 1.2.24.1 07-Feb-2013  matt Sync with HEAD
 1.2.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.5.40.1 20-Jul-2024  martin Pull up following revision(s) (requested by rin in ticket #745):

common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.7
common/lib/libc/arch/arm/string/memcpy_xscale.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_neon.S: revision 1.2
common/lib/libc/arch/arm/string/memset_naive.S: revision 1.2
common/lib/libc/arch/arm/string/memmove.S: revision 1.11
common/lib/libc/arch/arm/string/strlen_neon.S: revision 1.4
common/lib/libc/arch/arm/string/memset.S: revision 1.9

Use unsigned comparisons for pointers and size_t values.

Fix two signed comparisons that were missed in the last patch.

Found be rillig@
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 27-Feb-2025  andvar Fix various typos in comments.
 1.2 19-Jan-2023  mlelstv branches: 1.2.6;
Use unsigned comparisons for pointers and size_t values.
 1.1 03-Jan-2013  matt branches: 1.1.2; 1.1.4; 1.1.8; 1.1.38; 1.1.52;
This is a working version of memcpy implemented using NEON instructions.
Still needs tuning as it is still about 15% than the non-NEON version.
 1.1.52.1 20-Jul-2024  martin Pull up following revision(s) (requested by rin in ticket #745):

common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.7
common/lib/libc/arch/arm/string/memcpy_xscale.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_neon.S: revision 1.2
common/lib/libc/arch/arm/string/memset_naive.S: revision 1.2
common/lib/libc/arch/arm/string/memmove.S: revision 1.11
common/lib/libc/arch/arm/string/strlen_neon.S: revision 1.4
common/lib/libc/arch/arm/string/memset.S: revision 1.9

Use unsigned comparisons for pointers and size_t values.

Fix two signed comparisons that were missed in the last patch.

Found be rillig@
 1.1.38.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.38.1 21-Apr-2020  martin Sync with HEAD
 1.1.8.2 25-Feb-2013  tls resync with head
 1.1.8.1 03-Jan-2013  tls file memcpy_neon.S was added on branch tls-maxphys on 2013-02-25 00:23:56 +0000
 1.1.4.2 07-Feb-2013  matt Sync with HEAD
 1.1.4.1 03-Jan-2013  matt file memcpy_neon.S was added on branch matt-nb6-plus on 2013-02-07 07:05:59 +0000
 1.1.2.2 23-Jan-2013  yamt sync with head
 1.1.2.1 03-Jan-2013  yamt file memcpy_neon.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.2.6.1 02-Aug-2025  perseant Sync with HEAD
 1.6 19-Jan-2023  mlelstv Use unsigned comparisons for pointers and size_t values.
 1.5 17-Dec-2013  joerg branches: 1.5.26; 1.5.40;
Write out register pairs for strd.
 1.4 19-Aug-2013  matt Add END(memcpy)
 1.3 11-Aug-2013  matt Switch to unified syntax
use RET/RETc
use push/pop
 1.2 21-Jun-2007  scw branches: 1.2.4; 1.2.34; 1.2.40;
Apply the patch, with some minor tweaks, supplied in PR/36513.
This prevents a possible prefetch past the end of the source buffer.

Note that the semantics of the pld instruction mean that it is unlikely
that this would have caused any problems except in very specific
circumstances in some types of device drivers.
 1.1 20-Dec-2005  christos branches: 1.1.6; 1.1.14;
Merge libkern + libc common files. As requested by core.
 1.1.14.1 03-Sep-2007  wrstuden Sync w/ NetBSD-4-RC_1
 1.1.6.1 22-Jun-2007  liamjfoy Pull up following revision(s) (requested by scw in ticket #741):
common/lib/libc/arch/arm/string/memcpy_xscale.S: revision 1.2
Apply the patch, with some minor tweaks, supplied in PR/36513.
This prevents a possible prefetch past the end of the source buffer.
Note that the semantics of the pld instruction mean that it is unlikely
that this would have caused any problems except in very specific
circumstances in some types of device drivers.
 1.2.40.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.34.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.5.40.1 20-Jul-2024  martin Pull up following revision(s) (requested by rin in ticket #745):

common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.7
common/lib/libc/arch/arm/string/memcpy_xscale.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_neon.S: revision 1.2
common/lib/libc/arch/arm/string/memset_naive.S: revision 1.2
common/lib/libc/arch/arm/string/memmove.S: revision 1.11
common/lib/libc/arch/arm/string/strlen_neon.S: revision 1.4
common/lib/libc/arch/arm/string/memset.S: revision 1.9

Use unsigned comparisons for pointers and size_t values.

Fix two signed comparisons that were missed in the last patch.

Found be rillig@
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.11 19-Jan-2023  mlelstv Use unsigned comparisons for pointers and size_t values.
 1.10 13-Apr-2017  skrll branches: 1.10.12; 1.10.26;
Comment alignment. No functional change.
 1.9 26-Mar-2015  justin branches: 1.9.2; 1.9.4;
Don't build the __eabi_ aliases in the arm rump kernel.

These are provided by libc in userspace, so leave out of kernel
 1.8 02-Dec-2013  joerg Improve EABI handling of string functions. Most importantly, fix
__aeabi_memset, which has the arguments in the wrong order.
 1.7 30-Nov-2013  joerg Use PLT_SYM.
 1.6 19-Aug-2013  matt Add END()
 1.5 11-Aug-2013  matt Switch to unified syntax
use RET/RETc
use push/pop
 1.4 28-Jan-2013  matt Add aeabi strong aliases.
 1.3 28-Apr-2008  martin branches: 1.3.4; 1.3.24; 1.3.26;
Remove clause 3 and 4 from TNF licenses
 1.2 20-Jun-2007  scw branches: 1.2.4; 1.2.10;
Apply the patch supplied in PR/36512 to fix the buffer overlap check.
 1.1 20-Dec-2005  christos branches: 1.1.6; 1.1.14;
Merge libkern + libc common files. As requested by core.
 1.1.14.1 03-Sep-2007  wrstuden Sync w/ NetBSD-4-RC_1
 1.1.6.1 21-Jun-2007  liamjfoy Pull up following revision(s) (requested by scw in ticket #740):
common/lib/libc/arch/arm/string/memmove.S: revision 1.2
Apply the patch supplied in PR/36512 to fix the buffer overlap check.
 1.2.10.1 18-May-2008  yamt sync with head.
 1.2.4.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.3.26.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.26.1 25-Feb-2013  tls resync with head
 1.3.24.1 07-Feb-2013  matt Sync with HEAD
 1.3.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.9.4.1 21-Apr-2017  bouyer Sync with HEAD
 1.9.2.1 26-Apr-2017  pgoyette Sync with HEAD
 1.10.26.1 20-Jul-2024  martin Pull up following revision(s) (requested by rin in ticket #745):

common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.7
common/lib/libc/arch/arm/string/memcpy_xscale.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_neon.S: revision 1.2
common/lib/libc/arch/arm/string/memset_naive.S: revision 1.2
common/lib/libc/arch/arm/string/memmove.S: revision 1.11
common/lib/libc/arch/arm/string/strlen_neon.S: revision 1.4
common/lib/libc/arch/arm/string/memset.S: revision 1.9

Use unsigned comparisons for pointers and size_t values.

Fix two signed comparisons that were missed in the last patch.

Found be rillig@
 1.10.12.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.10.12.1 21-Apr-2020  martin Sync with HEAD
 1.9 19-Jan-2023  mlelstv Use unsigned comparisons for pointers and size_t values.
 1.8 26-Mar-2015  justin branches: 1.8.16; 1.8.30;
Don't build the __eabi_ aliases in the arm rump kernel.

These are provided by libc in userspace, so leave out of kernel
 1.7 02-Dec-2013  joerg Improve EABI handling of string functions. Most importantly, fix
__aeabi_memset, which has the arguments in the wrong order.
 1.6 30-Nov-2013  joerg Use explicit form of register pair operations by specifying both.
 1.5 19-Aug-2013  matt Add END()
 1.4 11-Aug-2013  matt Switch to unified syntax
use RET/RETc
use push/pop
 1.3 28-Jan-2013  matt Add aeabi strong aliases.
 1.2 12-Dec-2012  matt Change __XSCALE__ to _ARM_ARCH_DWORD_OK so that any cpu with strd can use it.
 1.1 20-Dec-2005  christos branches: 1.1.18; 1.1.50; 1.1.54; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.56.1 25-Feb-2013  tls resync with head
 1.1.54.1 07-Feb-2013  matt Sync with HEAD
 1.1.50.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.50.1 16-Jan-2013  yamt sync with (a bit old) head
 1.1.18.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.8.30.1 20-Jul-2024  martin Pull up following revision(s) (requested by rin in ticket #745):

common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.7
common/lib/libc/arch/arm/string/memcpy_xscale.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_neon.S: revision 1.2
common/lib/libc/arch/arm/string/memset_naive.S: revision 1.2
common/lib/libc/arch/arm/string/memmove.S: revision 1.11
common/lib/libc/arch/arm/string/strlen_neon.S: revision 1.4
common/lib/libc/arch/arm/string/memset.S: revision 1.9

Use unsigned comparisons for pointers and size_t values.

Fix two signed comparisons that were missed in the last patch.

Found be rillig@
 1.8.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 14-Jan-2013  matt branches: 1.2.2; 1.2.4; 1.2.8; 1.2.38;
Fix two typos
 1.1 12-Jan-2013  matt A version of memset that can do NEON, VFP as well as normal arm instructions
 1.2.38.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.38.1 21-Apr-2020  martin Sync with HEAD
 1.2.8.2 25-Feb-2013  tls resync with head
 1.2.8.1 14-Jan-2013  tls file memset_arm.S was added on branch tls-maxphys on 2013-02-25 00:23:56 +0000
 1.2.4.2 07-Feb-2013  matt Sync with HEAD
 1.2.4.1 14-Jan-2013  matt file memset_arm.S was added on branch matt-nb6-plus on 2013-02-07 07:05:59 +0000
 1.2.2.2 23-Jan-2013  yamt sync with head
 1.2.2.1 14-Jan-2013  yamt file memset_arm.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.2 19-Jan-2023  mlelstv Use unsigned comparisons for pointers and size_t values.
 1.1 08-Jan-2013  matt branches: 1.1.2; 1.1.4; 1.1.8; 1.1.38; 1.1.52;
Add a simplier version of memset which is less than 1/2 the size of the
current one. On a Cortex-A9, this is about 15%-30% faster than the current
libc version. This is not a trivial implementation since that was an order
magnitude slower than the existing libc version.
 1.1.52.1 20-Jul-2024  martin Pull up following revision(s) (requested by rin in ticket #745):

common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.7
common/lib/libc/arch/arm/string/memcpy_xscale.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_neon.S: revision 1.2
common/lib/libc/arch/arm/string/memset_naive.S: revision 1.2
common/lib/libc/arch/arm/string/memmove.S: revision 1.11
common/lib/libc/arch/arm/string/strlen_neon.S: revision 1.4
common/lib/libc/arch/arm/string/memset.S: revision 1.9

Use unsigned comparisons for pointers and size_t values.

Fix two signed comparisons that were missed in the last patch.

Found be rillig@
 1.1.38.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.38.1 21-Apr-2020  martin Sync with HEAD
 1.1.8.2 25-Feb-2013  tls resync with head
 1.1.8.1 08-Jan-2013  tls file memset_naive.S was added on branch tls-maxphys on 2013-02-25 00:23:56 +0000
 1.1.4.2 07-Feb-2013  matt Sync with HEAD
 1.1.4.1 08-Jan-2013  matt file memset_naive.S was added on branch matt-nb6-plus on 2013-02-07 07:05:59 +0000
 1.1.2.2 23-Jan-2013  yamt sync with head
 1.1.2.1 08-Jan-2013  yamt file memset_naive.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.1 13-Dec-2012  matt branches: 1.1.2; 1.1.4; 1.1.8; 1.1.38;
Add a NEON(only) implementation of memset.
This is a work in progress.
 1.1.38.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.38.1 21-Apr-2020  martin Sync with HEAD
 1.1.8.2 25-Feb-2013  tls resync with head
 1.1.8.1 13-Dec-2012  tls file memset_neon.S was added on branch tls-maxphys on 2013-02-25 00:23:56 +0000
 1.1.4.2 07-Feb-2013  matt Sync with HEAD
 1.1.4.1 13-Dec-2012  matt file memset_neon.S was added on branch matt-nb6-plus on 2013-02-07 07:06:00 +0000
 1.1.2.2 16-Jan-2013  yamt sync with (a bit old) head
 1.1.2.1 13-Dec-2012  yamt file memset_neon.S was added on branch yamt-pagecache on 2013-01-16 05:25:53 +0000
 1.1 23-Jan-2013  matt branches: 1.1.2; 1.1.6; 1.1.14; 1.1.38;
Switch to using ARM assembly versions of strcat, strchr, strrchr.
 1.1.38.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.38.1 21-Apr-2020  martin Sync with HEAD
 1.1.14.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.14.1 23-Jan-2013  yamt file strcat.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.1.6.2 25-Feb-2013  tls resync with head
 1.1.6.1 23-Jan-2013  tls file strcat.S was added on branch tls-maxphys on 2013-02-25 00:23:57 +0000
 1.1.2.2 07-Feb-2013  matt Sync with HEAD
 1.1.2.1 23-Jan-2013  matt file strcat.S was added on branch matt-nb6-plus on 2013-02-07 07:06:00 +0000
 1.6 06-May-2014  joerg branches: 1.6.24;
Make EHABI optional.
 1.5 05-Sep-2013  matt branches: 1.5.2;
Use __ARM_EABI__ and new __UNWIND_TABLES__ to decide when to use .cfi ops
 1.4 22-Aug-2013  matt Don't include .cfi info if _KERNEL || _STANDALONE
 1.3 19-Aug-2013  matt Add .cfi ops if EABI.
Thumbify.
 1.2 15-Jan-2013  matt branches: 1.2.2; 1.2.4; 1.2.8;
Add missing ! on str
 1.1 14-Jan-2013  matt Add a native version of strcat which uses the optimized strlen and strcpy
routines.
 1.2.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.2 25-Feb-2013  tls resync with head
 1.2.8.1 15-Jan-2013  tls file strcat_arm.S was added on branch tls-maxphys on 2013-02-25 00:23:57 +0000
 1.2.4.2 07-Feb-2013  matt Sync with HEAD
 1.2.4.1 15-Jan-2013  matt file strcat_arm.S was added on branch matt-nb6-plus on 2013-02-07 07:06:00 +0000
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 23-Jan-2013  yamt sync with head
 1.2.2.1 15-Jan-2013  yamt file strcat_arm.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.5.2.1 10-Aug-2014  tls Rebase.
 1.6.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.24.1 21-Apr-2020  martin Sync with HEAD
 1.3 19-Aug-2013  matt branches: 1.3.26;
cbnz/cbz can not branch backwards so nuke 'em.
Use the same register usage in strlen as in strnlen
 1.2 19-Aug-2013  matt Thumbify
 1.1 14-Jan-2013  matt branches: 1.1.2; 1.1.4; 1.1.8;
Add a simple version of strcat.
 1.1.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.8.2 25-Feb-2013  tls resync with head
 1.1.8.1 14-Jan-2013  tls file strcat_naive.S was added on branch tls-maxphys on 2013-02-25 00:23:57 +0000
 1.1.4.2 07-Feb-2013  matt Sync with HEAD
 1.1.4.1 14-Jan-2013  matt file strcat_naive.S was added on branch matt-nb6-plus on 2013-02-07 07:06:00 +0000
 1.1.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.2.2 23-Jan-2013  yamt sync with head
 1.1.2.1 14-Jan-2013  yamt file strcat_naive.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 20-Aug-2013  matt branches: 1.4.4; 1.4.28;
Use the arm versions of strlen/strchr/strrchr if compiling thumb2
 1.3 19-Aug-2013  matt For Thumb, use naive version
 1.2 23-Jan-2013  matt branches: 1.2.2; 1.2.6;
Add index/rindex strong aiases.
 1.1 23-Jan-2013  matt Switch to using ARM assembly versions of strcat, strchr, strrchr.
 1.2.6.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.6.2 25-Feb-2013  tls resync with head
 1.2.6.1 23-Jan-2013  tls file strchr.S was added on branch tls-maxphys on 2013-02-25 00:23:57 +0000
 1.2.2.2 07-Feb-2013  matt Sync with HEAD
 1.2.2.1 23-Jan-2013  matt file strchr.S was added on branch matt-nb6-plus on 2013-02-07 07:06:00 +0000
 1.4.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.28.1 21-Apr-2020  martin Sync with HEAD
 1.4.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.4.1 20-Aug-2013  yamt file strchr.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.8 19-Aug-2013  matt branches: 1.8.26;
Add two thumb2 bits.
 1.7 19-Aug-2013  matt Missing one teq -> cmp
 1.6 19-Aug-2013  matt Swap use of r1 and ip
teq -> cmp.
add s to few instructions
(thumbify part 1)
 1.5 08-Feb-2013  matt branches: 1.5.4;
Fix corner cases when searching for NUL.
 1.4 07-Feb-2013  matt branches: 1.4.2;
orrne wants 3 registers
 1.3 26-Jan-2013  matt Fix bug in detecting EOS/match on armv6+.
 1.2 15-Jan-2013  matt branches: 1.2.2;
Fix case when searching for NUL.
 1.1 15-Jan-2013  matt Add an ARM optimized version of strchr.
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 23-Jan-2013  yamt sync with head
 1.2.2.1 15-Jan-2013  yamt file strchr_arm.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.4.2.3 08-Feb-2013  matt Sync with HEAD.
 1.4.2.2 07-Feb-2013  matt Sync with HEAD
 1.4.2.1 07-Feb-2013  matt file strchr_arm.S was added on branch matt-nb6-plus on 2013-02-07 07:06:00 +0000
 1.5.4.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.4.2 25-Feb-2013  tls resync with head
 1.5.4.1 08-Feb-2013  tls file strchr_arm.S was added on branch tls-maxphys on 2013-02-25 00:23:57 +0000
 1.8.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 19-Aug-2013  matt branches: 1.4.26;
cbnz/cbz can not branch backwards so nuke 'em.
Use the same register usage in strlen as in strnlen
 1.3 19-Aug-2013  matt Thumbify
 1.2 19-Aug-2013  matt ip -> r2
teq -> cmp
 1.1 15-Jan-2013  matt branches: 1.1.2; 1.1.4; 1.1.8;
Add simple/small versions of strchr/strrchr for ARM.
 1.1.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.8.2 25-Feb-2013  tls resync with head
 1.1.8.1 15-Jan-2013  tls file strchr_naive.S was added on branch tls-maxphys on 2013-02-25 00:23:57 +0000
 1.1.4.2 07-Feb-2013  matt Sync with HEAD
 1.1.4.1 15-Jan-2013  matt file strchr_naive.S was added on branch matt-nb6-plus on 2013-02-07 07:06:00 +0000
 1.1.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.2.2 23-Jan-2013  yamt sync with head
 1.1.2.1 15-Jan-2013  yamt file strchr_naive.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.4.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 19-Aug-2013  matt branches: 1.2.26;
Thumbify
 1.1 20-Dec-2005  christos branches: 1.1.18; 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.18.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.5 20-Aug-2013  matt branches: 1.5.26;
If compiling standalone with Thumb, use the thumb version instead of the
naive version.
 1.4 10-Jan-2013  matt branches: 1.4.2; 1.4.4; 1.4.8;
use #if defined(xxx) instead of ifdef
 1.3 08-Jan-2013  matt Depending on _STANDALONE include the "naive" version or the normal arm version.
 1.2 02-Jan-2013  matt Deal with _LIBC (aliases, etc).
Add missing #endif.
 1.1 02-Jan-2013  matt Add an assembly version of strcpy/strncpy/strlcpy.
(they all use a common source with defines to determine which to build).
 1.4.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.8.2 25-Feb-2013  tls resync with head
 1.4.8.1 10-Jan-2013  tls file strcpy.S was added on branch tls-maxphys on 2013-02-25 00:23:57 +0000
 1.4.4.2 07-Feb-2013  matt Sync with HEAD
 1.4.4.1 10-Jan-2013  matt file strcpy.S was added on branch matt-nb6-plus on 2013-02-07 07:06:01 +0000
 1.4.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.2 23-Jan-2013  yamt sync with head
 1.4.2.1 10-Jan-2013  yamt file strcpy.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.7 08-Feb-2024  andvar fix misplaced or missing "e" in words with "ment" ending (argument, implement,
increment, decrement, alignment), in comments, documentation, log messages.
 1.6 14-Jan-2017  christos branches: 1.6.14;
fix rump
 1.5 14-Jan-2017  christos fix weak symbols. More work needs to be done for memcpy/strlen etc.
 1.4 13-Jan-2017  christos No need to include namespace.h; no other assembly code does.
 1.3 11-Aug-2013  matt branches: 1.3.8;
Switch to unified syntax
use RET/RETc
use push/pop
 1.2 10-Jan-2013  matt branches: 1.2.2; 1.2.4; 1.2.8;
Fix a typo in strlcpy which caused to not deal with NULs predecing the
string properly.
 1.1 08-Jan-2013  matt Rename strlen.S and strcpy.S to strlen_arm.S and strcpy_arm.S
 1.2.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.2 25-Feb-2013  tls resync with head
 1.2.8.1 10-Jan-2013  tls file strcpy_arm.S was added on branch tls-maxphys on 2013-02-25 00:23:58 +0000
 1.2.4.2 07-Feb-2013  matt Sync with HEAD
 1.2.4.1 10-Jan-2013  matt file strcpy_arm.S was added on branch matt-nb6-plus on 2013-02-07 07:06:01 +0000
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 23-Jan-2013  yamt sync with head
 1.2.2.1 10-Jan-2013  yamt file strcpy_arm.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.3.8.1 20-Mar-2017  pgoyette Sync with HEAD
 1.6.14.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.14.1 21-Apr-2020  martin Sync with HEAD
 1.8 14-Jan-2017  christos branches: 1.8.14;
fix standalone
 1.7 14-Jan-2017  christos fix standalone
 1.6 14-Jan-2017  christos fix rump
 1.5 13-Jan-2017  christos No need to include namespace.h; no other assembly code does.
 1.4 20-Aug-2013  matt branches: 1.4.8;
write of final NUL in strlcpy doesn't need to be post-incremented
 1.3 11-Aug-2013  matt Convert some more conditional instructions to unified syntax
 1.2 10-Jan-2013  matt branches: 1.2.2; 1.2.4; 1.2.8;
Add weak alias for strlcpy
 1.1 08-Jan-2013  matt Add simple/small versions of the str* functions. Suitable for libsa, etc.
 1.2.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.2 25-Feb-2013  tls resync with head
 1.2.8.1 10-Jan-2013  tls file strcpy_naive.S was added on branch tls-maxphys on 2013-02-25 00:23:58 +0000
 1.2.4.2 07-Feb-2013  matt Sync with HEAD
 1.2.4.1 10-Jan-2013  matt file strcpy_naive.S was added on branch matt-nb6-plus on 2013-02-07 07:06:01 +0000
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 23-Jan-2013  yamt sync with head
 1.2.2.1 10-Jan-2013  yamt file strcpy_naive.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.4.8.1 20-Mar-2017  pgoyette Sync with HEAD
 1.8.14.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.14.1 21-Apr-2020  martin Sync with HEAD
 1.3 14-Jan-2017  christos branches: 1.3.14;
fix weak symbols. More work needs to be done for memcpy/strlen etc.
 1.2 13-Jan-2017  christos No need to include namespace.h; no other assembly code does.
 1.1 20-Aug-2013  matt branches: 1.1.4; 1.1.8; 1.1.12;
Thumb versions of strcpy/strlcpy/strncpy
 1.1.12.1 20-Mar-2017  pgoyette Sync with HEAD
 1.1.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.8.1 20-Aug-2013  tls file strcpy_thumb.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000
 1.1.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.4.1 20-Aug-2013  yamt file strcpy_thumb.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.3.14.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.14.1 21-Apr-2020  martin Sync with HEAD
 1.4 14-Jan-2017  christos branches: 1.4.14;
fix weak symbols. More work needs to be done for memcpy/strlen etc.
 1.3 13-Jan-2017  christos No need to include namespace.h; no other assembly code does.
 1.2 20-Aug-2013  matt branches: 1.2.4; 1.2.10;
strlcat_arm.S is smaller than strlcat_naive.S so always use it.
 1.1 23-Jan-2013  matt branches: 1.1.2; 1.1.6;
Enable ARM assembly versions of strlcat and strnlen.
 1.1.6.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.6.2 25-Feb-2013  tls resync with head
 1.1.6.1 23-Jan-2013  tls file strlcat.S was added on branch tls-maxphys on 2013-02-25 00:23:58 +0000
 1.1.2.2 07-Feb-2013  matt Sync with HEAD
 1.1.2.1 23-Jan-2013  matt file strlcat.S was added on branch matt-nb6-plus on 2013-02-07 07:06:01 +0000
 1.2.10.1 20-Mar-2017  pgoyette Sync with HEAD
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 20-Aug-2013  yamt file strlcat.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.4.14.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.14.1 21-Apr-2020  martin Sync with HEAD
 1.6 06-May-2014  joerg branches: 1.6.2; 1.6.26;
Make EHABI optional.
 1.5 05-Sep-2013  matt branches: 1.5.2;
Use __ARM_EABI__ and new __UNWIND_TABLES__ to decide when to use .cfi ops
 1.4 22-Aug-2013  matt Don't include .cfi info if _KERNEL || _STANDALONE
 1.3 19-Aug-2013  matt fix cfi_register -> cfi_offset
 1.2 19-Aug-2013  matt Add .cfi for __ARM_EABI__
Thumbify
 1.1 23-Jan-2013  matt branches: 1.1.2; 1.1.6;
Add ARM assembly of strlcat which is implemented using (the ARM optimized)
strnlen and strlcpy.
 1.1.6.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.6.2 25-Feb-2013  tls resync with head
 1.1.6.1 23-Jan-2013  tls file strlcat_arm.S was added on branch tls-maxphys on 2013-02-25 00:23:58 +0000
 1.1.2.2 07-Feb-2013  matt Sync with HEAD
 1.1.2.1 23-Jan-2013  matt file strlcat_arm.S was added on branch matt-nb6-plus on 2013-02-07 07:06:01 +0000
 1.5.2.1 10-Aug-2014  tls Rebase.
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.6.2.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.2.1 06-May-2014  yamt file strlcat_arm.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.3 22-Aug-2013  matt branches: 1.3.26;
Don't include .cfi info if _KERNEL || _STANDALONE
 1.2 11-Aug-2013  matt Convert some more conditional instructions to unified syntax
 1.1 18-Jan-2013  matt branches: 1.1.2; 1.1.4; 1.1.8;
Simple version of strlcat for ARM.
 1.1.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.8.2 25-Feb-2013  tls resync with head
 1.1.8.1 18-Jan-2013  tls file strlcat_naive.S was added on branch tls-maxphys on 2013-02-25 00:23:58 +0000
 1.1.4.2 07-Feb-2013  matt Sync with HEAD
 1.1.4.1 18-Jan-2013  matt file strlcat_naive.S was added on branch matt-nb6-plus on 2013-02-07 07:06:01 +0000
 1.1.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.2.2 23-Jan-2013  yamt sync with head
 1.1.2.1 18-Jan-2013  yamt file strlcat_naive.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.5 20-Aug-2013  matt branches: 1.5.26;
If compiling standalone with Thumb, use the thumb version instead of the
naive version.
 1.4 10-Jan-2013  matt branches: 1.4.2; 1.4.4; 1.4.8;
Back out workaround.
 1.3 10-Jan-2013  matt Use the naive version of strlcpy until the longer one is fixed (it's still
faster than the C version).
 1.2 10-Jan-2013  matt use #if defined(xxx) instead of ifdef
 1.1 02-Jan-2013  matt Add an assembly version of strcpy/strncpy/strlcpy.
(they all use a common source with defines to determine which to build).
 1.4.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.8.2 25-Feb-2013  tls resync with head
 1.4.8.1 10-Jan-2013  tls file strlcpy.S was added on branch tls-maxphys on 2013-02-25 00:23:58 +0000
 1.4.4.2 07-Feb-2013  matt Sync with HEAD
 1.4.4.1 10-Jan-2013  matt file strlcpy.S was added on branch matt-nb6-plus on 2013-02-07 07:06:01 +0000
 1.4.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.2 23-Jan-2013  yamt sync with head
 1.4.2.1 10-Jan-2013  yamt file strlcpy.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 20-Aug-2013  matt branches: 1.4.26;
Use the arm versions of strlen/strchr/strrchr if compiling thumb2
 1.3 19-Aug-2013  matt For Thumb, use naive version
 1.2 08-Jan-2013  matt branches: 1.2.2; 1.2.4; 1.2.8;
Depending on _STANDALONE include the "naive" version or the normal arm version.
 1.1 02-Jan-2013  matt Rename strlen_armv6.S to strlen.S since this is no longer armv6 dependent.
 1.2.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.2 25-Feb-2013  tls resync with head
 1.2.8.1 08-Jan-2013  tls file strlen.S was added on branch tls-maxphys on 2013-02-25 00:23:58 +0000
 1.2.4.2 07-Feb-2013  matt Sync with HEAD
 1.2.4.1 08-Jan-2013  matt file strlen.S was added on branch matt-nb6-plus on 2013-02-07 07:06:01 +0000
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 23-Jan-2013  yamt sync with head
 1.2.2.1 08-Jan-2013  yamt file strlen.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.4.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.26.1 21-Apr-2020  martin Sync with HEAD
 1.11 15-Jan-2023  skrll Fix strnlen with a large maxlen argument by using unsigned comparison
conditions - from mlelstv.

I had a similar, but not quite as good patch.
 1.10 03-Dec-2022  skrll branches: 1.10.2;
Fix some comments
 1.9 06-May-2014  joerg branches: 1.9.24; 1.9.26;
Make EHABI optional.
 1.8 05-Sep-2013  matt branches: 1.8.2;
Use __ARM_EABI__ and new __UNWIND_TABLES__ to decide when to use .cfi ops
 1.7 22-Aug-2013  matt Don't include .cfi info if _KERNEL || _STANDALONE
 1.6 20-Aug-2013  matt Add a missing it gt before movgt for thumb
 1.5 20-Aug-2013  matt thumbify (part2)
 1.4 19-Aug-2013  matt For EABI, add .cfi ops
 1.3 23-Jan-2013  matt branches: 1.3.2; 1.3.6;
Add support for strnlen.
 1.2 09-Jan-2013  matt branches: 1.2.2;
Use movw on armv7 to fill uppper halfword.
 1.1 08-Jan-2013  matt Rename strlen.S and strcpy.S to strlen_arm.S and strcpy_arm.S
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 23-Jan-2013  yamt sync with head
 1.2.2.1 09-Jan-2013  yamt file strlen_arm.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.3.6.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.6.2 25-Feb-2013  tls resync with head
 1.3.6.1 23-Jan-2013  tls file strlen_arm.S was added on branch tls-maxphys on 2013-02-25 00:23:59 +0000
 1.3.2.2 07-Feb-2013  matt Sync with HEAD
 1.3.2.1 23-Jan-2013  matt file strlen_arm.S was added on branch matt-nb6-plus on 2013-02-07 07:06:02 +0000
 1.8.2.1 10-Aug-2014  tls Rebase.
 1.9.26.1 19-Jan-2023  martin Pull up following revision(s) (requested by skrll in ticket #1567):

common/lib/libc/arch/arm/string/strlen_arm.S: revision 1.11

Fix strnlen with a large maxlen argument by using unsigned comparison
conditions - from mlelstv.

I had a similar, but not quite as good patch.
 1.9.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.9.24.1 21-Apr-2020  martin Sync with HEAD
 1.10.2.1 19-Jan-2023  martin Pull up following revision(s) (requested by skrll in ticket #51):

common/lib/libc/arch/arm/string/strlen_arm.S: revision 1.11

Fix strnlen with a large maxlen argument by using unsigned comparison
conditions - from mlelstv.

I had a similar, but not quite as good patch.
 1.4 02-Jan-2013  matt Rename strlen_armv6.S to strlen.S since this is no longer armv6 dependent.
 1.3 31-Dec-2012  matt Make this work on all ARMs but keep the armv6 optimizations. It as fast as
the existing strlen for small string and once strings are 8 bytes or more in
length it starts getting significantly faster. For really long strings,
compared to the existing strlen, this uses about 1/2 of the cycles for the
non-armv6 version and about 1/3 of the cycles for the armv6 version.
 1.2 29-Dec-2012  matt A few slight speedups (remove one instruction from the main loop).
 1.1 28-Dec-2012  matt strlen implementation for armv6 and later. Uses clz and uqadd8 to really
speed the search for NUL. as fast as normal strlen at about a length of
6 or 7 and 2-3 times faster starting around 10.
 1.9 03-Dec-2022  skrll improve a comment
 1.8 19-Aug-2013  matt branches: 1.8.26;
cbnz/cbz can not branch backwards so nuke 'em.
Use the same register usage in strlen as in strnlen
 1.7 19-Aug-2013  matt Thumbify
 1.6 19-Aug-2013  matt Use ip as a temporary
 1.5 19-Aug-2013  matt Change previous use of r2 to r3
 1.4 19-Aug-2013  matt teq -> cmp
ip -> r2
add/sub -> adds/subs
(thumbify part 1)
 1.3 23-Jan-2013  matt branches: 1.3.2; 1.3.6;
Add support for strnlen.
 1.2 08-Jan-2013  pgoyette branches: 1.2.2;
Add missing quote - fix build
 1.1 08-Jan-2013  matt Add simple/small versions of the str* functions. Suitable for libsa, etc.
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 23-Jan-2013  yamt sync with head
 1.2.2.1 08-Jan-2013  yamt file strlen_naive.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.3.6.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.6.2 25-Feb-2013  tls resync with head
 1.3.6.1 23-Jan-2013  tls file strlen_naive.S was added on branch tls-maxphys on 2013-02-25 00:23:59 +0000
 1.3.2.2 07-Feb-2013  matt Sync with HEAD
 1.3.2.1 23-Jan-2013  matt file strlen_naive.S was added on branch matt-nb6-plus on 2013-02-07 07:06:02 +0000
 1.8.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 19-Jan-2023  mlelstv Use unsigned comparisons for pointers and size_t values.
 1.3 28-Dec-2012  matt branches: 1.3.2; 1.3.4; 1.3.8; 1.3.38; 1.3.52;
Debug. This becomes faster than the normal strlen at about 80 characters.
 1.2 15-Dec-2012  matt Slighly improved (can deal with all 16 bytes being non-NUL and quickly
proceed to next qword).
 1.1 15-Dec-2012  matt Add a NEON implementation of strlen.
 1.3.52.1 20-Jul-2024  martin Pull up following revision(s) (requested by rin in ticket #745):

common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_arm.S: revision 1.7
common/lib/libc/arch/arm/string/memcpy_xscale.S: revision 1.6
common/lib/libc/arch/arm/string/memcpy_neon.S: revision 1.2
common/lib/libc/arch/arm/string/memset_naive.S: revision 1.2
common/lib/libc/arch/arm/string/memmove.S: revision 1.11
common/lib/libc/arch/arm/string/strlen_neon.S: revision 1.4
common/lib/libc/arch/arm/string/memset.S: revision 1.9

Use unsigned comparisons for pointers and size_t values.

Fix two signed comparisons that were missed in the last patch.

Found be rillig@
 1.3.38.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.38.1 21-Apr-2020  martin Sync with HEAD
 1.3.8.2 25-Feb-2013  tls resync with head
 1.3.8.1 28-Dec-2012  tls file strlen_neon.S was added on branch tls-maxphys on 2013-02-25 00:23:59 +0000
 1.3.4.2 07-Feb-2013  matt Sync with HEAD
 1.3.4.1 28-Dec-2012  matt file strlen_neon.S was added on branch matt-nb6-plus on 2013-02-07 07:06:02 +0000
 1.3.2.2 23-Jan-2013  yamt sync with head
 1.3.2.1 28-Dec-2012  yamt file strlen_neon.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.2 19-Aug-2013  matt branches: 1.2.26;
Thumbify
 1.1 20-Dec-2005  christos branches: 1.1.18; 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.18.1 28-Aug-2007  matt Add thumb versions (or force to arm32 mode when in thumb mode).
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 20-Aug-2013  matt branches: 1.3.26;
If compiling standalone with Thumb, use the thumb version instead of the
naive version.
 1.2 10-Jan-2013  matt branches: 1.2.2; 1.2.4; 1.2.8;
use #if defined(xxx) instead of ifdef
 1.1 02-Jan-2013  matt Add an assembly version of strcpy/strncpy/strlcpy.
(they all use a common source with defines to determine which to build).
 1.2.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.2 25-Feb-2013  tls resync with head
 1.2.8.1 10-Jan-2013  tls file strncpy.S was added on branch tls-maxphys on 2013-02-25 00:23:59 +0000
 1.2.4.2 07-Feb-2013  matt Sync with HEAD
 1.2.4.1 10-Jan-2013  matt file strncpy.S was added on branch matt-nb6-plus on 2013-02-07 07:06:02 +0000
 1.2.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.2.2 23-Jan-2013  yamt sync with head
 1.2.2.1 10-Jan-2013  yamt file strncpy.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 20-Aug-2013  matt branches: 1.3.4; 1.3.28;
Use the arm versions of strnlen if compiling thumb2
 1.2 19-Aug-2013  matt For Thumb, use naive version
 1.1 23-Jan-2013  matt branches: 1.1.2; 1.1.6;
Enable ARM assembly versions of strlcat and strnlen.
 1.1.6.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.6.2 25-Feb-2013  tls resync with head
 1.1.6.1 23-Jan-2013  tls file strnlen.S was added on branch tls-maxphys on 2013-02-25 00:23:59 +0000
 1.1.2.2 07-Feb-2013  matt Sync with HEAD
 1.1.2.1 23-Jan-2013  matt file strnlen.S was added on branch matt-nb6-plus on 2013-02-07 07:06:02 +0000
 1.3.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.28.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.4.1 20-Aug-2013  yamt file strnlen.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.4 20-Aug-2013  matt branches: 1.4.4; 1.4.28;
Use the arm versions of strlen/strchr/strrchr if compiling thumb2
 1.3 19-Aug-2013  matt For Thumb, use naive version
 1.2 23-Jan-2013  matt branches: 1.2.2; 1.2.6;
Add index/rindex strong aiases.
 1.1 23-Jan-2013  matt Switch to using ARM assembly versions of strcat, strchr, strrchr.
 1.2.6.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.6.2 25-Feb-2013  tls resync with head
 1.2.6.1 23-Jan-2013  tls file strrchr.S was added on branch tls-maxphys on 2013-02-25 00:23:59 +0000
 1.2.2.2 07-Feb-2013  matt Sync with HEAD
 1.2.2.1 23-Jan-2013  matt file strrchr.S was added on branch matt-nb6-plus on 2013-02-07 07:06:02 +0000
 1.4.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.28.1 21-Apr-2020  martin Sync with HEAD
 1.4.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.4.1 20-Aug-2013  yamt file strrchr.S was added on branch yamt-pagecache on 2014-05-22 11:26:28 +0000
 1.6 25-Aug-2013  matt branches: 1.6.26;
move a misplaced #endif
 1.5 20-Aug-2013  matt swap r1 & ip
use adds, eors, etc.
teq -> cmp
 1.4 11-Aug-2013  matt Switch to unified syntax
use RET/RETc
use push/pop
 1.3 08-Feb-2013  matt branches: 1.3.4;
Fix corner cases when searching for NUL.
 1.2 28-Jan-2013  matt branches: 1.2.2;
Deal with an end-of-string condition properly.
 1.1 15-Jan-2013  matt branches: 1.1.2;
Add ARM optimized version of strrchr.
 1.1.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.2.2 23-Jan-2013  yamt sync with head
 1.1.2.1 15-Jan-2013  yamt file strrchr_arm.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.2.2.3 08-Feb-2013  matt Sync with HEAD.
 1.2.2.2 07-Feb-2013  matt Sync with HEAD
 1.2.2.1 28-Jan-2013  matt file strrchr_arm.S was added on branch matt-nb6-plus on 2013-02-07 07:06:02 +0000
 1.3.4.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.2 25-Feb-2013  tls resync with head
 1.3.4.1 08-Feb-2013  tls file strrchr_arm.S was added on branch tls-maxphys on 2013-02-25 00:23:59 +0000
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 19-Aug-2013  matt branches: 1.4.26;
cbnz/cbz can not branch backwards so nuke 'em.
Use the same register usage in strlen as in strnlen
 1.3 19-Aug-2013  matt Thumbify
 1.2 19-Aug-2013  matt ip -> r2
teq -> cmp
 1.1 15-Jan-2013  matt branches: 1.1.2; 1.1.4; 1.1.8;
Add simple/small versions of strchr/strrchr for ARM.
 1.1.8.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.8.2 25-Feb-2013  tls resync with head
 1.1.8.1 15-Jan-2013  tls file strrchr_naive.S was added on branch tls-maxphys on 2013-02-25 00:23:59 +0000
 1.1.4.2 07-Feb-2013  matt Sync with HEAD
 1.1.4.1 15-Jan-2013  matt file strrchr_naive.S was added on branch matt-nb6-plus on 2013-02-07 07:06:02 +0000
 1.1.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.2.2 23-Jan-2013  yamt sync with head
 1.1.2.1 15-Jan-2013  yamt file strrchr_naive.S was added on branch yamt-pagecache on 2013-01-23 00:04:06 +0000
 1.4.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.26.1 21-Apr-2020  martin Sync with HEAD
 1.16 29-Oct-2023  skrll Define __HAVE_ATOMIC_CAS_64_UP and provide __sync_val_compare_and_swap_8

The new santizer code in gcc12 needs this.
 1.15 30-Mar-2023  riastradh libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.
For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.

XXX pullup-10
 1.14 28-Feb-2019  isaki branches: 1.14.14;
Add missing atomic_and_{8,16}_nv_cas.c for __sync_and_and_fetch_{1,2}.
XXX why is not only atomic_and_* symmetric unlike the others?
(in common/lib/libc/atomic/)
 1.13 14-Oct-2014  martin branches: 1.13.16;
Provide C++ 2011 <atomic> support functions for hppa and arm.
 1.12 22-Feb-2014  skrll branches: 1.12.4;
Add the missing __sync_* ops to libc
 1.11 22-Feb-2014  skrll Another layout change
 1.10 22-Feb-2014  skrll Layout changes - no functional change.
 1.9 29-Jan-2014  skrll Provide _atomic_cas_8_up and _atomic_cas_16_up functions
 1.8 17-Jan-2011  skrll branches: 1.8.6; 1.8.12;
Tweak layout - no functional change.
 1.7 06-Jan-2009  mjf The .if conditional crosses a line boundary, so add a backslash.

This fixes the kernel build for hppa.
 1.6 04-Jan-2009  pooka allow inclusion of atomic ops in librump
 1.5 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.4 11-Aug-2008  skrll Provide assembly _atomic_cas_up() as the compiler cannot be trusted to
generate fully restartable code sequences.
 1.3 11-Feb-2008  ad branches: 1.3.6;
Only build atomic ops for libkern/libc.
 1.2 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.1 29-Nov-2007  ad branches: 1.1.4;
Atomic ops for hppa.
 1.1.4.3 23-Mar-2008  matt sync with HEAD
 1.1.4.2 09-Jan-2008  matt sync with HEAD
 1.1.4.1 29-Nov-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:20:52 +0000
 1.3.6.1 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.8.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.8.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.12.4.1 12-Nov-2014  snj Pull up following revision(s) (requested by martin in ticket #218):
common/lib/libc/arch/arm/atomic/Makefile.inc: revision 1.24-1.26
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.7
common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.18
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.7
common/lib/libc/atomic/atomic_and_16_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_and_8_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_16.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_32.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_8.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_cas_by_cas32.c: revision 1.4
common/lib/libc/atomic/atomic_op_namespace.h: revision 1.7
Add __sync_val_compare_and_swap_{1,2} aliases for _atomic_cas_{8,16}
--
Provide __atomic_compare_exchange_N (as needed for the C11 2011 <atomic>
ops) via the corresponding CAS.
--
Hook __atomic_compare_exchange_N into vax libc.
--
Provide __sync_and_and_fetch_2 and __sync_and_and_fetch_1 for pre-ARMv6,
they are needed for the C++ 2011 <atomic> stuff.
--
Add C++ 2011 <atomic> support functions.
--
Move the and_{16,8}_nv sources into the right (libc only) block.
--
Provide <atomic> C++ 2011 support functions for mips and sh3.
--
Provide C++ 2011 <atomic> support functions for hppa and arm.
--
Provide prototypes to fix build with clang.
 1.13.16.2 21-Apr-2020  martin Sync with HEAD
 1.13.16.1 10-Jun-2019  christos Sync with HEAD
 1.14.14.1 31-Jul-2023  martin Pull up following revision(s) (requested by riastradh in ticket #275):

common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.24
common/lib/libc/arch/m68k/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.15
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.9
common/lib/libc/atomic/atomic_is_lock_free.c: revision 1.1
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.9

libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.

For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.
 1.4 29-Jan-2014  skrll branches: 1.4.26;
Copyright
 1.3 29-Jan-2014  skrll Provide _atomic_cas_8_up and _atomic_cas_16_up functions
 1.2 16-Jan-2011  skrll branches: 1.2.6; 1.2.12;
Fix RCSId.
 1.1 11-Aug-2008  skrll branches: 1.1.2;
Provide assembly _atomic_cas_up() as the compiler cannot be trusted to
generate fully restartable code sequences.
 1.1.2.2 18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.1.2.1 11-Aug-2008  wrstuden file atomic_cas_up.S was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000
 1.2.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.2 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.1 17-Jan-2011  skrll branches: 1.1.48;
Add an membar_ops.S - not used currently.
 1.1.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.48.1 21-Apr-2020  martin Sync with HEAD
 1.8 18-Feb-2014  martin branches: 1.8.26;
Provide most missing __sync_*64 primitives for i386
 1.7 04-Jan-2009  pooka branches: 1.7.8; 1.7.14;
allow inclusion of atomic ops in librump
 1.6 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.5 11-Feb-2008  ad Only build atomic ops for libkern/libc.
 1.4 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.3 20-Dec-2007  ad branches: 1.3.2;
64-bit atomic ops for i386.
 1.2 28-Nov-2007  ad x86 atomic ops.
 1.1 17-Apr-2007  thorpej branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.
 1.1.2.1 17-Apr-2007  thorpej Add build glue for i386 atomic ops.
 1.3.2.3 23-Mar-2008  matt sync with HEAD
 1.3.2.2 09-Jan-2008  matt sync with HEAD
 1.3.2.1 20-Dec-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:20:52 +0000
 1.7.14.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.7.8.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.8.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.26.1 21-Apr-2020  martin Sync with HEAD
 1.38 06-Sep-2025  riastradh paravirt_membar_sync(9): New memory barrier.

For use in paravirtualized drivers which require store-before-load
ordering -- irrespective of whether the kernel is built for a single
processor, or whether the (virtual) machine is booted with a single
processor.

This is even required on architectures that don't even have a
store-before-load ordering barrier, like m68k; adding, e.g., a virtio
bus is _as if_ the architecture has been extended with relaxed memory
ordering when talking with that new bus. Such architectures need
some way to request the hypervisor enforce that ordering -- on m68k,
that's done by issuing a CASL instruction, which qemu maps to an
atomic r/m/w with sequential consistency ordering in the host.

PR kern/59618: occasional virtio block device lock ups/hangs
 1.37 16-Jul-2024  riastradh xen: Don't hotpatch away LOCK prefix in xen_mb, even on UP boots.

Both xen_mb and membar_sync are designed to provide store-before-load
ordering, but xen_mb has to provide it in synchronizing guest with
hypervisor, while membar_sync only has to provide it in synchronizing
one (guest) CPU with another (guest) CPU.

It is safe to hotpatch away the LOCK prefix in membar_sync on a
uniprocessor boot because membar_sync is only designed to coordinate
between normal memory on multiple CPUs, and is never necessary when
there's only one CPU involved.

But xen_mb is used to coordinate between the guest and the `device'
implemented by a hypervisor, which might be running on another
_physical_ CPU even if the NetBSD guest only sees one `CPU', i.e.,
one _virtual_ CPU. So even on `uniprocessor' boots, xen_mb must
still issue an instruction with store-before-load ordering on
multiprocessor systems, such as a LOCK ADD (or MFENCE, but MFENCE is
costlier for no benefit here).

No need to change xen_wmb (release ordering, load/store-before-store)
or xen_rmb (acquire ordering, load-before-load/store) because every
x86 store is a store-release and every x86 load is a load-acquire,
even on multiprocessor systems, so there's no hotpatching involved
anyway.

PR kern/57199
 1.36 30-Jul-2022  riastradh branches: 1.36.2; 1.36.8;
x86: Eliminate mfence hotpatch for membar_sync.

The more-compatible LOCK ADD $0,-N(%rsp) turns out to be cheaper
than MFENCE anyway. Let's save some space and maintenance and rip
out the hotpatching for it.
 1.35 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.34 09-Apr-2022  riastradh i386/membar_ops: Upgrade membar_enter from R/RW to RW/RW.

This will be deprecated soon but let's avoid leaving rakes to trip on
with it arising from disagreement over the documentation (W/RW) and
implementation and usage (R/RW).
 1.33 09-Apr-2022  riastradh x86: Add a note on membar_sync and mfence.
 1.32 09-Apr-2022  riastradh x86: Omit needless store in membar_producer/exit.

On x86, every store is a store-release, so there is no need for any
barrier. But this wasn't a barrier anyway; it was just a store,
which was redundant with the store of the return address to the stack
implied by CALL even if issuing a store made a difference.
 1.31 09-Apr-2022  riastradh x86: Every load is a load-acquire, so membar_consumer is a noop.

lfence is only needed for MD logic, such as operations on I/O memory
rather than normal cacheable memory, or special instructions like
RDTSC -- never for MI synchronization between threads/CPUs. No need
for hot-patching to do lfence here.

(The x86_lfence function might reasonably be patched on i386 to do
lfence for MD logic, but it isn't now and this doesn't change that.)
 1.30 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.29 01-May-2020  maxv Use the hotpatch framework when patching _atomic_cas_64.
 1.28 26-Apr-2020  maxv Use the hotpatch framework for LFENCE/MFENCE.
 1.27 26-Apr-2020  maxv Remove unused argument in macro.
 1.26 26-Apr-2020  maxv Remove unused.
 1.25 26-Apr-2020  maxv Drop the hardcoded array, use the hotpatch section.
 1.24 25-Apr-2020  bouyer Merge the bouyer-xenpvh branch, bringing in Xen PV drivers support under HVM
guests in GENERIC.
Xen support can be disabled at runtime with
boot -c
disable hypervisor
 1.23 18-Jul-2018  bouyer branches: 1.23.8;
On Xen, always alias _atomic_cas_64 to _atomic_cas_cx8. AFAIK Xen doesn't
support CPUs that don't support cx8.
i386 XENPAE_DOMU boots again.
 1.22 23-May-2014  uebayasi branches: 1.22.22; 1.22.24;
Put missing END() markers to set ELF symbol size.
 1.21 22-Apr-2014  christos The kernel uses 64 bit atomic ops.
 1.20 18-Feb-2014  martin branches: 1.20.2;
Provide most missing __sync_*64 primitives for i386
 1.19 12-Jan-2011  joerg branches: 1.19.6; 1.19.12;
Allow use of traditional CPP to be set on a per platform base in sys.mk.
Honour this for dependency processing in bsd.dep.mk. Switch i386 and
amd64 assembly to use ISO C90 preprocessor concat and drop the
-traditional-cpp on this platform.
 1.18 26-Nov-2009  pooka Use strong alias within the kernel namespace regardless of if we're
dealing with a hard or soft kernel (kernel linker doesn't support
weak symbols).
 1.17 02-Apr-2009  enami So that profile kernel runs again,
- Adjust the size of functions used to patch.
- Fix the jump offset of mcount call when patching functions.

Approved by Andrew Doran.
 1.16 12-Jan-2009  pooka branches: 1.16.2;
include sys/param.h for _HARDKERNEL instead of homegrown def.
 1.15 04-Jan-2009  pooka Opt for libc versions in case of _KERNEL && !_RUMPKERNEL.
(kernel version uses sti/cli and is not PIC)
 1.14 19-Dec-2008  ad PR kern/40213 my i386 machine can't boot because of tsc

- Patch in atomic_cas_64() twice. The first patch is early and makes it
the MP-atomic version available if we have cmpxchg8b. The second patch
strips the lock prefix if ncpu==1.

- Fix the i486 atomic_cas_64() to not unconditionally enable interrupts.
 1.13 25-May-2008  chs branches: 1.13.4;
enable profiling of assembly functions.
 1.12 03-May-2008  yamt branches: 1.12.2;
whitespace.
 1.11 03-May-2008  yamt rename END to ENDLABEL. i'll use END for other purpose. ok by Andrew Doran.
 1.10 28-Apr-2008  martin Remove clause 3 and 4 from TNF licenses
 1.9 10-Feb-2008  ad branches: 1.9.4;
Add atomic_cas_foo_ni().
 1.8 09-Feb-2008  ad membar_enter was doing the wrong thing. For x86 we can alias:

membar_enter -> membar_consumer
membar_exit -> membar_producer
 1.7 20-Dec-2007  ad branches: 1.7.2;
- Make __cpu_simple_lock and similar real functions and patch at runtime.
- Remove old x86 atomic ops.
- Drop text alignment back to 16 on i386 (really, this time).
- Minor cleanup.
 1.6 20-Dec-2007  ad 64-bit atomic ops for i386.
 1.5 09-Dec-2007  ad Add missing strong aliases.
 1.4 29-Nov-2007  ad atomic_add_* takes signed integers, the others take unsigned.
 1.3 28-Nov-2007  ad A lock prefix on xchg is meaningless.
 1.2 28-Nov-2007  ad Fix up a few minor problems.
 1.1 28-Nov-2007  ad x86 atomic ops.
 1.7.2.3 23-Mar-2008  matt sync with HEAD
 1.7.2.2 09-Jan-2008  matt sync with HEAD
 1.7.2.1 20-Dec-2007  matt file atomic.S was added on branch matt-armv6 on 2008-01-09 01:20:53 +0000
 1.9.4.2 04-Jun-2008  yamt sync with head
 1.9.4.1 18-May-2008  yamt sync with head.
 1.12.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.13.4.2 03-Apr-2009  snj Pull up following revision(s) (requested by enami in ticket #645):
common/lib/libc/arch/i386/atomic/atomic.S: revision 1.17
sys/arch/amd64/amd64/spl.S: revision 1.21
sys/arch/x86/x86/patch.c: revision 1.17
So that profile kernel runs again,
- Adjust the size of functions used to patch.
- Fix the jump offset of mcount call when patching functions.
Approved by Andrew Doran.
 1.13.4.1 02-Feb-2009  snj Pull up following revision(s) (requested by ad in ticket #343):
common/lib/libc/arch/i386/atomic/atomic.S: revision 1.14
sys/arch/x86/include/cpufunc.h: revision 1.9
sys/arch/x86/x86/identcpu.c: revision 1.12
sys/arch/x86/x86/cpu.c: revision 1.60
sys/arch/x86/x86/patch.c: revision 1.15
PR kern/40213 my i386 machine can't boot because of tsc
- Patch in atomic_cas_64() twice. The first patch is early and makes it
the MP-atomic version available if we have cmpxchg8b. The second patch
strips the lock prefix if ncpu==1.
- Fix the i486 atomic_cas_64() to not unconditionally enable interrupts.
 1.16.2.1 13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.19.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.19.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.20.2.1 10-Aug-2014  tls Rebase.
 1.22.24.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.22.24.2 21-Apr-2020  martin Sync with HEAD
 1.22.24.1 10-Jun-2019  christos Sync with HEAD
 1.22.22.1 28-Jul-2018  pgoyette Sync with HEAD
 1.23.8.1 14-Apr-2020  bouyer Force _atomic_cas_cx8 only for XENPV; x86_patch works fine for (PV)HVM
 1.36.8.1 02-Aug-2025  perseant Sync with HEAD
 1.36.2.1 20-Jul-2024  martin Pull up following revision(s) (requested by riastradh in ticket #764):

common/lib/libc/arch/i386/atomic/atomic.S: revision 1.37
sys/arch/xen/include/xenring.h: revision 1.8
sys/arch/i386/i386/cpufunc.S: revision 1.52
sys/arch/amd64/amd64/cpufunc.S: revision 1.68
sys/arch/xen/include/hypervisor.h: revision 1.60
common/lib/libc/arch/x86_64/atomic/atomic.S: revision 1.30

xen: Don't hotpatch away LOCK prefix in xen_mb, even on UP boots.

Both xen_mb and membar_sync are designed to provide store-before-load
ordering, but xen_mb has to provide it in synchronizing guest with
hypervisor, while membar_sync only has to provide it in synchronizing
one (guest) CPU with another (guest) CPU.

It is safe to hotpatch away the LOCK prefix in membar_sync on a
uniprocessor boot because membar_sync is only designed to coordinate
between normal memory on multiple CPUs, and is never necessary when
there's only one CPU involved.

But xen_mb is used to coordinate between the guest and the `device'
implemented by a hypervisor, which might be running on another
_physical_ CPU even if the NetBSD guest only sees one `CPU', i.e.,
one _virtual_ CPU. So even on `uniprocessor' boots, xen_mb must
still issue an instruction with store-before-load ordering on
multiprocessor systems, such as a LOCK ADD (or MFENCE, but MFENCE is
costlier for no benefit here).

No need to change xen_wmb (release ordering, load/store-before-store)
or xen_rmb (acquire ordering, load-before-load/store) because every
x86 store is a store-release and every x86 load is a load-acquire,
even on multiprocessor systems, so there's no hotpatching involved
anyway.

PR kern/57199
 1.1 16-Apr-2007  thorpej branches: 1.1.2;
file atomic_add.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 16-Apr-2007  thorpej Add atomic op implementations for x86.
 1.1 16-Apr-2007  thorpej branches: 1.1.2;
file atomic_and.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 16-Apr-2007  thorpej Add atomic op implementations for x86.
 1.1 16-Apr-2007  thorpej branches: 1.1.2;
file atomic_cas_32.S was initially added on branch thorpej-atomic.
 1.1.2.5 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.4 17-Apr-2007  thorpej Fix the end-of-function padding so both versions of cas-32 end up the
same size.
 1.1.2.3 17-Apr-2007  thorpej Give the namespace treatment to _atomic_cas_32_486(), too.
 1.1.2.2 17-Apr-2007  thorpej Add the necessary aliasa for _atomic_cas_32().
 1.1.2.1 16-Apr-2007  thorpej Add atomic op implementations for x86.
 1.1 16-Apr-2007  thorpej branches: 1.1.2;
file atomic_dec.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 16-Apr-2007  thorpej Add atomic op implementations for x86.
 1.1 16-Apr-2007  thorpej branches: 1.1.2;
file atomic_inc.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 16-Apr-2007  thorpej Add atomic op implementations for x86.
 1.1 16-Apr-2007  thorpej branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.
 1.1.2.1 16-Apr-2007  thorpej Add atomic op implementations for x86.
 1.1 16-Apr-2007  thorpej branches: 1.1.2;
file atomic_or.S was initially added on branch thorpej-atomic.
 1.1.2.3 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.2 17-Apr-2007  thorpej Fix copy-and-pasto.
 1.1.2.1 16-Apr-2007  thorpej Add atomic op implementations for x86.
 1.1 16-Apr-2007  thorpej branches: 1.1.2;
file atomic_swap.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 16-Apr-2007  thorpej Add atomic op implementations for x86.
 1.1 16-Apr-2007  thorpej branches: 1.1.2;
file membar_ops.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 16-Apr-2007  thorpej Add atomic op implementations for x86.
 1.4 23-May-2014  uebayasi branches: 1.4.24;
Put missing END() markers to set ELF symbol size.
 1.3 28-Nov-2007  ad branches: 1.3.34; 1.3.48;
Use xchgb, bswapl and not rotate/shift.
 1.2 27-Dec-2005  yamt branches: 1.2.18;
don't rename bswap{16,32} if defined(_KERNEL) || defined(_STANDALONE).
(they are hidden by gcc-optimized versions, though.)
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.18.1 09-Jan-2008  matt sync with HEAD
 1.3.48.1 10-Aug-2014  tls Rebase.
 1.3.34.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.24.1 21-Apr-2020  martin Sync with HEAD
 1.4 23-May-2014  uebayasi branches: 1.4.24;
Put missing END() markers to set ELF symbol size.
 1.3 28-Nov-2007  ad branches: 1.3.34; 1.3.48;
Use xchgb, bswapl and not rotate/shift.
 1.2 27-Dec-2005  yamt branches: 1.2.18;
don't rename bswap{16,32} if defined(_KERNEL) || defined(_STANDALONE).
(they are hidden by gcc-optimized versions, though.)
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.18.1 09-Jan-2008  matt sync with HEAD
 1.3.48.1 10-Aug-2014  tls Rebase.
 1.3.34.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.24.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 27-Jan-2020  ad x86 uses the C versions of bcmp() and memcmp() now.
 1.3 22-Mar-2014  jakllsch branches: 1.3.26; 1.3.30;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.2 12-Nov-2007  ad branches: 1.2.28; 1.2.34;
Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.
 1.1 20-Dec-2005  christos branches: 1.1.18;
Merge libkern + libc common files. As requested by core.
 1.1.18.1 09-Jan-2008  matt sync with HEAD
 1.2.34.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.28.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.30.1 29-Feb-2020  ad Sync with head.
 1.3.26.1 08-Apr-2020  martin Merge changes from current as of 20200406
 1.4 22-Mar-2014  jakllsch branches: 1.4.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.3 12-Nov-2007  ad branches: 1.3.28; 1.3.34;
Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.
 1.2 07-Apr-2006  yamt branches: 1.2.18;
prefix local labels with ".L".
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.18.1 09-Jan-2008  matt sync with HEAD
 1.3.34.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.28.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.26.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.5 23-May-2014  uebayasi branches: 1.5.24;
Put missing END() markers to set ELF symbol size.
 1.4 29-Apr-2008  martin branches: 1.4.26; 1.4.40;
Convert to new 2 clause license
 1.3 12-Nov-2007  ad branches: 1.3.6;
Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.
 1.2 07-Apr-2006  yamt branches: 1.2.18;
prefix local labels with ".L".
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.18.1 09-Jan-2008  matt sync with HEAD
 1.3.6.1 18-May-2008  yamt sync with head.
 1.4.40.1 10-Aug-2014  tls Rebase.
 1.4.26.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.24.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 22-Mar-2014  jakllsch branches: 1.3.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.2 17-Jul-2009  dsl branches: 1.2.6; 1.2.12;
Change all archs so that strchr.[cS] and strrchr.[cS] exist and generate
duplicate symbols for index() and rindex().
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.2.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.5 30-Mar-2024  andvar s/Westley/Wesley/ in a book reference (in comments).
 1.4 08-Dec-2021  andvar s/efficent/efficient/ in comments.
 1.3 04-Oct-2021  andvar remove duplicate the article in comments.
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 22-Mar-2014  jakllsch branches: 1.3.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.2 17-Jul-2009  dsl branches: 1.2.6; 1.2.12;
Change all archs so that strchr.[cS] and strrchr.[cS] exist and generate
duplicate symbols for index() and rindex().
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.2.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 16-Jun-2011  joerg branches: 1.1.2; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.
 1.1.8.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 23-May-2014  uebayasi branches: 1.2.24;
Put missing END() markers to set ELF symbol size.
 1.1 16-Jun-2011  joerg branches: 1.1.8; 1.1.20;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.
 1.1.20.1 10-Aug-2014  tls Rebase.
 1.1.8.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.24.1 21-Apr-2020  martin Sync with HEAD
 1.1 16-Jun-2011  joerg branches: 1.1.44;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.
 1.1.44.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.44.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 16-Jun-2011  joerg branches: 1.1.2; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.
 1.1.8.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 22-Sep-2014  khorben branches: 1.3.16;
Look for the character to locate before checking for the NUL character

As documented in PR port-i386/49208, this fixes strchr(s, '\0'), as used by
the FAT first-stage bootloader on x86 (bootxx_msdos).
strchr(s, '\0') is otherwise equivalent to strlen(string), which would
probably look nicer in the original file, dosfs.c from libsa.

Confirmed working in qemu and on real hardware.
ok joerg@

XXX pull-up to netbsd-6 and netbsd-7
 1.2 22-Mar-2014  jakllsch branches: 1.2.4;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 16-Jun-2011  joerg branches: 1.1.2; 1.1.4; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.
 1.1.8.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.4.1 27-Oct-2014  msaitoh Pull up following revision(s) (requested by khorben in ticket #1158):
common/lib/libc/arch/i386/string/small/strchr.S: revision 1.3
Look for the character to locate before checking for the NUL character
As documented in PR port-i386/49208, this fixes strchr(s, '\0'), as used by
the FAT first-stage bootloader on x86 (bootxx_msdos).
strchr(s, '\0') is otherwise equivalent to strlen(string), which would
probably look nicer in the original file, dosfs.c from libsa.
Confirmed working in qemu and on real hardware.
ok joerg@
XXX pull-up to netbsd-6 and netbsd-7
 1.1.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 29-Sep-2014  msaitoh Pull up following revision(s) (requested by khorben in ticket #118):
common/lib/libc/arch/i386/string/small/strchr.S: revision 1.3
Look for the character to locate before checking for the NUL character
As documented in PR port-i386/49208, this fixes strchr(s, '\0'), as used by
the FAT first-stage bootloader on x86 (bootxx_msdos).
strchr(s, '\0') is otherwise equivalent to strlen(string), which would
probably look nicer in the original file, dosfs.c from libsa.
Confirmed working in qemu and on real hardware.
ok joerg@
XXX pull-up to netbsd-6 and netbsd-7
 1.3.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 16-Jun-2011  joerg branches: 1.1.2; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.
 1.1.8.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 16-Jun-2011  joerg branches: 1.1.2; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.
 1.1.8.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all i386 string assembly functions that don't overlap use END() so
that symbol size information is available.
 1.1 16-Jun-2011  joerg branches: 1.1.2; 1.1.8;
Add support for size optimised versions of the assembler functions in
the small subdirectory. Provide more compact versions of the functions
used by bootxx in x86.
 1.1.8.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 04-Jan-2009  pooka branches: 1.3.50;
allow inclusion of atomic ops in librump
 1.2 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.1 20-Mar-2008  kochi branches: 1.1.2; 1.1.4;
Add ia64 MI atomic ops.
(Not tested at all, just for reference)
 1.1.4.2 24-Mar-2008  keiichi sync with head.
 1.1.4.1 20-Mar-2008  keiichi file Makefile.inc was added on branch keiichi-mipv6 on 2008-03-24 07:14:29 +0000
 1.1.2.2 23-Mar-2008  matt sync with HEAD
 1.1.2.1 20-Mar-2008  matt file Makefile.inc was added on branch matt-armv6 on 2008-03-23 00:12:41 +0000
 1.3.50.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.50.1 21-Apr-2020  martin Sync with HEAD
 1.7 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.6 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.5 09-Nov-2009  skrll branches: 1.5.48;
New binutils seems to want end of functions marked.
 1.4 25-May-2008  chs enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 24-Mar-2008  kochi branches: 1.2.2;
Add atomic_cas_foo_ni alias.
 1.1 20-Mar-2008  kochi branches: 1.1.2; 1.1.4;
Add ia64 MI atomic ops.
(Not tested at all, just for reference)
 1.1.4.2 24-Mar-2008  keiichi sync with head.
 1.1.4.1 20-Mar-2008  keiichi file atomic.S was added on branch keiichi-mipv6 on 2008-03-24 07:14:29 +0000
 1.1.2.2 23-Mar-2008  matt sync with HEAD
 1.1.2.1 20-Mar-2008  matt file atomic.S was added on branch matt-armv6 on 2008-03-23 00:12:41 +0000
 1.2.2.2 04-Jun-2008  yamt sync with head
 1.2.2.1 18-May-2008  yamt sync with head.
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.5.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.48.1 21-Apr-2020  martin Sync with HEAD
 1.1 05-Aug-2016  scole branches: 1.1.2; 1.1.16;
PR port-ia64/51261

Imported from FreeBSD with a minor change or two
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1.2.2 06-Aug-2016  pgoyette Sync with HEAD
 1.1.2.1 05-Aug-2016  pgoyette file bcopy.S was added on branch pgoyette-localcount on 2016-08-06 00:18:38 +0000
 1.1 05-Aug-2016  scole branches: 1.1.2; 1.1.16;
PR port-ia64/51261

Imported from FreeBSD with a minor change or two
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1.2.2 06-Aug-2016  pgoyette Sync with HEAD
 1.1.2.1 05-Aug-2016  pgoyette file bzero.S was added on branch pgoyette-localcount on 2016-08-06 00:18:38 +0000
 1.1 05-Aug-2016  scole branches: 1.1.2; 1.1.16;
PR port-ia64/51261

Imported from FreeBSD with a minor change or two
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1.2.2 06-Aug-2016  pgoyette Sync with HEAD
 1.1.2.1 05-Aug-2016  pgoyette file ffs.S was added on branch pgoyette-localcount on 2016-08-06 00:18:38 +0000
 1.1 05-Aug-2016  scole branches: 1.1.2; 1.1.16;
PR port-ia64/51261

Imported from FreeBSD with a minor change or two
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1.2.2 06-Aug-2016  pgoyette Sync with HEAD
 1.1.2.1 05-Aug-2016  pgoyette file memcpy.S was added on branch pgoyette-localcount on 2016-08-06 00:18:38 +0000
 1.1 05-Aug-2016  scole branches: 1.1.2; 1.1.16;
PR port-ia64/51261

Imported from FreeBSD with a minor change or two
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1.2.2 06-Aug-2016  pgoyette Sync with HEAD
 1.1.2.1 05-Aug-2016  pgoyette file memmove.S was added on branch pgoyette-localcount on 2016-08-06 00:18:38 +0000
 1.16 30-Mar-2023  riastradh libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.
For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.

XXX pullup-10
 1.15 28-Feb-2019  isaki branches: 1.15.14;
Add missing atomic_and_{8,16}_nv_cas.c for __sync_and_and_fetch_{1,2}.
XXX why is not only atomic_and_* symmetric unlike the others?
(in common/lib/libc/atomic/)
 1.14 20-Nov-2014  matt branches: 1.14.16;
Add atomic_c11_compare_exchange_cas_{8,16,32}.c
 1.13 22-Feb-2014  martin Add missing __sunc_* ops for sun2
 1.12 20-Feb-2014  martin Add missing __sync_* ops
 1.11 29-Jan-2014  martin Provide _atomic_cas_8_up and _atomic_cas_16_up as assembler functions
 1.10 18-Jul-2013  matt invert tests ${MACHINE} != "m68000" -> ${MACHINE} == "m68k"
 1.9 04-Jan-2009  pooka branches: 1.9.8; 1.9.14; 1.9.18;
allow inclusion of atomic ops in librump
 1.8 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.7 29-Apr-2008  scw Implement _atomic_cas_up() in assembly code as the compiler cannot be
trusted to generate fully restartable code sequences.

Addresses lib/38482 for ARM and m68000.
 1.6 06-Apr-2008  tsutsui branches: 1.6.2;
Fix pasto.
 1.5 05-Apr-2008  tsutsui Use .c version atomic ops for m68010.
 1.4 11-Feb-2008  ad Only build atomic ops for libkern/libc.
 1.3 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.2 29-Nov-2007  ad branches: 1.2.4;
Pull in m68k atomic ops from the thorpej-atomic branch.
 1.1 22-Apr-2007  thorpej branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.
 1.1.2.1 22-Apr-2007  thorpej Atomic ops for 68020 and better processors. 68010 will need to use
something else for anything that uses the CAS instruction.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 29-Nov-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:20:54 +0000
 1.6.2.1 18-May-2008  yamt sync with head.
 1.9.18.1 23-Jul-2013  riastradh sync with HEAD
 1.9.14.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.9.8.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.14.16.2 21-Apr-2020  martin Sync with HEAD
 1.14.16.1 10-Jun-2019  christos Sync with HEAD
 1.15.14.1 31-Jul-2023  martin Pull up following revision(s) (requested by riastradh in ticket #275):

common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.24
common/lib/libc/arch/m68k/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.15
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.9
common/lib/libc/atomic/atomic_is_lock_free.c: revision 1.1
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.9

libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.

For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.
 1.11 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.10 23-Jun-2014  joerg branches: 1.10.24;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.9 22-Feb-2014  martin branches: 1.9.2;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.8 20-Feb-2014  martin Add missing __sync_* ops
 1.7 18-Feb-2014  martin Add a few __sync_* and atomic functions.
 1.6 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.5 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.4 25-May-2008  chs branches: 1.4.2; 1.4.24;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 29-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in m68k atomic ops from the thorpej-atomic branch.
 1.1 22-Apr-2007  thorpej branches: 1.1.2;
file atomic_add.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 22-Apr-2007  thorpej Atomic ops for 68020 and better processors. 68010 will need to use
something else for anything that uses the CAS instruction.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 29-Nov-2007  matt file atomic_add.S was added on branch matt-armv6 on 2008-01-09 01:20:55 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.24.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.9.2.1 10-Aug-2014  tls Rebase.
 1.10.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.10.24.1 21-Apr-2020  martin Sync with HEAD
 1.12 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.11 23-Jun-2014  joerg branches: 1.11.24;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.10 22-Feb-2014  martin branches: 1.10.2;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.9 20-Feb-2014  martin Add missing __sync_* ops
 1.8 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.7 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.6 25-May-2008  chs branches: 1.6.2; 1.6.24;
enable profiling of assembly functions.
 1.5 28-Apr-2008  martin branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses
 1.4 12-Jan-2008  ad branches: 1.4.4;
atomic_and/atomic_or don't operate on pointers, so don't copy the return
to a0. From thorpej@.
 1.3 29-Nov-2007  ad branches: 1.3.4;
Apparently pointers are also returned in %a0.
 1.2 29-Nov-2007  ad Pull in m68k atomic ops from the thorpej-atomic branch.
 1.1 22-Apr-2007  thorpej branches: 1.1.2;
file atomic_and.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 22-Apr-2007  thorpej Atomic ops for 68020 and better processors. 68010 will need to use
something else for anything that uses the CAS instruction.
 1.3.4.3 23-Mar-2008  matt sync with HEAD
 1.3.4.2 09-Jan-2008  matt sync with HEAD
 1.3.4.1 29-Nov-2007  matt file atomic_and.S was added on branch matt-armv6 on 2008-01-09 01:20:55 +0000
 1.4.4.2 04-Jun-2008  yamt sync with head
 1.4.4.1 18-May-2008  yamt sync with head.
 1.5.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.6.24.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.10.2.1 10-Aug-2014  tls Rebase.
 1.11.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.11.24.1 21-Apr-2020  martin Sync with HEAD
 1.14 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.13 10-Jul-2015  mlelstv branches: 1.13.16;
fix comment
 1.12 04-Jul-2015  isaki Improve the code in __sync_bool_compare_and_swap_{1,2,4}.
- bccs is smaller and faster than bcc(.w) in this case.
- it can be used movql in this case (assembler optimise it though).
 1.11 04-Jul-2015  isaki atomic_cas_{8,16}:
- Correct the wrong offset in stack.
__sync_bool_compare_and_swap_{1,2,4}:
- Correct the wrong offset in stack.
- D3 must be preserved in subroutines.
PR/49995.
 1.10 22-Feb-2014  martin branches: 1.10.4;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.9 20-Feb-2014  martin Simplify the variants returning a bool
 1.8 18-Feb-2014  martin Add a few __sync_* and atomic functions.
 1.7 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.6 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.5 25-May-2008  chs branches: 1.5.2; 1.5.24;
enable profiling of assembly functions.
 1.4 28-Apr-2008  martin branches: 1.4.2;
Remove clause 3 and 4 from TNF licenses
 1.3 10-Feb-2008  ad branches: 1.3.4;
Add atomic_cas_foo_ni().
 1.2 29-Nov-2007  ad branches: 1.2.4;
Pull in m68k atomic ops from the thorpej-atomic branch.
 1.1 22-Apr-2007  thorpej branches: 1.1.2;
file atomic_cas.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 22-Apr-2007  thorpej Atomic ops for 68020 and better processors. 68010 will need to use
something else for anything that uses the CAS instruction.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 29-Nov-2007  matt file atomic_cas.S was added on branch matt-armv6 on 2008-01-09 01:20:56 +0000
 1.3.4.2 04-Jun-2008  yamt sync with head
 1.3.4.1 18-May-2008  yamt sync with head.
 1.4.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.5.24.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.10.4.1 16-Jul-2015  snj Pull up following revision(s) (requested by isaki in ticket #865):
common/lib/libc/arch/m68k/atomic/atomic_cas.S: revisions 1.11, 1.12
atomic_cas_{8,16}:
- Correct the wrong offset in stack.
__sync_bool_compare_and_swap_{1,2,4}:
- Correct the wrong offset in stack.
- D3 must be preserved in subroutines.
PR/49995.
--
Improve the code in __sync_bool_compare_and_swap_{1,2,4}.
- bccs is smaller and faster than bcc(.w) in this case.
- it can be used movql in this case (assembler optimise it though).
 1.13.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.13.16.1 21-Apr-2020  martin Sync with HEAD
 1.7 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.6 29-Jan-2014  martin branches: 1.6.26;
Provide _atomic_cas_8_up and _atomic_cas_16_up as assembler functions
 1.5 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.4 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.3 25-May-2008  chs branches: 1.3.2; 1.3.24;
enable profiling of assembly functions.
 1.2 30-Apr-2008  tsutsui branches: 1.2.2; 1.2.4;
Make this compile (missing register prefix).
 1.1 29-Apr-2008  scw Implement _atomic_cas_up() in assembly code as the compiler cannot be
trusted to generate fully restartable code sequences.

Addresses lib/38482 for ARM and m68000.
 1.2.4.3 04-Jun-2008  yamt sync with head
 1.2.4.2 18-May-2008  yamt sync with head.
 1.2.4.1 30-Apr-2008  yamt file atomic_cas_68000.S was added on branch yamt-pf42 on 2008-05-18 12:28:45 +0000
 1.2.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.3.24.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.7 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.6 16-Jul-2013  matt branches: 1.6.28;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.5 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.4 25-May-2008  chs branches: 1.4.2; 1.4.24;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 29-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in m68k atomic ops from the thorpej-atomic branch.
 1.1 22-Apr-2007  thorpej branches: 1.1.2;
file atomic_dec.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 22-Apr-2007  thorpej Atomic ops for 68020 and better processors. 68010 will need to use
something else for anything that uses the CAS instruction.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 29-Nov-2007  matt file atomic_dec.S was added on branch matt-armv6 on 2008-01-09 01:20:56 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.24.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.28.1 21-Apr-2020  martin Sync with HEAD
 1.7 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.6 16-Jul-2013  matt branches: 1.6.28;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.5 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.4 25-May-2008  chs branches: 1.4.2; 1.4.24;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 29-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in m68k atomic ops from the thorpej-atomic branch.
 1.1 22-Apr-2007  thorpej branches: 1.1.2;
file atomic_inc.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 22-Apr-2007  thorpej Atomic ops for 68020 and better processors. 68010 will need to use
something else for anything that uses the CAS instruction.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 29-Nov-2007  matt file atomic_inc.S was added on branch matt-armv6 on 2008-01-09 01:20:57 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.24.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.28.1 21-Apr-2020  martin Sync with HEAD
 1.3 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.2 22-Feb-2014  martin branches: 1.2.4; 1.2.8; 1.2.30;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.1 20-Feb-2014  martin Add missing __sync_* ops
 1.2.30.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.30.1 21-Apr-2020  martin Sync with HEAD
 1.2.8.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.8.1 22-Feb-2014  tls file atomic_nand.S was added on branch tls-maxphys on 2014-08-19 23:45:13 +0000
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 22-Feb-2014  yamt file atomic_nand.S was added on branch yamt-pagecache on 2014-05-22 11:26:29 +0000
 1.6 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.5 23-Feb-2014  martin branches: 1.5.26;
Remove misplaced #endif
 1.4 22-Feb-2014  martin Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.3 28-Apr-2008  martin branches: 1.3.4; 1.3.26;
Remove clause 3 and 4 from TNF licenses
 1.2 29-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in m68k atomic ops from the thorpej-atomic branch.
 1.1 22-Apr-2007  thorpej branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.
 1.1.2.1 22-Apr-2007  thorpej Atomic ops for 68020 and better processors. 68010 will need to use
something else for anything that uses the CAS instruction.
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 29-Nov-2007  matt file atomic_op_asm.h was added on branch matt-armv6 on 2008-01-09 01:20:57 +0000
 1.3.26.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.12 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.11 23-Jun-2014  joerg branches: 1.11.24;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.10 22-Feb-2014  martin branches: 1.10.2;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.9 20-Feb-2014  martin Add missing __sync_* ops
 1.8 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.7 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.6 25-May-2008  chs branches: 1.6.2; 1.6.24;
enable profiling of assembly functions.
 1.5 28-Apr-2008  martin branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses
 1.4 12-Jan-2008  ad branches: 1.4.4;
atomic_and/atomic_or don't operate on pointers, so don't copy the return
to a0. From thorpej@.
 1.3 29-Nov-2007  ad branches: 1.3.4;
Apparently pointers are also returned in %a0.
 1.2 29-Nov-2007  ad Pull in m68k atomic ops from the thorpej-atomic branch.
 1.1 22-Apr-2007  thorpej branches: 1.1.2;
file atomic_or.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 22-Apr-2007  thorpej Atomic ops for 68020 and better processors. 68010 will need to use
something else for anything that uses the CAS instruction.
 1.3.4.3 23-Mar-2008  matt sync with HEAD
 1.3.4.2 09-Jan-2008  matt sync with HEAD
 1.3.4.1 29-Nov-2007  matt file atomic_or.S was added on branch matt-armv6 on 2008-01-09 01:20:57 +0000
 1.4.4.2 04-Jun-2008  yamt sync with head
 1.4.4.1 18-May-2008  yamt sync with head.
 1.5.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.6.24.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.10.2.1 10-Aug-2014  tls Rebase.
 1.11.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.11.24.1 21-Apr-2020  martin Sync with HEAD
 1.4 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 22-Feb-2014  martin branches: 1.2.2; 1.2.4;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.1 20-Feb-2014  martin Add missing __sync_* ops
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 22-Feb-2014  yamt file atomic_sub.S was added on branch yamt-pagecache on 2014-05-22 11:26:29 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_sub.S was added on branch tls-maxphys on 2014-08-19 23:45:13 +0000
 1.10 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.9 28-Jun-2014  joerg branches: 1.9.24;
Add aliases for the C11/C++11 spelling of the CAS primitives.
 1.8 22-Feb-2014  martin branches: 1.8.2;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.7 18-Feb-2014  martin Add a few __sync_* and atomic functions.
 1.6 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.5 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.4 25-May-2008  chs branches: 1.4.2; 1.4.24;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 29-Nov-2007  ad branches: 1.2.4; 1.2.8;
Pull in m68k atomic ops from the thorpej-atomic branch.
 1.1 22-Apr-2007  thorpej branches: 1.1.2;
file atomic_swap.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 22-Apr-2007  thorpej Atomic ops for 68020 and better processors. 68010 will need to use
something else for anything that uses the CAS instruction.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 29-Nov-2007  matt file atomic_swap.S was added on branch matt-armv6 on 2008-01-09 01:20:58 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.24.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.8.2.1 10-Aug-2014  tls Rebase.
 1.9.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.9.24.1 21-Apr-2020  martin Sync with HEAD
 1.4 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.3 23-Jun-2014  joerg branches: 1.3.4; 1.3.26;
Add aliases for the builtins used to implement C11/C++11 atomics.
 1.2 22-Feb-2014  martin branches: 1.2.2; 1.2.4;
Try to hide the C runtime implementation specific __sync_* ops from librump,
to avoid duplicates.
 1.1 20-Feb-2014  martin Add missing __sync_* ops
 1.2.4.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.4.1 22-Feb-2014  yamt file atomic_xor.S was added on branch yamt-pagecache on 2014-05-22 11:26:29 +0000
 1.2.2.1 10-Aug-2014  tls Rebase.
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.3.4.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.4.1 23-Jun-2014  tls file atomic_xor.S was added on branch tls-maxphys on 2014-08-19 23:45:13 +0000
 1.6 18-Mar-2014  riastradh branches: 1.6.26;
Merge riastradh-drm2 to HEAD.
 1.5 18-Jul-2013  matt Adjust for Coldfire
 1.4 16-Jul-2013  matt branches: 1.4.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 19-Sep-2007  he branches: 1.2.28; 1.2.34;
Improve consistency between <sys/bswap.h> and what's actually
defined by the C library for the various m68k ports by borrowing
some #ifs from the i386 port.

Also, align sun2 with the other m68k ports as to whether they
define __bswap{16,32} or bswap{16,32} in their C library (all
now define the __ variants).

This should make the m68k ports build sys/rump again, except
for sun2 which hits another problem later on.
 1.1 20-Dec-2005  christos branches: 1.1.18;
Merge libkern + libc common files. As requested by core.
 1.1.18.1 06-Nov-2007  matt sync with HEAD
 1.2.34.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.28.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.1 23-Jul-2013  riastradh sync with HEAD
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.6 18-Mar-2014  riastradh branches: 1.6.26;
Merge riastradh-drm2 to HEAD.
 1.5 18-Jul-2013  matt Adjust for Coldfire
 1.4 16-Jul-2013  matt branches: 1.4.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 19-Sep-2007  he branches: 1.2.28; 1.2.34;
Improve consistency between <sys/bswap.h> and what's actually
defined by the C library for the various m68k ports by borrowing
some #ifs from the i386 port.

Also, align sun2 with the other m68k ports as to whether they
define __bswap{16,32} or bswap{16,32} in their C library (all
now define the __ variants).

This should make the m68k ports build sys/rump again, except
for sun2 which hits another problem later on.
 1.1 20-Dec-2005  christos branches: 1.1.18;
Merge libkern + libc common files. As requested by core.
 1.1.18.1 06-Nov-2007  matt sync with HEAD
 1.2.34.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.28.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.1 23-Jul-2013  riastradh sync with HEAD
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.5 18-Mar-2014  riastradh branches: 1.5.26;
Merge riastradh-drm2 to HEAD.
 1.4 18-Jul-2013  matt Adjust for Coldfire
 1.3 16-Jul-2013  matt branches: 1.3.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.2 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.2.1 23-Jul-2013  riastradh sync with HEAD
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.6 22-Apr-2020  rin Restrict usage of m68k assembler versions of {,u}divsi3 and {,u}divsi3 to
kernel and bootloader for 68010.

They requires a special calling convention to udivsi3, and cannot to be
mixed up in normal routines provided by libgcc or compiler_rt. Although,
there's no problem for using them in a controlled situation, i.e., kernel
and standalone programs.

Note that this does not affect at all m68k ports other than sun2, since
codes generated by gcc do not call these routines.

Assembler files are moved from common/lib/libc/arch/m68k/gen to
sys/lib/libkern/arch/m68k in order not to be compiled in libc.

Revert hack introduced to lib/libc/compiler_rt/Makefile.inc rev 1.37:
http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/compiler_rt/Makefile.inc#rev1.37

Proposed on port-sun2@ with no response...
(Again, this does not affect m68k ports other than sun2.)
http://mail-index.netbsd.org/port-sun2/2020/03/10/msg000102.html
 1.5 07-Sep-2013  chs branches: 1.5.26; 1.5.34;
switch to ELF naming for local labels.
 1.4 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 05-Jan-2006  he branches: 1.2.50; 1.2.56;
Add the m68010 versions of the integer multiply divide, modulo,
and multiply functions from the old libkern. The m68010 versions
are only compiled when __mc68010__ is defined, presently that's
only done when building for sun2.

Reviewed by christos.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.5.34.1 25-Apr-2020  bouyer Sync with HEAD
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.8 22-Apr-2020  rin Restrict usage of m68k assembler versions of {,u}divsi3 and {,u}divsi3 to
kernel and bootloader for 68010.

They requires a special calling convention to udivsi3, and cannot to be
mixed up in normal routines provided by libgcc or compiler_rt. Although,
there's no problem for using them in a controlled situation, i.e., kernel
and standalone programs.

Note that this does not affect at all m68k ports other than sun2, since
codes generated by gcc do not call these routines.

Assembler files are moved from common/lib/libc/arch/m68k/gen to
sys/lib/libkern/arch/m68k in order not to be compiled in libc.

Revert hack introduced to lib/libc/compiler_rt/Makefile.inc rev 1.37:
http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/compiler_rt/Makefile.inc#rev1.37

Proposed on port-sun2@ with no response...
(Again, this does not affect m68k ports other than sun2.)
http://mail-index.netbsd.org/port-sun2/2020/03/10/msg000102.html
 1.7 04-Mar-2014  matt branches: 1.7.26; 1.7.34;
Make this coldfire compatible
 1.6 07-Sep-2013  chs switch to ELF naming for local labels.
 1.5 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.4 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.3 13-Jan-2006  christos branches: 1.3.50; 1.3.56;
fix the mc68010 case.
 1.2 05-Jan-2006  he Add the m68010 versions of the integer multiply divide, modulo,
and multiply functions from the old libkern. The m68010 versions
are only compiled when __mc68010__ is defined, presently that's
only done when building for sun2.

Reviewed by christos.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.3.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.7.34.1 25-Apr-2020  bouyer Sync with HEAD
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 31-May-2020  rin Fix typo in comment.
 1.1 31-May-2020  rin Add m68k assembler version of __muldi3().

This is intended for 68060:
- GCC does not emit __muldi3() for 68020-40, that have 32 * 32 --> 64 mulul
- mulsl (and moveml), used in this code, are not implemented for 68010

In comparison with that from compiler_rt, this version saves:
- 12% of processing time
- 12 bytes of stack
- 50 bytes of code size
Also, slightly faster, memory saving, and smaller than libgcc version.

By examining with evcnt(9), __muldi3() is invoked more than 1000 times per
sec by kernel, which should justify to introduce assembler version of this
function.
 1.5 09-Mar-2020  rin Remove wrong comment (copy-paste from somewhere);
__mulsi3 does not depend on __udivsi3.
 1.4 16-Jul-2013  matt branches: 1.4.28;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 05-Jan-2006  he branches: 1.2.50; 1.2.56;
Add the m68010 versions of the integer multiply divide, modulo,
and multiply functions from the old libkern. The m68010 versions
are only compiled when __mc68010__ is defined, presently that's
only done when building for sun2.

Reviewed by christos.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.28.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.28.2 21-Apr-2020  martin Sync with HEAD
 1.4.28.1 08-Apr-2020  martin Merge changes from current as of 20200406
 1.5 22-Apr-2020  rin Restrict usage of m68k assembler versions of {,u}divsi3 and {,u}divsi3 to
kernel and bootloader for 68010.

They requires a special calling convention to udivsi3, and cannot to be
mixed up in normal routines provided by libgcc or compiler_rt. Although,
there's no problem for using them in a controlled situation, i.e., kernel
and standalone programs.

Note that this does not affect at all m68k ports other than sun2, since
codes generated by gcc do not call these routines.

Assembler files are moved from common/lib/libc/arch/m68k/gen to
sys/lib/libkern/arch/m68k in order not to be compiled in libc.

Revert hack introduced to lib/libc/compiler_rt/Makefile.inc rev 1.37:
http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/compiler_rt/Makefile.inc#rev1.37

Proposed on port-sun2@ with no response...
(Again, this does not affect m68k ports other than sun2.)
http://mail-index.netbsd.org/port-sun2/2020/03/10/msg000102.html
 1.4 16-Jul-2013  matt branches: 1.4.28; 1.4.36;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 05-Jan-2006  he branches: 1.2.50; 1.2.56;
Add the m68010 versions of the integer multiply divide, modulo,
and multiply functions from the old libkern. The m68010 versions
are only compiled when __mc68010__ is defined, presently that's
only done when building for sun2.

Reviewed by christos.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.36.1 25-Apr-2020  bouyer Sync with HEAD
 1.4.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.28.1 21-Apr-2020  martin Sync with HEAD
 1.6 22-Apr-2020  rin Restrict usage of m68k assembler versions of {,u}divsi3 and {,u}divsi3 to
kernel and bootloader for 68010.

They requires a special calling convention to udivsi3, and cannot to be
mixed up in normal routines provided by libgcc or compiler_rt. Although,
there's no problem for using them in a controlled situation, i.e., kernel
and standalone programs.

Note that this does not affect at all m68k ports other than sun2, since
codes generated by gcc do not call these routines.

Assembler files are moved from common/lib/libc/arch/m68k/gen to
sys/lib/libkern/arch/m68k in order not to be compiled in libc.

Revert hack introduced to lib/libc/compiler_rt/Makefile.inc rev 1.37:
http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/compiler_rt/Makefile.inc#rev1.37

Proposed on port-sun2@ with no response...
(Again, this does not affect m68k ports other than sun2.)
http://mail-index.netbsd.org/port-sun2/2020/03/10/msg000102.html
 1.5 04-Mar-2014  matt branches: 1.5.26; 1.5.34;
Make this work with coldfire.
 1.4 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 05-Jan-2006  he branches: 1.2.50; 1.2.56;
Add the m68010 versions of the integer multiply divide, modulo,
and multiply functions from the old libkern. The m68010 versions
are only compiled when __mc68010__ is defined, presently that's
only done when building for sun2.

Reviewed by christos.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.5.34.1 25-Apr-2020  bouyer Sync with HEAD
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 16-Jul-2013  matt branches: 1.3.28;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.2 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.28.1 21-Apr-2020  martin Sync with HEAD
 1.3 16-Jul-2013  matt branches: 1.3.28;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.2 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.28.1 21-Apr-2020  martin Sync with HEAD
 1.3 16-Jul-2013  matt branches: 1.3.28;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.2 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.28.1 21-Apr-2020  martin Sync with HEAD
 1.3 16-Jul-2013  matt branches: 1.3.28;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.2 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.28.1 21-Apr-2020  martin Sync with HEAD
 1.6 07-Sep-2013  chs branches: 1.6.26;
switch to ELF naming for local labels.
 1.5 18-Jul-2013  matt Adjust for Coldfire
 1.4 16-Jul-2013  matt branches: 1.4.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 28-Apr-2008  martin branches: 1.2.4; 1.2.26;
Remove clause 3 and 4 from TNF licenses
 1.1 20-Dec-2005  christos branches: 1.1.26;
Merge libkern + libc common files. As requested by core.
 1.1.26.1 18-May-2008  yamt sync with head.
 1.2.26.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.1 23-Jul-2013  riastradh sync with HEAD
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.6 07-Sep-2013  chs branches: 1.6.26;
switch to ELF naming for local labels.
 1.5 18-Jul-2013  matt Adjust for Coldfire
 1.4 16-Jul-2013  matt branches: 1.4.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 28-Apr-2008  martin branches: 1.2.4; 1.2.26;
Remove clause 3 and 4 from TNF licenses
 1.1 20-Dec-2005  christos branches: 1.1.26;
Merge libkern + libc common files. As requested by core.
 1.1.26.1 18-May-2008  yamt sync with head.
 1.2.26.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.1 23-Jul-2013  riastradh sync with HEAD
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.6 07-Sep-2013  chs branches: 1.6.26;
switch to ELF naming for local labels.
 1.5 18-Jul-2013  matt Adjust for Coldfire
 1.4 16-Jul-2013  matt branches: 1.4.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 28-Apr-2008  martin branches: 1.2.4; 1.2.26;
Remove clause 3 and 4 from TNF licenses
 1.1 20-Dec-2005  christos branches: 1.1.26;
Merge libkern + libc common files. As requested by core.
 1.1.26.1 18-May-2008  yamt sync with head.
 1.2.26.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.1 23-Jul-2013  riastradh sync with HEAD
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.8 10-Mar-2020  rin For kernel, rename ffs to __ffssi2 rather than having a weak symbol.
This enables us to load modules depended to __ffssi2.

It is difficult to deal with weak symbols consistently in in-kernel
linker. See explanation by pgoyette on tech-kern:

http://mail-index.netbsd.org/tech-kern/2020/03/09/msg026148.html

Also, we do not currently provide ffs(9) as a kernel routine.
 1.7 09-Mar-2020  rin Add missing END() for coldfire.
 1.6 07-Sep-2013  chs branches: 1.6.26;
switch to ELF naming for local labels.
 1.5 18-Jul-2013  matt Adjust for Coldfire
 1.4 16-Jul-2013  matt branches: 1.4.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 09-Jul-2011  mrg branches: 1.2.2; 1.2.8;
add a weak alias for __ffssi2
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.8.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.1 23-Jul-2013  riastradh sync with HEAD
 1.6.26.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.2 21-Apr-2020  martin Sync with HEAD
 1.6.26.1 08-Apr-2020  martin Merge changes from current as of 20200406
 1.2 17-Jul-2009  dsl Delete files that are no longer needed.
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.5 07-Sep-2013  chs branches: 1.5.26;
switch to ELF naming for local labels.
 1.4 18-Jul-2013  matt Adjust for Coldfire
 1.3 16-Jul-2013  matt branches: 1.3.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.2 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.2.1 23-Jul-2013  riastradh sync with HEAD
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.7 07-Sep-2013  chs branches: 1.7.26;
switch to ELF naming for local labels.
 1.6 19-Jul-2013  matt With these changes, they now produce identical binary as before the coldfire
changes.
 1.5 18-Jul-2013  matt Adjust for Coldfire
 1.4 16-Jul-2013  matt branches: 1.4.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 28-Apr-2008  martin branches: 1.2.4; 1.2.26;
Remove clause 3 and 4 from TNF licenses
 1.1 20-Dec-2005  christos branches: 1.1.26;
Merge libkern + libc common files. As requested by core.
 1.1.26.1 18-May-2008  yamt sync with head.
 1.2.26.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.1 23-Jul-2013  riastradh sync with HEAD
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 17-Jul-2009  dsl Delete files that are no longer needed.
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.4 07-Sep-2013  chs branches: 1.4.26;
switch to ELF naming for local labels.
 1.3 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.2 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 16-Jul-2013  matt branches: 1.4.28;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 17-Jul-2009  dsl branches: 1.2.6; 1.2.12;
Change all archs so that strchr.[cS] and strrchr.[cS] exist and generate
duplicate symbols for index() and rindex().
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.2.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.28.1 21-Apr-2020  martin Sync with HEAD
 1.7 07-Sep-2013  chs branches: 1.7.26;
switch to ELF naming for local labels.
 1.6 19-Jul-2013  matt With these changes, they now produce identical binary as before the coldfire
changes.
 1.5 18-Jul-2013  matt Adjust for Coldfire
 1.4 16-Jul-2013  matt branches: 1.4.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 28-Apr-2008  martin branches: 1.2.4; 1.2.26;
Remove clause 3 and 4 from TNF licenses
 1.1 20-Dec-2005  christos branches: 1.1.26;
Merge libkern + libc common files. As requested by core.
 1.1.26.1 18-May-2008  yamt sync with head.
 1.2.26.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.1 23-Jul-2013  riastradh sync with HEAD
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 07-Sep-2013  chs branches: 1.4.26;
switch to ELF naming for local labels.
 1.3 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.2 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 07-Sep-2013  chs branches: 1.4.26;
switch to ELF naming for local labels.
 1.3 16-Jul-2013  matt Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.2 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.26.1 21-Apr-2020  martin Sync with HEAD
 1.7 07-Sep-2013  chs branches: 1.7.26;
switch to ELF naming for local labels.
 1.6 19-Jul-2013  matt With these changes, they now produce identical binary as before the coldfire
changes.
 1.5 18-Jul-2013  matt Adjust for Coldfire
 1.4 16-Jul-2013  matt branches: 1.4.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 28-Apr-2008  martin branches: 1.2.4; 1.2.26;
Remove clause 3 and 4 from TNF licenses
 1.1 20-Dec-2005  christos branches: 1.1.26;
Merge libkern + libc common files. As requested by core.
 1.1.26.1 18-May-2008  yamt sync with head.
 1.2.26.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.1 23-Jul-2013  riastradh sync with HEAD
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.5 07-Sep-2013  chs branches: 1.5.26;
switch to ELF naming for local labels.
 1.4 18-Jul-2013  matt Adjust for Coldfire
 1.3 16-Jul-2013  matt branches: 1.3.2;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.2 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.2.1 23-Jul-2013  riastradh sync with HEAD
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 16-Jul-2013  matt branches: 1.4.28;
Add END(func)
Fix some straggling %r@<eol> to (%r)
 1.3 16-Jul-2013  matt Convert to normal Motorola syntax for addressing.
%r@ -> (%r)
%r@- -> -(%r)
%r@+ -> (%r)+
%r@(n) n(%r)
etc.
No object differences.
 1.2 17-Jul-2009  dsl branches: 1.2.6; 1.2.12;
Change all archs so that strchr.[cS] and strrchr.[cS] exist and generate
duplicate symbols for index() and rindex().
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.2.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.28.1 21-Apr-2020  martin Sync with HEAD
 1.16 30-Mar-2023  riastradh libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.
For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.

XXX pullup-10
 1.15 25-Apr-2021  christos branches: 1.15.6;
use ${MACHINE_MIPS64}
 1.14 28-Feb-2019  isaki Add missing atomic_and_{8,16}_nv_cas.c for __sync_and_and_fetch_{1,2}.
XXX why is not only atomic_and_* symmetric unlike the others?
(in common/lib/libc/atomic/)
 1.13 13-Oct-2014  martin branches: 1.13.16;
Provide <atomic> C++ 2011 support functions for mips and sh3.
 1.12 24-Feb-2014  martin branches: 1.12.4;
Provide cas_16 and cas_8 emulation via cas_32 and use that for mips64
 1.11 21-Feb-2014  martin Provide all __sync_* ops in libc.
 1.10 14-Dec-2009  matt branches: 1.10.6; 1.10.12;
Merge from matt-nb5-mips64
 1.9 04-Jan-2009  pooka allow inclusion of atomic ops in librump
 1.8 29-Sep-2008  ad branches: 1.8.8;
Allow atomic ops to be built as part of libpthread.
 1.7 30-Apr-2008  ad Assembly _atomic_cas_up() for mips. PR lib/38482.
 1.6 11-Feb-2008  ad branches: 1.6.4;
Only build atomic ops for libkern/libc.
 1.5 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.4 30-Nov-2007  ad branches: 1.4.4;
Memory barriers for MIPS.
 1.3 29-Nov-2007  ad Use the CAS-based inc/dec variants, since these CPUs don't have atomic
add in hardware (does arm?).
 1.2 29-Nov-2007  ad Make the 64-bit operations available when possible.
 1.1 29-Nov-2007  ad Atomic ops for MIPS. Use the CAS functions already provided by the kernel,
and use the generic C code to provide the rest. Unfortunatley the C code
assembles up pretty badly on MIPS but at least it will work.
 1.4.4.3 23-Mar-2008  matt sync with HEAD
 1.4.4.2 09-Jan-2008  matt sync with HEAD
 1.4.4.1 30-Nov-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:20:58 +0000
 1.6.4.1 18-May-2008  yamt sync with head.
 1.8.8.2 05-Sep-2009  matt Resolve some conflicts.
 1.8.8.1 05-Sep-2009  matt Enable the new atomic op routines on mips64e[bl].
 1.10.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.10.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.12.4.1 12-Nov-2014  snj Pull up following revision(s) (requested by martin in ticket #218):
common/lib/libc/arch/arm/atomic/Makefile.inc: revision 1.24-1.26
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.7
common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.18
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.7
common/lib/libc/atomic/atomic_and_16_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_and_8_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_16.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_32.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_8.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_cas_by_cas32.c: revision 1.4
common/lib/libc/atomic/atomic_op_namespace.h: revision 1.7
Add __sync_val_compare_and_swap_{1,2} aliases for _atomic_cas_{8,16}
--
Provide __atomic_compare_exchange_N (as needed for the C11 2011 <atomic>
ops) via the corresponding CAS.
--
Hook __atomic_compare_exchange_N into vax libc.
--
Provide __sync_and_and_fetch_2 and __sync_and_and_fetch_1 for pre-ARMv6,
they are needed for the C++ 2011 <atomic> stuff.
--
Add C++ 2011 <atomic> support functions.
--
Move the and_{16,8}_nv sources into the right (libc only) block.
--
Provide <atomic> C++ 2011 support functions for mips and sh3.
--
Provide C++ 2011 <atomic> support functions for hppa and arm.
--
Provide prototypes to fix build with clang.
 1.13.16.2 21-Apr-2020  martin Sync with HEAD
 1.13.16.1 10-Jun-2019  christos Sync with HEAD
 1.15.6.1 31-Jul-2023  martin Pull up following revision(s) (requested by riastradh in ticket #275):

common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.24
common/lib/libc/arch/m68k/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.15
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.9
common/lib/libc/atomic/atomic_is_lock_free.c: revision 1.1
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.9

libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.

For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.
 1.7 06-Aug-2020  skrll Centralise SYNC/BDSYNC in asm.h and introduce a new LLCSCSYNC and use it
before any ll/sc sequences.

Define LLSCSYNC as syncw; syncw for cnMIPS - issue two as early cnMIPS
has errat{um,a} that means the first can fail.
 1.6 01-Aug-2020  skrll Trailing whitespace
 1.5 01-Jun-2015  matt branches: 1.5.16;
Include OCTEON support for syncw and saa/saad (Store Atomic Add).
 1.4 14-Mar-2012  christos don't include <sys/cdefs.h> from assembly.
 1.3 27-Aug-2011  bouyer branches: 1.3.2;
loongson2f support:
- Add some loongson2 definitions to cpuregs.h, from OpenBSD
- Make sure that the at register is useable before every jump register
instruction (exept when register is k0 or k1) because -mfix-loongson2f-btb
needs the at register for its workaround
- add code to mips_fixup.c to handle the instructions added by
-mfix-loongson2f-btb
- Add a ls2-specific tlb miss handler: it doesn't have separate handler
for the xtlbmiss exeption.
- Fixes for some #ifdef MIPS3_LOONGSON2 assembly code (using the wrong
register)
 1.2 14-Dec-2009  matt Merge from matt-nb5-mips64
 1.1 05-Sep-2009  matt branches: 1.1.2;
file atomic_add.S was initially added on branch matt-nb5-mips64.
 1.1.2.1 05-Sep-2009  matt Add native ll/sc or lld/scd versions of the atomic ops.
 1.3.2.1 17-Apr-2012  yamt sync with head
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.6 06-Aug-2020  skrll Centralise SYNC/BDSYNC in asm.h and introduce a new LLCSCSYNC and use it
before any ll/sc sequences.

Define LLSCSYNC as syncw; syncw for cnMIPS - issue two as early cnMIPS
has errat{um,a} that means the first can fail.
 1.5 01-Aug-2020  skrll Trailing whitespace
 1.4 14-Mar-2012  christos branches: 1.4.34;
don't include <sys/cdefs.h> from assembly.
 1.3 27-Aug-2011  bouyer branches: 1.3.2;
loongson2f support:
- Add some loongson2 definitions to cpuregs.h, from OpenBSD
- Make sure that the at register is useable before every jump register
instruction (exept when register is k0 or k1) because -mfix-loongson2f-btb
needs the at register for its workaround
- add code to mips_fixup.c to handle the instructions added by
-mfix-loongson2f-btb
- Add a ls2-specific tlb miss handler: it doesn't have separate handler
for the xtlbmiss exeption.
- Fixes for some #ifdef MIPS3_LOONGSON2 assembly code (using the wrong
register)
 1.2 14-Dec-2009  matt Merge from matt-nb5-mips64
 1.1 05-Sep-2009  matt branches: 1.1.2;
file atomic_and.S was initially added on branch matt-nb5-mips64.
 1.1.2.1 05-Sep-2009  matt Add native ll/sc or lld/scd versions of the atomic ops.
 1.3.2.1 17-Apr-2012  yamt sync with head
 1.4.34.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.34.1 21-Apr-2020  martin Sync with HEAD
 1.9 27-Feb-2022  riastradh mips: Membar audit.

This change should be safe because it doesn't remove or weaken any
memory barriers, but does add, clarify, or strengthen barriers.

Goals:

- Make sure mutex_enter/exit and mutex_spin_enter/exit have
acquire/release semantics.

- New macros make maintenance easier and purpose clearer:

. SYNC_ACQ is for load-before-load/store barrier, and BDSYNC_ACQ
for a branch delay slot -- currently defined as plain sync for MP
and nothing, or nop, for UP; thus it is no weaker than SYNC and
BDSYNC as currently defined, which is syncw on Octeon, plain sync
on non-Octeon MP, and nothing/nop on UP.

It is not clear to me whether load-then-syncw or ll/sc-then-syncw
or even bare load provides load-acquire semantics on Octeon -- if
no, this will fix bugs; if yes (like it is on SPARC PSO), we can
relax SYNC_ACQ to be syncw or nothing later.

. SYNC_REL is for load/store-before-store barrier -- currently
defined as plain sync for MP and nothing for UP.

It is not clear to me whether syncw-then-store is enough for
store-release on Octeon -- if no, we can leave this as is; if
yes, we can relax SYNC_REL to be syncw on Octeon.

. SYNC_PLUNGER is there to flush clogged Cavium store buffers, and
BDSYNC_PLUNGER for a branch delay slot -- syncw on Octeon,
nothing or nop on non-Octeon.

=> This is not necessary (or, as far as I'm aware, sufficient)
for acquire semantics -- it serves only to flush store buffers
where stores might otherwise linger for hundreds of thousands
of cycles, which would, e.g., cause spin locks to be held for
unreasonably long durations.

Newerish revisions of the MIPS ISA also have finer-grained sync
variants that could be plopped in here.

Mechanism:

Insert these barriers in the right places, replacing only those where
the definition is currently equivalent, so this change is safe.

- Replace #ifdef _MIPS_ARCH_OCTEONP / syncw / #endif at the end of
atomic_cas_* by SYNC_PLUNGER, which is `sync 4' (a.k.a. syncw) if
__OCTEON__ and empty otherwise.

=> From what I can tell, __OCTEON__ is defined in at least as many
contexts as _MIPS_ARCH_OCTEONP -- i.e., there are some Octeons
with no _MIPS_ARCH_OCTEONP, but I don't know if any of them are
relevant to us or ever saw the light of day outside Cavium; we
seem to buid with `-march=octeonp' so this is unlikely to make a
difference. If it turns out that we do care, well, now there's
a central place to make the distinction for sync instructions.

- Replace post-ll/sc SYNC by SYNC_ACQ in _atomic_cas_*, which are
internal kernel versions used in sys/arch/mips/include/lock.h where
it assumes they have load-acquire semantics. Should move this to
lock.h later, since we _don't_ define __HAVE_ATOMIC_AS_MEMBAR on
MIPS and so the extra barrier might be costly.

- Insert SYNC_REL before ll/sc, and replace post-ll/sc SYNC by
SYNC_ACQ, in _ucas_*, which is used without any barriers in futex
code and doesn't mention barriers in the man page so I have to
assume it is required to be a release/acquire barrier.

- Change BDSYNC to BDSYNC_ACQ in mutex_enter and mutex_spin_enter.
This is necessary to provide load-acquire semantics -- unclear if
it was provided already by syncw on Octeon, but it seems more
likely that either (a) no sync or syncw is needed at all, or (b)
syncw is not enough and sync is needed, since syncw is only a
store-before-store ordering barrier.

- Insert SYNC_REL before ll/sc in mutex_exit and mutex_spin_exit.
This is currently redundant with the SYNC already there, but
SYNC_REL more clearly identifies the necessary semantics in case we
want to define it differently on different systems, and having a
sync in the middle of an ll/sc is a bit weird and possibly not a
good idea, so I intend to (carefully) remove the redundant SYNC in
a later change.

- Change BDSYNC to BDSYNC_PLUNGER at the end of mutex_exit. This has
no semantic change right now -- it's syncw on Octeon, sync on
non-Octeon MP, nop on UP -- but we can relax it later to nop on
non-Cavium MP.

- Leave LLSCSYNC in for now -- it is apparently there for a Cavium
erratum, but I'm not sure what the erratum is, exactly, and I have
no reference for it. I suspect these can be safely removed, but we
might have to double up some other syncw instructions -- Linux uses
it only in store-release sequences, not at the head of every ll/sc.
 1.8 06-Aug-2020  skrll Centralise SYNC/BDSYNC in asm.h and introduce a new LLCSCSYNC and use it
before any ll/sc sequences.

Define LLSCSYNC as syncw; syncw for cnMIPS - issue two as early cnMIPS
has errat{um,a} that means the first can fail.
 1.7 01-Aug-2020  skrll Trailing whitespace
 1.6 20-Feb-2019  rin Export atomic_cas_32_ni in a similar manner to its 64-bit counterpart.

Compile test only, but seems trivial enough for me.

Fix build error due to test/lib/libc/atomic/t_atomic_cas.

Note that mips32 does not use atomic_cas.S.
 1.5 19-Feb-2019  martin Add atomic_cas_64_ni alias
 1.4 01-Jun-2015  matt branches: 1.4.16;
Include OCTEON support for syncw and saa/saad (Store Atomic Add).
 1.3 14-Mar-2012  christos don't include <sys/cdefs.h> from assembly.
 1.2 14-Dec-2009  matt branches: 1.2.6;
Merge from matt-nb5-mips64
 1.1 05-Sep-2009  matt branches: 1.1.2;
file atomic_cas.S was initially added on branch matt-nb5-mips64.
 1.1.2.1 05-Sep-2009  matt Add native ll/sc or lld/scd versions of the atomic ops.
 1.2.6.1 17-Apr-2012  yamt sync with head
 1.4.16.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.16.2 21-Apr-2020  martin Sync with HEAD
 1.4.16.1 10-Jun-2019  christos Sync with HEAD
 1.3 01-Aug-2020  skrll Trailing whitespace
 1.2 25-May-2008  chs branches: 1.2.62;
enable profiling of assembly functions.
 1.1 30-Apr-2008  ad branches: 1.1.2; 1.1.4;
Assembly _atomic_cas_up() for mips. PR lib/38482.
 1.1.4.3 04-Jun-2008  yamt sync with head
 1.1.4.2 18-May-2008  yamt sync with head.
 1.1.4.1 30-Apr-2008  yamt file atomic_cas_up.S was added on branch yamt-pf42 on 2008-05-18 12:28:45 +0000
 1.1.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.2.62.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.62.1 21-Apr-2020  martin Sync with HEAD
 1.7 06-Aug-2020  skrll Centralise SYNC/BDSYNC in asm.h and introduce a new LLCSCSYNC and use it
before any ll/sc sequences.

Define LLSCSYNC as syncw; syncw for cnMIPS - issue two as early cnMIPS
has errat{um,a} that means the first can fail.
 1.6 01-Aug-2020  skrll Trailing whitespace
 1.5 01-Jun-2015  matt branches: 1.5.16;
Include OCTEON support for syncw and saa/saad (Store Atomic Add).
 1.4 14-Mar-2012  christos don't include <sys/cdefs.h> from assembly.
 1.3 27-Aug-2011  bouyer branches: 1.3.2;
loongson2f support:
- Add some loongson2 definitions to cpuregs.h, from OpenBSD
- Make sure that the at register is useable before every jump register
instruction (exept when register is k0 or k1) because -mfix-loongson2f-btb
needs the at register for its workaround
- add code to mips_fixup.c to handle the instructions added by
-mfix-loongson2f-btb
- Add a ls2-specific tlb miss handler: it doesn't have separate handler
for the xtlbmiss exeption.
- Fixes for some #ifdef MIPS3_LOONGSON2 assembly code (using the wrong
register)
 1.2 14-Dec-2009  matt Merge from matt-nb5-mips64
 1.1 05-Sep-2009  matt branches: 1.1.2;
file atomic_dec.S was initially added on branch matt-nb5-mips64.
 1.1.2.1 05-Sep-2009  matt Add native ll/sc or lld/scd versions of the atomic ops.
 1.3.2.1 17-Apr-2012  yamt sync with head
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.7 06-Aug-2020  skrll Centralise SYNC/BDSYNC in asm.h and introduce a new LLCSCSYNC and use it
before any ll/sc sequences.

Define LLSCSYNC as syncw; syncw for cnMIPS - issue two as early cnMIPS
has errat{um,a} that means the first can fail.
 1.6 01-Aug-2020  skrll Trailing whitespace
 1.5 01-Jun-2015  matt branches: 1.5.16;
Include OCTEON support for syncw and saa/saad (Store Atomic Add).
 1.4 14-Mar-2012  christos don't include <sys/cdefs.h> from assembly.
 1.3 27-Aug-2011  bouyer branches: 1.3.2;
loongson2f support:
- Add some loongson2 definitions to cpuregs.h, from OpenBSD
- Make sure that the at register is useable before every jump register
instruction (exept when register is k0 or k1) because -mfix-loongson2f-btb
needs the at register for its workaround
- add code to mips_fixup.c to handle the instructions added by
-mfix-loongson2f-btb
- Add a ls2-specific tlb miss handler: it doesn't have separate handler
for the xtlbmiss exeption.
- Fixes for some #ifdef MIPS3_LOONGSON2 assembly code (using the wrong
register)
 1.2 14-Dec-2009  matt Merge from matt-nb5-mips64
 1.1 05-Sep-2009  matt branches: 1.1.2;
file atomic_inc.S was initially added on branch matt-nb5-mips64.
 1.1.2.1 05-Sep-2009  matt Add native ll/sc or lld/scd versions of the atomic ops.
 1.3.2.1 17-Apr-2012  yamt sync with head
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.5 27-Feb-2022  riastradh mips: Membar audit.

This change should be safe because it doesn't remove or weaken any
memory barriers, but does add, clarify, or strengthen barriers.

Goals:

- Make sure mutex_enter/exit and mutex_spin_enter/exit have
acquire/release semantics.

- New macros make maintenance easier and purpose clearer:

. SYNC_ACQ is for load-before-load/store barrier, and BDSYNC_ACQ
for a branch delay slot -- currently defined as plain sync for MP
and nothing, or nop, for UP; thus it is no weaker than SYNC and
BDSYNC as currently defined, which is syncw on Octeon, plain sync
on non-Octeon MP, and nothing/nop on UP.

It is not clear to me whether load-then-syncw or ll/sc-then-syncw
or even bare load provides load-acquire semantics on Octeon -- if
no, this will fix bugs; if yes (like it is on SPARC PSO), we can
relax SYNC_ACQ to be syncw or nothing later.

. SYNC_REL is for load/store-before-store barrier -- currently
defined as plain sync for MP and nothing for UP.

It is not clear to me whether syncw-then-store is enough for
store-release on Octeon -- if no, we can leave this as is; if
yes, we can relax SYNC_REL to be syncw on Octeon.

. SYNC_PLUNGER is there to flush clogged Cavium store buffers, and
BDSYNC_PLUNGER for a branch delay slot -- syncw on Octeon,
nothing or nop on non-Octeon.

=> This is not necessary (or, as far as I'm aware, sufficient)
for acquire semantics -- it serves only to flush store buffers
where stores might otherwise linger for hundreds of thousands
of cycles, which would, e.g., cause spin locks to be held for
unreasonably long durations.

Newerish revisions of the MIPS ISA also have finer-grained sync
variants that could be plopped in here.

Mechanism:

Insert these barriers in the right places, replacing only those where
the definition is currently equivalent, so this change is safe.

- Replace #ifdef _MIPS_ARCH_OCTEONP / syncw / #endif at the end of
atomic_cas_* by SYNC_PLUNGER, which is `sync 4' (a.k.a. syncw) if
__OCTEON__ and empty otherwise.

=> From what I can tell, __OCTEON__ is defined in at least as many
contexts as _MIPS_ARCH_OCTEONP -- i.e., there are some Octeons
with no _MIPS_ARCH_OCTEONP, but I don't know if any of them are
relevant to us or ever saw the light of day outside Cavium; we
seem to buid with `-march=octeonp' so this is unlikely to make a
difference. If it turns out that we do care, well, now there's
a central place to make the distinction for sync instructions.

- Replace post-ll/sc SYNC by SYNC_ACQ in _atomic_cas_*, which are
internal kernel versions used in sys/arch/mips/include/lock.h where
it assumes they have load-acquire semantics. Should move this to
lock.h later, since we _don't_ define __HAVE_ATOMIC_AS_MEMBAR on
MIPS and so the extra barrier might be costly.

- Insert SYNC_REL before ll/sc, and replace post-ll/sc SYNC by
SYNC_ACQ, in _ucas_*, which is used without any barriers in futex
code and doesn't mention barriers in the man page so I have to
assume it is required to be a release/acquire barrier.

- Change BDSYNC to BDSYNC_ACQ in mutex_enter and mutex_spin_enter.
This is necessary to provide load-acquire semantics -- unclear if
it was provided already by syncw on Octeon, but it seems more
likely that either (a) no sync or syncw is needed at all, or (b)
syncw is not enough and sync is needed, since syncw is only a
store-before-store ordering barrier.

- Insert SYNC_REL before ll/sc in mutex_exit and mutex_spin_exit.
This is currently redundant with the SYNC already there, but
SYNC_REL more clearly identifies the necessary semantics in case we
want to define it differently on different systems, and having a
sync in the middle of an ll/sc is a bit weird and possibly not a
good idea, so I intend to (carefully) remove the redundant SYNC in
a later change.

- Change BDSYNC to BDSYNC_PLUNGER at the end of mutex_exit. This has
no semantic change right now -- it's syncw on Octeon, sync on
non-Octeon MP, nop on UP -- but we can relax it later to nop on
non-Cavium MP.

- Leave LLSCSYNC in for now -- it is apparently there for a Cavium
erratum, but I'm not sure what the erratum is, exactly, and I have
no reference for it. I suspect these can be safely removed, but we
might have to double up some other syncw instructions -- Linux uses
it only in store-release sequences, not at the head of every ll/sc.
 1.4 01-Aug-2020  skrll Trailing whitespace
 1.3 01-Jun-2015  matt branches: 1.3.16;
Include OCTEON support for syncw and saa/saad (Store Atomic Add).
 1.2 28-Apr-2008  martin Remove clause 3 and 4 from TNF licenses
 1.1 30-Nov-2007  ad branches: 1.1.4; 1.1.8;
Memory barriers for MIPS.
 1.1.8.1 18-May-2008  yamt sync with head.
 1.1.4.2 09-Jan-2008  matt sync with HEAD
 1.1.4.1 30-Nov-2007  matt file atomic_op_asm.h was added on branch matt-armv6 on 2008-01-09 01:20:58 +0000
 1.3.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.16.1 21-Apr-2020  martin Sync with HEAD
 1.6 06-Aug-2020  skrll Centralise SYNC/BDSYNC in asm.h and introduce a new LLCSCSYNC and use it
before any ll/sc sequences.

Define LLSCSYNC as syncw; syncw for cnMIPS - issue two as early cnMIPS
has errat{um,a} that means the first can fail.
 1.5 01-Aug-2020  skrll Trailing whitespace
 1.4 14-Mar-2012  christos branches: 1.4.34;
don't include <sys/cdefs.h> from assembly.
 1.3 27-Aug-2011  bouyer branches: 1.3.2;
loongson2f support:
- Add some loongson2 definitions to cpuregs.h, from OpenBSD
- Make sure that the at register is useable before every jump register
instruction (exept when register is k0 or k1) because -mfix-loongson2f-btb
needs the at register for its workaround
- add code to mips_fixup.c to handle the instructions added by
-mfix-loongson2f-btb
- Add a ls2-specific tlb miss handler: it doesn't have separate handler
for the xtlbmiss exeption.
- Fixes for some #ifdef MIPS3_LOONGSON2 assembly code (using the wrong
register)
 1.2 14-Dec-2009  matt Merge from matt-nb5-mips64
 1.1 05-Sep-2009  matt branches: 1.1.2;
file atomic_or.S was initially added on branch matt-nb5-mips64.
 1.1.2.1 05-Sep-2009  matt Add native ll/sc or lld/scd versions of the atomic ops.
 1.3.2.1 17-Apr-2012  yamt sync with head
 1.4.34.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.34.1 21-Apr-2020  martin Sync with HEAD
 1.8 27-Feb-2022  riastradh mips: Membar audit.

This change should be safe because it doesn't remove or weaken any
memory barriers, but does add, clarify, or strengthen barriers.

Goals:

- Make sure mutex_enter/exit and mutex_spin_enter/exit have
acquire/release semantics.

- New macros make maintenance easier and purpose clearer:

. SYNC_ACQ is for load-before-load/store barrier, and BDSYNC_ACQ
for a branch delay slot -- currently defined as plain sync for MP
and nothing, or nop, for UP; thus it is no weaker than SYNC and
BDSYNC as currently defined, which is syncw on Octeon, plain sync
on non-Octeon MP, and nothing/nop on UP.

It is not clear to me whether load-then-syncw or ll/sc-then-syncw
or even bare load provides load-acquire semantics on Octeon -- if
no, this will fix bugs; if yes (like it is on SPARC PSO), we can
relax SYNC_ACQ to be syncw or nothing later.

. SYNC_REL is for load/store-before-store barrier -- currently
defined as plain sync for MP and nothing for UP.

It is not clear to me whether syncw-then-store is enough for
store-release on Octeon -- if no, we can leave this as is; if
yes, we can relax SYNC_REL to be syncw on Octeon.

. SYNC_PLUNGER is there to flush clogged Cavium store buffers, and
BDSYNC_PLUNGER for a branch delay slot -- syncw on Octeon,
nothing or nop on non-Octeon.

=> This is not necessary (or, as far as I'm aware, sufficient)
for acquire semantics -- it serves only to flush store buffers
where stores might otherwise linger for hundreds of thousands
of cycles, which would, e.g., cause spin locks to be held for
unreasonably long durations.

Newerish revisions of the MIPS ISA also have finer-grained sync
variants that could be plopped in here.

Mechanism:

Insert these barriers in the right places, replacing only those where
the definition is currently equivalent, so this change is safe.

- Replace #ifdef _MIPS_ARCH_OCTEONP / syncw / #endif at the end of
atomic_cas_* by SYNC_PLUNGER, which is `sync 4' (a.k.a. syncw) if
__OCTEON__ and empty otherwise.

=> From what I can tell, __OCTEON__ is defined in at least as many
contexts as _MIPS_ARCH_OCTEONP -- i.e., there are some Octeons
with no _MIPS_ARCH_OCTEONP, but I don't know if any of them are
relevant to us or ever saw the light of day outside Cavium; we
seem to buid with `-march=octeonp' so this is unlikely to make a
difference. If it turns out that we do care, well, now there's
a central place to make the distinction for sync instructions.

- Replace post-ll/sc SYNC by SYNC_ACQ in _atomic_cas_*, which are
internal kernel versions used in sys/arch/mips/include/lock.h where
it assumes they have load-acquire semantics. Should move this to
lock.h later, since we _don't_ define __HAVE_ATOMIC_AS_MEMBAR on
MIPS and so the extra barrier might be costly.

- Insert SYNC_REL before ll/sc, and replace post-ll/sc SYNC by
SYNC_ACQ, in _ucas_*, which is used without any barriers in futex
code and doesn't mention barriers in the man page so I have to
assume it is required to be a release/acquire barrier.

- Change BDSYNC to BDSYNC_ACQ in mutex_enter and mutex_spin_enter.
This is necessary to provide load-acquire semantics -- unclear if
it was provided already by syncw on Octeon, but it seems more
likely that either (a) no sync or syncw is needed at all, or (b)
syncw is not enough and sync is needed, since syncw is only a
store-before-store ordering barrier.

- Insert SYNC_REL before ll/sc in mutex_exit and mutex_spin_exit.
This is currently redundant with the SYNC already there, but
SYNC_REL more clearly identifies the necessary semantics in case we
want to define it differently on different systems, and having a
sync in the middle of an ll/sc is a bit weird and possibly not a
good idea, so I intend to (carefully) remove the redundant SYNC in
a later change.

- Change BDSYNC to BDSYNC_PLUNGER at the end of mutex_exit. This has
no semantic change right now -- it's syncw on Octeon, sync on
non-Octeon MP, nop on UP -- but we can relax it later to nop on
non-Cavium MP.

- Leave LLSCSYNC in for now -- it is apparently there for a Cavium
erratum, but I'm not sure what the erratum is, exactly, and I have
no reference for it. I suspect these can be safely removed, but we
might have to double up some other syncw instructions -- Linux uses
it only in store-release sequences, not at the head of every ll/sc.
 1.7 06-Aug-2020  skrll Centralise SYNC/BDSYNC in asm.h and introduce a new LLCSCSYNC and use it
before any ll/sc sequences.

Define LLSCSYNC as syncw; syncw for cnMIPS - issue two as early cnMIPS
has errat{um,a} that means the first can fail.
 1.6 01-Aug-2020  skrll Trailing whitespace
 1.5 01-Jun-2015  matt branches: 1.5.16;
Include OCTEON support for syncw and saa/saad (Store Atomic Add).
 1.4 14-Mar-2012  christos don't include <sys/cdefs.h> from assembly.
 1.3 27-Aug-2011  bouyer branches: 1.3.2;
loongson2f support:
- Add some loongson2 definitions to cpuregs.h, from OpenBSD
- Make sure that the at register is useable before every jump register
instruction (exept when register is k0 or k1) because -mfix-loongson2f-btb
needs the at register for its workaround
- add code to mips_fixup.c to handle the instructions added by
-mfix-loongson2f-btb
- Add a ls2-specific tlb miss handler: it doesn't have separate handler
for the xtlbmiss exeption.
- Fixes for some #ifdef MIPS3_LOONGSON2 assembly code (using the wrong
register)
 1.2 14-Dec-2009  matt Merge from matt-nb5-mips64
 1.1 05-Sep-2009  matt branches: 1.1.2;
file atomic_swap.S was initially added on branch matt-nb5-mips64.
 1.1.2.1 05-Sep-2009  matt Add native ll/sc or lld/scd versions of the atomic ops.
 1.3.2.1 17-Apr-2012  yamt sync with head
 1.5.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.16.1 21-Apr-2020  martin Sync with HEAD
 1.13 21-Apr-2022  riastradh mips/cavium: Take advantage of Octeon's guaranteed r/rw ordering.
 1.12 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.11 12-Feb-2022  riastradh mips: Brush up __cpu_simple_lock.

- Eradicate last vestiges of mb_* barriers.

- In __cpu_simple_lock_init, omit needless barrier. It is the
caller's responsibility to ensure __cpu_simple_lock_init happens
before other operations on it anyway, so there was never any need
for a barrier here.

- In __cpu_simple_lock_try, leave comments about memory ordering
guarantees of the kernel's _atomic_cas_uint, which are inexplicably
different from the non-underscored atomic_cas_uint.

- In __cpu_simple_unlock, use membar_exit instead of mb_memory, and do
it unconditionally.

This ensures that in __cpu_simple_lock/.../__cpu_simple_unlock, all
memory operations in the ellipsis happen before the store that
releases the lock.

- On Octeon, the barrier was omitted altogether, which is a bug --
it needs to be there or else there is no happens-before relation
and whoever takes the lock next might see stale values stored or
even stomp over the unlocking CPU's delayed loads.

- On non-Octeon, the mb_memory was sync. Using membar_exit
preserves this.

XXX On Octeon, membar_exit only issues syncw -- this seems wrong,
only store-before-store and not load/store-before-store, unless the
CNMIPS architecture guarantees it is sufficient here like
SPARCv8/v9 PSO (`Partial Store Order').

- Leave an essay with citations about why we have an apparently
pointless syncw _after_ releasing a lock, to work around a design
bug^W^Wquirk in cnmips which sometimes buffers stores for hundreds
of thousands of cycles for fun unless you issue syncw.
 1.10 10-Aug-2020  skrll More SYNC centralisation
 1.9 01-Aug-2020  skrll Trailing whitespace
 1.8 23-Jun-2015  matt branches: 1.8.16;
Always use sync if mips3 or later or not using O32 ABI. (A little redundant
since not using O32 means you are using mips3 or later.)
 1.7 22-Jun-2015  matt #include "assym.h"
Don't include "assym.h" with _RUMPKERNEL defined.
 1.6 01-Jun-2015  matt Include OCTEON support for syncw and saa/saad (Store Atomic Add).
 1.5 03-Aug-2012  matt Add a missing .set noreorder
 1.4 14-Dec-2009  matt branches: 1.4.6;
Merge from matt-nb5-mips64
 1.3 25-May-2008  chs branches: 1.3.10;
enable profiling of assembly functions.
 1.2 28-Apr-2008  martin branches: 1.2.2;
Remove clause 3 and 4 from TNF licenses
 1.1 30-Nov-2007  ad branches: 1.1.4; 1.1.8;
Memory barriers for MIPS.
 1.1.8.2 04-Jun-2008  yamt sync with head
 1.1.8.1 18-May-2008  yamt sync with head.
 1.1.4.2 09-Jan-2008  matt sync with HEAD
 1.1.4.1 30-Nov-2007  matt file membar_ops.S was added on branch matt-armv6 on 2008-01-09 01:20:59 +0000
 1.2.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.3.10.3 03-Aug-2012  matt Add missing .set noreorder
 1.3.10.2 05-Sep-2009  matt Resolve some conflicts.
 1.3.10.1 05-Sep-2009  matt Only allow to null on o32
 1.4.6.1 30-Oct-2012  yamt sync with head
 1.8.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.16.1 21-Apr-2020  martin Sync with HEAD
 1.3 14-Dec-2009  matt branches: 1.3.48;
Merge from matt-nb5-mips64
 1.2 08-Feb-2006  simonb branches: 1.2.40;
Don't rename bswap{16,32} if either _KERNEL or _STANDALONE are defined,
instead of just if _KERNEL was defined.
Fixes sbmips bootblocks build problems.
Thanks to Valeriy Ushakov for showing me where the problem was.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.40.2 30-Aug-2009  matt If compiling for mips32r2 or mips64r2, use new wshb/rotr instructions.
 1.2.40.1 16-Aug-2009  matt First pass at making ABI agnostic. Generates same O32 code.
 1.3.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.48.1 21-Apr-2020  martin Sync with HEAD
 1.3 14-Dec-2009  matt branches: 1.3.48;
Merge from matt-nb5-mips64
 1.2 08-Feb-2006  simonb branches: 1.2.40;
Don't rename bswap{16,32} if either _KERNEL or _STANDALONE are defined,
instead of just if _KERNEL was defined.
Fixes sbmips bootblocks build problems.
Thanks to Valeriy Ushakov for showing me where the problem was.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.40.2 30-Aug-2009  matt If compiling for mips32r2 or mips64r2, use new wshb/rotr instructions.
 1.2.40.1 16-Aug-2009  matt First pass at making ABI agnostic. Generates same O32 code.
 1.3.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.48.1 21-Apr-2020  martin Sync with HEAD
 1.2 14-Dec-2009  matt branches: 1.2.48;
Merge from matt-nb5-mips64
 1.1 30-Aug-2009  matt branches: 1.1.2;
file byte_swap_8.S was initially added on branch matt-nb5-mips64.
 1.1.2.3 05-Sep-2009  matt Add an assmebly version of bswap64
 1.1.2.2 30-Aug-2009  matt Fix typo (dshb -> dshd)
 1.1.2.1 30-Aug-2009  matt Add bswap64 implementation for mips. for mips32r2 and mips64r2, use the
new dshd/dshb and wshb/rotr instructions.
 1.2.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.48.1 21-Apr-2020  martin Sync with HEAD
 1.4 27-Aug-2011  bouyer branches: 1.4.44;
loongson2f support:
- Add some loongson2 definitions to cpuregs.h, from OpenBSD
- Make sure that the at register is useable before every jump register
instruction (exept when register is k0 or k1) because -mfix-loongson2f-btb
needs the at register for its workaround
- add code to mips_fixup.c to handle the instructions added by
-mfix-loongson2f-btb
- Add a ls2-specific tlb miss handler: it doesn't have separate handler
for the xtlbmiss exeption.
- Fixes for some #ifdef MIPS3_LOONGSON2 assembly code (using the wrong
register)
 1.3 14-Dec-2009  matt Merge from matt-nb5-mips64
 1.2 27-Dec-2005  tsutsui branches: 1.2.40;
Don't redefine _LOCORE if it's already defined.
Some Makefiles for standalone programs already have it.

XXX Old src/sys/lib/libkern/arch/mips/memcpy.S had some #ifdef MIPS3_5900
XXX which added some extra nops, but this new common bcopy.S doesn't.
 1.1 21-Dec-2005  christos move from libc.
 1.2.40.4 23-Aug-2009  matt Use #if 0 method for RCSID selection.
 1.2.40.3 20-Aug-2009  matt Fix screwup when converting. Use 64bit versions of lwr/lwl ldr/ldl as
appropriate.
 1.2.40.2 19-Aug-2009  matt Make this emit exactly the same code as before under O32.
 1.2.40.1 16-Aug-2009  matt First pass at making ABI agnostic. Generates same O32 code.
 1.4.44.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.44.1 21-Apr-2020  martin Sync with HEAD
 1.4 04-Jul-2011  mrg branches: 1.4.44;
add a weak alias from ffs to __ffssi2. newsmips kernels link now.
 1.3 23-Jan-2011  matt Add a new O(log(2) implementation. On mips32/mips64, use clz/dclz.
 1.2 14-Dec-2009  matt branches: 1.2.4;
Merge from matt-nb5-mips64
 1.1 20-Dec-2005  christos branches: 1.1.40;
Merge libkern + libc common files. As requested by core.
 1.1.40.3 22-Apr-2010  matt Fix mips32/mips64 versions of ffs.
 1.1.40.2 21-Apr-2010  matt sync to netbsd-5
 1.1.40.1 16-Aug-2009  matt First pass at making ABI agnostic. Generates same O32 code.
 1.2.4.1 08-Feb-2011  bouyer Sync with HEAD
 1.4.44.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.44.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.2 14-Dec-2009  matt Merge from matt-nb5-mips64
 1.1 20-Dec-2005  christos branches: 1.1.40;
Merge libkern + libc common files. As requested by core.
 1.1.40.2 18-Aug-2009  matt No longer used.
 1.1.40.1 16-Aug-2009  matt First pass at making ABI agnostic. Generates same O32 code.
 1.5 25-Feb-2017  joerg branches: 1.5.12;
Switch from __ABICALLS__ to __mips_abicalls like upstream GCC does in
the generic MIPS target logic.
 1.4 26-May-2015  matt branches: 1.4.2; 1.4.4;
force 2nd to argument to unsigned byte value.
(found by t_strchr and t_strrchr tests)
 1.3 02-Jan-2011  matt Make these 64-bit clean.
 1.2 17-Jul-2009  dsl branches: 1.2.2;
Change all archs so that strchr.[cS] and strrchr.[cS] exist and generate
duplicate symbols for index() and rindex().
 1.1 16-Jul-2009  dsl Add asm files for str(r)chr by copying the files for (r)index from
src/libc/arch/mips/string.
Add XLEAF() for index entry points.
 1.2.2.2 23-Jul-2009  jym Sync with HEAD.
 1.2.2.1 17-Jul-2009  jym file strchr.S was added on branch jym-xensuspend on 2009-07-23 23:31:34 +0000
 1.4.4.1 21-Apr-2017  bouyer Sync with HEAD
 1.4.2.1 20-Mar-2017  pgoyette Sync with HEAD
 1.5.12.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.12.1 21-Apr-2020  martin Sync with HEAD
 1.2 14-Dec-2009  matt branches: 1.2.48;
Merge from matt-nb5-mips64
 1.1 20-Dec-2005  christos branches: 1.1.40;
Merge libkern + libc common files. As requested by core.
 1.1.40.1 16-Aug-2009  matt First pass at making ABI agnostic. Generates same O32 code.
 1.2.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.48.1 21-Apr-2020  martin Sync with HEAD
 1.2 14-Dec-2009  matt branches: 1.2.48;
Merge from matt-nb5-mips64
 1.1 20-Dec-2005  christos branches: 1.1.40;
Merge libkern + libc common files. As requested by core.
 1.1.40.1 16-Aug-2009  matt First pass at making ABI agnostic. Generates same O32 code.
 1.2.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.48.1 21-Apr-2020  martin Sync with HEAD
 1.4 25-Feb-2017  joerg branches: 1.4.12;
Switch from __ABICALLS__ to __mips_abicalls like upstream GCC does in
the generic MIPS target logic.
 1.3 26-May-2015  matt branches: 1.3.2; 1.3.4;
force 2nd to argument to unsigned byte value.
(found by t_strchr and t_strrchr tests)
 1.2 02-Jan-2011  matt Make these 64-bit clean.
 1.1 16-Jul-2009  dsl branches: 1.1.2;
Add asm files for str(r)chr by copying the files for (r)index from
src/libc/arch/mips/string.
Add XLEAF() for index entry points.
 1.1.2.2 23-Jul-2009  jym Sync with HEAD.
 1.1.2.1 16-Jul-2009  jym file strrchr.S was added on branch jym-xensuspend on 2009-07-23 23:31:34 +0000
 1.3.4.1 21-Apr-2017  bouyer Sync with HEAD
 1.3.2.1 20-Mar-2017  pgoyette Sync with HEAD
 1.4.12.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.12.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 03-Sep-2014  matt branches: 1.1.16;
New files for OR1K support
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.6 04-Jan-2009  pooka branches: 1.6.50;
allow inclusion of atomic ops in librump
 1.5 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.4 11-Feb-2008  ad Only build atomic ops for libkern/libc.
 1.3 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.2 28-Nov-2007  ad branches: 1.2.4;
Atomic ops for powerpc. Please review.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej W-I-P atomic ops for powerpc. No memory barrier ops yet.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:20:59 +0000
 1.6.50.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.50.1 21-Apr-2020  martin Sync with HEAD
 1.9 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.8 07-Mar-2014  matt branches: 1.8.26;
Reduce duplication.
Use beqlr to quicken returns
 1.7 15-Jan-2011  matt branches: 1.7.6; 1.7.12;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.6 25-May-2008  chs enable profiling of assembly functions.
 1.5 28-Apr-2008  martin branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses
 1.4 13-Feb-2008  dogcow branches: 1.4.4;
add missing %. (hi matt!)
 1.3 12-Feb-2008  matt Use %rN as all new code is supposed to.
 1.2 28-Nov-2007  ad branches: 1.2.4;
Atomic ops for powerpc. Please review.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_add.S was initially added on branch thorpej-atomic.
 1.1.2.2 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.1.2.1 26-Apr-2007  thorpej W-I-P atomic ops for powerpc. No memory barrier ops yet.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_add.S was added on branch matt-armv6 on 2008-01-09 01:21:00 +0000
 1.4.4.2 04-Jun-2008  yamt sync with head
 1.4.4.1 18-May-2008  yamt sync with head.
 1.5.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.7.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.7.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.8.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.26.1 21-Apr-2020  martin Sync with HEAD
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 07-Mar-2014  matt branches: 1.7.26;
Reduce duplication.
Use beqlr to quicken returns
 1.6 15-Jan-2011  matt branches: 1.6.6; 1.6.12;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.5 25-May-2008  chs enable profiling of assembly functions.
 1.4 28-Apr-2008  martin branches: 1.4.2;
Remove clause 3 and 4 from TNF licenses
 1.3 12-Feb-2008  matt branches: 1.3.4;
Use %rN as all new code is supposed to.
 1.2 28-Nov-2007  ad branches: 1.2.4;
Atomic ops for powerpc. Please review.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_and.S was initially added on branch thorpej-atomic.
 1.1.2.2 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.1.2.1 26-Apr-2007  thorpej W-I-P atomic ops for powerpc. No memory barrier ops yet.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_and.S was added on branch matt-armv6 on 2008-01-09 01:21:00 +0000
 1.3.4.2 04-Jun-2008  yamt sync with head
 1.3.4.1 18-May-2008  yamt sync with head.
 1.4.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.6.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.10 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.9 01-Mar-2020  rin Implement workaround for IBM405 Errata 77 (aka CPU_210), where
interrupted stwcx. may errantly write data to memory:

https://elinux.org/images/1/1d/Ppc405gp-errata.pdf

This is because stwcx. is split into two pieces in the pipeline.

We need to
(1) insert dcbt before every stwcx. instruction, as well as
(2) insert sync before every rfi/rfci instruction.

It is unclear which processors are affected, but according to Linux,
all 405-based cores up until 405GPR and 405EP are affected:

https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/40x/Kconfig#L140

For kernel, this workaround can be restricted to affected processors.
However, for kernel modules and userland, we have to enable it for all
32bit powerpc archs in order to share common binaries as before.

Proposed on port-powerpc:

http://mail-index.netbsd.org/port-powerpc/2020/02/21/msg003583.html
 1.8 07-Mar-2014  matt branches: 1.8.26; 1.8.28;
Reduce duplication.
Use beqlr to quicken returns
 1.7 15-Jan-2011  matt branches: 1.7.6; 1.7.12;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.6 25-May-2008  chs enable profiling of assembly functions.
 1.5 28-Apr-2008  martin branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses
 1.4 12-Feb-2008  matt branches: 1.4.4;
Use %rN as all new code is supposed to.
 1.3 10-Feb-2008  ad Add atomic_cas_foo_ni().
 1.2 28-Nov-2007  ad branches: 1.2.4;
Atomic ops for powerpc. Please review.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_cas.S was initially added on branch thorpej-atomic.
 1.1.2.2 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.1.2.1 26-Apr-2007  thorpej W-I-P atomic ops for powerpc. No memory barrier ops yet.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_cas.S was added on branch matt-armv6 on 2008-01-09 01:21:00 +0000
 1.4.4.2 04-Jun-2008  yamt sync with head
 1.4.4.1 18-May-2008  yamt sync with head.
 1.5.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.7.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.7.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.8.28.1 03-Mar-2020  martin Pull up following revision(s) (requested by rin in ticket #755):

sys/arch/evbppc/conf/std.virtex: revision 1.4
sys/arch/powerpc/powerpc/trap_subr.S: revision 1.81
sys/arch/powerpc/ibm4xx/4xx_trap_subr.S: revision 1.8
sys/arch/evbppc/conf/std.walnut: revision 1.9
common/lib/libc/arch/powerpc/atomic/atomic_op_asm.h: revision 1.7
sys/arch/powerpc/include/asm.h: revision 1.49
common/lib/libc/arch/powerpc/atomic/atomic_cas.S: revision 1.9
sys/arch/powerpc/ibm4xx/trap_subr.S: revision 1.28
sys/arch/powerpc/include/lock.h: revision 1.15
sys/arch/evbppc/conf/std.obs266: revision 1.3
common/lib/libc/arch/powerpc/atomic/atomic_swap.S: revision 1.8
sys/arch/powerpc/powerpc/locore_subr.S: revision 1.61
sys/arch/powerpc/powerpc/lock_stubs.S: revision 1.12
sys/arch/evbppc/conf/std.obs200: revision 1.5

Implement workaround for IBM405 Errata 77 (aka CPU_210), where
interrupted stwcx. may errantly write data to memory:

https://elinux.org/images/1/1d/Ppc405gp-errata.pdf

This is because stwcx. is split into two pieces in the pipeline.

We need to
(1) insert dcbt before every stwcx. instruction, as well as
(2) insert sync before every rfi/rfci instruction.

It is unclear which processors are affected, but according to Linux,
all 405-based cores up until 405GPR and 405EP are affected:

https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/40x/Kconfig#L140

For kernel, this workaround can be restricted to affected processors.

However, for kernel modules and userland, we have to enable it for all
32bit powerpc archs in order to share common binaries as before.
Proposed on port-powerpc:

http://mail-index.netbsd.org/port-powerpc/2020/02/21/msg003583.html
 1.8.26.2 21-Apr-2020  martin Sync with HEAD
 1.8.26.1 08-Apr-2020  martin Merge changes from current as of 20200406
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 07-Mar-2014  matt branches: 1.7.26;
Reduce duplication.
Use beqlr to quicken returns
 1.6 15-Jan-2011  matt branches: 1.6.6; 1.6.12;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.5 25-May-2008  chs enable profiling of assembly functions.
 1.4 28-Apr-2008  martin branches: 1.4.2;
Remove clause 3 and 4 from TNF licenses
 1.3 12-Feb-2008  matt branches: 1.3.4;
Use %rN as all new code is supposed to.
 1.2 28-Nov-2007  ad branches: 1.2.4;
Atomic ops for powerpc. Please review.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_dec.S was initially added on branch thorpej-atomic.
 1.1.2.2 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.1.2.1 26-Apr-2007  thorpej W-I-P atomic ops for powerpc. No memory barrier ops yet.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_dec.S was added on branch matt-armv6 on 2008-01-09 01:21:01 +0000
 1.3.4.2 04-Jun-2008  yamt sync with head
 1.3.4.1 18-May-2008  yamt sync with head.
 1.4.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.6.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 07-Mar-2014  matt branches: 1.7.26;
Reduce duplication.
Use beqlr to quicken returns
 1.6 15-Jan-2011  matt branches: 1.6.6; 1.6.12;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.5 25-May-2008  chs enable profiling of assembly functions.
 1.4 28-Apr-2008  martin branches: 1.4.2;
Remove clause 3 and 4 from TNF licenses
 1.3 12-Feb-2008  matt branches: 1.3.4;
Use %rN as all new code is supposed to.
 1.2 28-Nov-2007  ad branches: 1.2.4;
Atomic ops for powerpc. Please review.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_inc.S was initially added on branch thorpej-atomic.
 1.1.2.2 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.1.2.1 26-Apr-2007  thorpej W-I-P atomic ops for powerpc. No memory barrier ops yet.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_inc.S was added on branch matt-armv6 on 2008-01-09 01:21:01 +0000
 1.3.4.2 04-Jun-2008  yamt sync with head
 1.3.4.1 18-May-2008  yamt sync with head.
 1.4.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.6.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 01-Mar-2020  rin Implement workaround for IBM405 Errata 77 (aka CPU_210), where
interrupted stwcx. may errantly write data to memory:

https://elinux.org/images/1/1d/Ppc405gp-errata.pdf

This is because stwcx. is split into two pieces in the pipeline.

We need to
(1) insert dcbt before every stwcx. instruction, as well as
(2) insert sync before every rfi/rfci instruction.

It is unclear which processors are affected, but according to Linux,
all 405-based cores up until 405GPR and 405EP are affected:

https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/40x/Kconfig#L140

For kernel, this workaround can be restricted to affected processors.
However, for kernel modules and userland, we have to enable it for all
32bit powerpc archs in order to share common binaries as before.

Proposed on port-powerpc:

http://mail-index.netbsd.org/port-powerpc/2020/02/21/msg003583.html
 1.6 07-Mar-2014  matt branches: 1.6.26; 1.6.28;
Reduce duplication.
Use beqlr to quicken returns
 1.5 15-Jan-2011  matt branches: 1.5.6; 1.5.12;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.4 28-Apr-2008  martin Remove clause 3 and 4 from TNF licenses
 1.3 12-Feb-2008  matt branches: 1.3.4;
Use %rN as all new code is supposed to.
 1.2 28-Nov-2007  ad branches: 1.2.4;
Atomic ops for powerpc. Please review.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej W-I-P atomic ops for powerpc. No memory barrier ops yet.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_op_asm.h was added on branch matt-armv6 on 2008-01-09 01:21:01 +0000
 1.3.4.1 18-May-2008  yamt sync with head.
 1.5.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.28.1 03-Mar-2020  martin Pull up following revision(s) (requested by rin in ticket #755):

sys/arch/evbppc/conf/std.virtex: revision 1.4
sys/arch/powerpc/powerpc/trap_subr.S: revision 1.81
sys/arch/powerpc/ibm4xx/4xx_trap_subr.S: revision 1.8
sys/arch/evbppc/conf/std.walnut: revision 1.9
common/lib/libc/arch/powerpc/atomic/atomic_op_asm.h: revision 1.7
sys/arch/powerpc/include/asm.h: revision 1.49
common/lib/libc/arch/powerpc/atomic/atomic_cas.S: revision 1.9
sys/arch/powerpc/ibm4xx/trap_subr.S: revision 1.28
sys/arch/powerpc/include/lock.h: revision 1.15
sys/arch/evbppc/conf/std.obs266: revision 1.3
common/lib/libc/arch/powerpc/atomic/atomic_swap.S: revision 1.8
sys/arch/powerpc/powerpc/locore_subr.S: revision 1.61
sys/arch/powerpc/powerpc/lock_stubs.S: revision 1.12
sys/arch/evbppc/conf/std.obs200: revision 1.5

Implement workaround for IBM405 Errata 77 (aka CPU_210), where
interrupted stwcx. may errantly write data to memory:

https://elinux.org/images/1/1d/Ppc405gp-errata.pdf

This is because stwcx. is split into two pieces in the pipeline.

We need to
(1) insert dcbt before every stwcx. instruction, as well as
(2) insert sync before every rfi/rfci instruction.

It is unclear which processors are affected, but according to Linux,
all 405-based cores up until 405GPR and 405EP are affected:

https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/40x/Kconfig#L140

For kernel, this workaround can be restricted to affected processors.

However, for kernel modules and userland, we have to enable it for all
32bit powerpc archs in order to share common binaries as before.
Proposed on port-powerpc:

http://mail-index.netbsd.org/port-powerpc/2020/02/21/msg003583.html
 1.6.26.2 21-Apr-2020  martin Sync with HEAD
 1.6.26.1 08-Apr-2020  martin Merge changes from current as of 20200406
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 07-Mar-2014  matt branches: 1.7.26;
Reduce duplication.
Use beqlr to quicken returns
 1.6 15-Jan-2011  matt branches: 1.6.6; 1.6.12;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.5 25-May-2008  chs enable profiling of assembly functions.
 1.4 28-Apr-2008  martin branches: 1.4.2;
Remove clause 3 and 4 from TNF licenses
 1.3 12-Feb-2008  matt branches: 1.3.4;
Use %rN as all new code is supposed to.
 1.2 28-Nov-2007  ad branches: 1.2.4;
Atomic ops for powerpc. Please review.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_or.S was initially added on branch thorpej-atomic.
 1.1.2.2 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.1.2.1 26-Apr-2007  thorpej W-I-P atomic ops for powerpc. No memory barrier ops yet.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_or.S was added on branch matt-armv6 on 2008-01-09 01:21:02 +0000
 1.3.4.2 04-Jun-2008  yamt sync with head
 1.3.4.1 18-May-2008  yamt sync with head.
 1.4.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.6.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.9 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.8 01-Mar-2020  rin Implement workaround for IBM405 Errata 77 (aka CPU_210), where
interrupted stwcx. may errantly write data to memory:

https://elinux.org/images/1/1d/Ppc405gp-errata.pdf

This is because stwcx. is split into two pieces in the pipeline.

We need to
(1) insert dcbt before every stwcx. instruction, as well as
(2) insert sync before every rfi/rfci instruction.

It is unclear which processors are affected, but according to Linux,
all 405-based cores up until 405GPR and 405EP are affected:

https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/40x/Kconfig#L140

For kernel, this workaround can be restricted to affected processors.
However, for kernel modules and userland, we have to enable it for all
32bit powerpc archs in order to share common binaries as before.

Proposed on port-powerpc:

http://mail-index.netbsd.org/port-powerpc/2020/02/21/msg003583.html
 1.7 07-Mar-2014  matt branches: 1.7.26; 1.7.28;
Reduce duplication.
Use beqlr to quicken returns
 1.6 15-Jan-2011  matt branches: 1.6.6; 1.6.12;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.5 25-May-2008  chs enable profiling of assembly functions.
 1.4 28-Apr-2008  martin branches: 1.4.2;
Remove clause 3 and 4 from TNF licenses
 1.3 12-Feb-2008  matt branches: 1.3.4;
Use %rN as all new code is supposed to.
 1.2 28-Nov-2007  ad branches: 1.2.4;
Atomic ops for powerpc. Please review.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_swap.S was initially added on branch thorpej-atomic.
 1.1.2.2 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.1.2.1 26-Apr-2007  thorpej W-I-P atomic ops for powerpc. No memory barrier ops yet.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_swap.S was added on branch matt-armv6 on 2008-01-09 01:21:02 +0000
 1.3.4.2 04-Jun-2008  yamt sync with head
 1.3.4.1 18-May-2008  yamt sync with head.
 1.4.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.6.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.7.28.1 03-Mar-2020  martin Pull up following revision(s) (requested by rin in ticket #755):

sys/arch/evbppc/conf/std.virtex: revision 1.4
sys/arch/powerpc/powerpc/trap_subr.S: revision 1.81
sys/arch/powerpc/ibm4xx/4xx_trap_subr.S: revision 1.8
sys/arch/evbppc/conf/std.walnut: revision 1.9
common/lib/libc/arch/powerpc/atomic/atomic_op_asm.h: revision 1.7
sys/arch/powerpc/include/asm.h: revision 1.49
common/lib/libc/arch/powerpc/atomic/atomic_cas.S: revision 1.9
sys/arch/powerpc/ibm4xx/trap_subr.S: revision 1.28
sys/arch/powerpc/include/lock.h: revision 1.15
sys/arch/evbppc/conf/std.obs266: revision 1.3
common/lib/libc/arch/powerpc/atomic/atomic_swap.S: revision 1.8
sys/arch/powerpc/powerpc/locore_subr.S: revision 1.61
sys/arch/powerpc/powerpc/lock_stubs.S: revision 1.12
sys/arch/evbppc/conf/std.obs200: revision 1.5

Implement workaround for IBM405 Errata 77 (aka CPU_210), where
interrupted stwcx. may errantly write data to memory:

https://elinux.org/images/1/1d/Ppc405gp-errata.pdf

This is because stwcx. is split into two pieces in the pipeline.

We need to
(1) insert dcbt before every stwcx. instruction, as well as
(2) insert sync before every rfi/rfci instruction.

It is unclear which processors are affected, but according to Linux,
all 405-based cores up until 405GPR and 405EP are affected:

https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/40x/Kconfig#L140

For kernel, this workaround can be restricted to affected processors.

However, for kernel modules and userland, we have to enable it for all
32bit powerpc archs in order to share common binaries as before.
Proposed on port-powerpc:

http://mail-index.netbsd.org/port-powerpc/2020/02/21/msg003583.html
 1.7.26.2 21-Apr-2020  martin Sync with HEAD
 1.7.26.1 08-Apr-2020  martin Merge changes from current as of 20200406
 1.6 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.5 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.4 15-Jan-2011  matt branches: 1.4.48;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.3 25-May-2008  chs enable profiling of assembly functions.
 1.2 28-Apr-2008  martin branches: 1.2.2;
Remove clause 3 and 4 from TNF licenses
 1.1 28-Nov-2007  ad branches: 1.1.4; 1.1.8;
Atomic ops for powerpc. Please review.
 1.1.8.2 04-Jun-2008  yamt sync with head
 1.1.8.1 18-May-2008  yamt sync with head.
 1.1.4.2 09-Jan-2008  matt sync with HEAD
 1.1.4.1 28-Nov-2007  matt file membar_ops.S was added on branch matt-armv6 on 2008-01-09 01:21:02 +0000
 1.2.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.48.1 21-Apr-2020  martin Sync with HEAD
 1.1 21-Feb-2008  garbled branches: 1.1.2; 1.1.4; 1.1.70;
Add tuned powerpc assembler written by IBM and released under a 3-clause
BSD Lisc as part of the perflib project.
http://sourceforge.net/projects/ppcperflib/

Tested the new functions with microbenchmarks on a number of different
CPU types, and found that most cpus either benefited greatly, or were
unaffected. Primarily G4 CPU's were unaffected, and all others showed
speedups. My 7044 (POWER3) went from a 70.6 to a 73.2 (thats good) in
bytebench with a complete release built with these. Also passed
regression tests.
 1.1.70.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.70.1 21-Apr-2020  martin Sync with HEAD
 1.1.4.2 24-Mar-2008  keiichi sync with head.
 1.1.4.1 21-Feb-2008  keiichi file bcopy.S was added on branch keiichi-mipv6 on 2008-03-24 07:14:29 +0000
 1.1.2.2 23-Mar-2008  matt sync with HEAD
 1.1.2.1 21-Feb-2008  matt file bcopy.S was added on branch matt-armv6 on 2008-03-23 00:12:42 +0000
 1.5 15-Jan-2011  matt branches: 1.5.48;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.4 07-Jul-2006  ross revert
 1.3 07-Jul-2006  ross And yet more .S files...
 1.2 23-Dec-2005  skrll Fix PR 32363 by defining _NOREGNAMES.

Someone should complete the transition.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.5.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.48.1 21-Apr-2020  martin Sync with HEAD
 1.4 04-Jul-2013  matt branches: 1.4.28;
remove branch prediction hint from bdnzt
 1.3 15-Jan-2011  matt branches: 1.3.6; 1.3.12;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.2 06-Mar-2008  phx branches: 1.2.2; 1.2.4;
Including assym.h is not needed and will break the build, because it does not
exist that early. Deleted it.
Approved by garbled.
 1.1 21-Feb-2008  garbled Add tuned powerpc assembler written by IBM and released under a 3-clause
BSD Lisc as part of the perflib project.
http://sourceforge.net/projects/ppcperflib/

Tested the new functions with microbenchmarks on a number of different
CPU types, and found that most cpus either benefited greatly, or were
unaffected. Primarily G4 CPU's were unaffected, and all others showed
speedups. My 7044 (POWER3) went from a 70.6 to a 73.2 (thats good) in
bytebench with a complete release built with these. Also passed
regression tests.
 1.2.4.2 24-Mar-2008  keiichi sync with head.
 1.2.4.1 06-Mar-2008  keiichi file memcmp.S was added on branch keiichi-mipv6 on 2008-03-24 07:14:29 +0000
 1.2.2.2 23-Mar-2008  matt sync with HEAD
 1.2.2.1 06-Mar-2008  matt file memcmp.S was added on branch matt-armv6 on 2008-03-23 00:12:42 +0000
 1.3.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.28.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.28.1 21-Apr-2020  martin Sync with HEAD
 1.6 04-Mar-2014  macallan branches: 1.6.26;
on 601 CPUs skip to normal memcpy if both source and destination are 32bit
aligned
 1.5 03-Mar-2014  matt Use _KERNEL_OPT around #include "opt_ppcarch.h"
 1.4 03-Mar-2014  macallan on ppc601 do byte-wise copies when in _KERNEL
from scole_mail, ok matt@
 1.3 15-Jan-2011  matt branches: 1.3.6; 1.3.12;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.2 06-Mar-2008  phx branches: 1.2.2; 1.2.4;
Including assym.h is not needed and will break the build, because it does not
exist that early. Deleted it.
Approved by garbled.
 1.1 21-Feb-2008  garbled Add tuned powerpc assembler written by IBM and released under a 3-clause
BSD Lisc as part of the perflib project.
http://sourceforge.net/projects/ppcperflib/

Tested the new functions with microbenchmarks on a number of different
CPU types, and found that most cpus either benefited greatly, or were
unaffected. Primarily G4 CPU's were unaffected, and all others showed
speedups. My 7044 (POWER3) went from a 70.6 to a 73.2 (thats good) in
bytebench with a complete release built with these. Also passed
regression tests.
 1.2.4.2 24-Mar-2008  keiichi sync with head.
 1.2.4.1 06-Mar-2008  keiichi file memcpy.S was added on branch keiichi-mipv6 on 2008-03-24 07:14:29 +0000
 1.2.2.2 23-Mar-2008  matt sync with HEAD
 1.2.2.1 06-Mar-2008  matt file memcpy.S was added on branch matt-armv6 on 2008-03-23 00:12:43 +0000
 1.3.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 15-Jan-2011  matt branches: 1.3.48;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.2 06-Mar-2008  phx branches: 1.2.2; 1.2.4;
Including assym.h is not needed and will break the build, because it does not
exist that early. Deleted it.
Approved by garbled.
 1.1 21-Feb-2008  garbled Add tuned powerpc assembler written by IBM and released under a 3-clause
BSD Lisc as part of the perflib project.
http://sourceforge.net/projects/ppcperflib/

Tested the new functions with microbenchmarks on a number of different
CPU types, and found that most cpus either benefited greatly, or were
unaffected. Primarily G4 CPU's were unaffected, and all others showed
speedups. My 7044 (POWER3) went from a 70.6 to a 73.2 (thats good) in
bytebench with a complete release built with these. Also passed
regression tests.
 1.2.4.2 24-Mar-2008  keiichi sync with head.
 1.2.4.1 06-Mar-2008  keiichi file memmove.S was added on branch keiichi-mipv6 on 2008-03-24 07:14:29 +0000
 1.2.2.2 23-Mar-2008  matt sync with HEAD
 1.2.2.1 06-Mar-2008  matt file memmove.S was added on branch matt-armv6 on 2008-03-23 00:12:43 +0000
 1.3.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.48.1 21-Apr-2020  martin Sync with HEAD
 1.6 15-Jan-2011  matt branches: 1.6.48;
Use END(foo) everywhere.
Make __cerror hidden.
Use non-PLT calls to __cerror.
Use assym.h when appropriate.
Use addi to adjust stack instead of loading it.
Add __RCSIDs
Force -D_NOREGNAMES for all .S files.
[this is all in preperation for secure plt support]
 1.5 15-Aug-2006  ross Fix bug involving addresses >= 4G on ppc64.
 1.4 07-Jul-2006  ross revert
 1.3 07-Jul-2006  ross And yet more .S files...
 1.2 23-Dec-2005  skrll Fix PR 32363 by defining _NOREGNAMES.

Someone should complete the transition.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.6.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.48.1 21-Apr-2020  martin Sync with HEAD
 1.7 07-Mar-2014  matt branches: 1.7.26;
Now the sources in ../../arch/powerpc
 1.6 04-Jan-2009  pooka branches: 1.6.8; 1.6.14;
allow inclusion of atomic ops in librump
 1.5 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.4 11-Feb-2008  ad Only build atomic ops for libkern/libc.
 1.3 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.2 28-Nov-2007  ad branches: 1.2.4;
ppc64 atomic ops.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:21:03 +0000
 1.6.14.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.8.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.6 07-Mar-2014  matt Use the sources in ../../arch/powerpc instead
 1.5 08-Mar-2009  he branches: 1.5.6; 1.5.12;
Follow up the powerpc side by using %rN instead of rN for registers.
 1.4 25-May-2008  chs branches: 1.4.6;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
ppc64 atomic ops.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_add.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_add.S was added on branch matt-armv6 on 2008-01-09 01:21:03 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.6.1 13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.5.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6 07-Mar-2014  matt Use the sources in ../../arch/powerpc instead
 1.5 08-Mar-2009  he branches: 1.5.6; 1.5.12;
Follow up the powerpc side by using %rN instead of rN for registers.
 1.4 25-May-2008  chs branches: 1.4.6;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
ppc64 atomic ops.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_and.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_and.S was added on branch matt-armv6 on 2008-01-09 01:21:04 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.6.1 13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.5.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.7 07-Mar-2014  matt Use the sources in ../../arch/powerpc instead
 1.6 08-Mar-2009  he branches: 1.6.6; 1.6.12;
Follow up the powerpc side by using %rN instead of rN for registers.
 1.5 25-May-2008  chs branches: 1.5.6;
enable profiling of assembly functions.
 1.4 28-Apr-2008  martin branches: 1.4.2;
Remove clause 3 and 4 from TNF licenses
 1.3 10-Feb-2008  ad branches: 1.3.4;
Add atomic_cas_foo_ni().
 1.2 28-Nov-2007  ad branches: 1.2.4;
ppc64 atomic ops.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_cas.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_cas.S was added on branch matt-armv6 on 2008-01-09 01:21:04 +0000
 1.3.4.2 04-Jun-2008  yamt sync with head
 1.3.4.1 18-May-2008  yamt sync with head.
 1.4.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.5.6.1 13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.6.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6 07-Mar-2014  matt Use the sources in ../../arch/powerpc instead
 1.5 08-Mar-2009  he branches: 1.5.6; 1.5.12;
Follow up the powerpc side by using %rN instead of rN for registers.
 1.4 25-May-2008  chs branches: 1.4.6;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
ppc64 atomic ops.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_dec.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_dec.S was added on branch matt-armv6 on 2008-01-09 01:21:04 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.6.1 13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.5.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6 07-Mar-2014  matt Use the sources in ../../arch/powerpc instead
 1.5 08-Mar-2009  he branches: 1.5.6; 1.5.12;
Follow up the powerpc side by using %rN instead of rN for registers.
 1.4 25-May-2008  chs branches: 1.4.6;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
ppc64 atomic ops.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_inc.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_inc.S was added on branch matt-armv6 on 2008-01-09 01:21:05 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.6.1 13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.5.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.5 07-Mar-2014  matt Use the sources in ../../arch/powerpc instead
 1.4 22-Mar-2010  mrg branches: 1.4.6; 1.4.12;
define _NOREGNAMES before including <machine/asm.h>, like powerpc does.

from dennis.c.ferguson@gmail.com in PR#43022.
 1.3 28-Apr-2008  martin Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
ppc64 atomic ops.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_op_asm.h was added on branch matt-armv6 on 2008-01-09 01:21:05 +0000
 1.4.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6 07-Mar-2014  matt Use the sources in ../../arch/powerpc instead
 1.5 08-Mar-2009  he branches: 1.5.6; 1.5.12;
Follow up the powerpc side by using %rN instead of rN for registers.
 1.4 25-May-2008  chs branches: 1.4.6;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
ppc64 atomic ops.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_or.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_or.S was added on branch matt-armv6 on 2008-01-09 01:21:05 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.6.1 13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.5.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6 07-Mar-2014  matt Use the sources in ../../arch/powerpc instead
 1.5 08-Mar-2009  he branches: 1.5.6; 1.5.12;
Follow up the powerpc side by using %rN instead of rN for registers.
 1.4 25-May-2008  chs branches: 1.4.6;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 28-Nov-2007  ad branches: 1.2.4; 1.2.8;
ppc64 atomic ops.
 1.1 26-Apr-2007  thorpej branches: 1.1.2;
file atomic_swap.S was initially added on branch thorpej-atomic.
 1.1.2.1 26-Apr-2007  thorpej First-cut at powerpc64 atomic ops.
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file atomic_swap.S was added on branch matt-armv6 on 2008-01-09 01:21:06 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.6.1 13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.5.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2 07-Mar-2014  matt Use the sources in ../../arch/powerpc instead
 1.1 28-Nov-2007  ad branches: 1.1.4; 1.1.30; 1.1.36;
ppc64 atomic ops.
 1.1.36.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.30.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.4.2 09-Jan-2008  matt sync with HEAD
 1.1.4.1 28-Nov-2007  matt file membar_ops.S was added on branch matt-armv6 on 2008-01-09 01:21:06 +0000
 1.1 07-Jul-2006  ross branches: 1.1.94;
add reachover ffs and strlen for ppc64
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 07-Jul-2006  ross branches: 1.1.94;
add reachover ffs and strlen for ppc64
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.4 01-Jun-2019  maxv Misc changes in RISC-V. Start changing the memory layout, too.
 1.3 17-Apr-2019  mrg add missing atomic_and_16_nv_cas.c atomic_and_8_nv_cas.c.
 1.2 16-Oct-2014  dennis branches: 1.2.16;
Add missing C11 atomic support functions to repair prior
build breakage. matt@ made me do this.
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Sync with HEAD
 1.2.16.1 10-Jun-2019  christos Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.3 17-Apr-2019  mrg add missing aliases for _atomic_cas_32_ni and atomic_cas_uint_ni.
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Sync with HEAD
 1.2.16.1 10-Jun-2019  christos Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 18-May-2015  justin branches: 1.2.16;
Do not set CRT_ALIAS for rump kernel; symbols provided by toolchain
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.3 07-May-2023  skrll RISC-V support that works on QEMU with a single hart.

Thanks for Simon Burge for plic(4).
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.2 27-Mar-2015  matt branches: 1.2.16;
Switch to new ABI (a0/a1 replace v0/v1)
 1.1 19-Sep-2014  matt New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.2.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.16.1 21-Apr-2020  martin Sync with HEAD
 1.4 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.3 09-Apr-2022  riastradh riscv/membar_ops: Upgrade membar_enter from W/RW to RW/RW.

This will be deprecated soon but let's avoid leaving rakes to trip on
with it arising from disagreement over the documentation (W/RW) and
implementation and usage (R/RW).
 1.2 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.1 19-Sep-2014  matt branches: 1.1.16;
New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.1 19-Sep-2014  matt branches: 1.1.16;
New files for Userland support of UCB RISC-V (both 32-bit and 64-bit)
 1.1.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.16.1 21-Apr-2020  martin Sync with HEAD
 1.9 30-Mar-2023  riastradh libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.
For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.

XXX pullup-10
 1.8 28-Feb-2019  isaki branches: 1.8.14;
Add missing atomic_and_{8,16}_nv_cas.c for __sync_and_and_fetch_{1,2}.
XXX why is not only atomic_and_* symmetric unlike the others?
(in common/lib/libc/atomic/)
 1.7 13-Oct-2014  martin branches: 1.7.16;
Provide <atomic> C++ 2011 support functions for mips and sh3.
 1.6 22-Feb-2014  martin branches: 1.6.4;
Add missing __sync_* ops to libc
 1.5 04-Jan-2009  pooka branches: 1.5.8; 1.5.14;
allow inclusion of atomic ops in librump
 1.4 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.3 11-Feb-2008  ad Only build atomic ops for libkern/libc.
 1.2 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.1 29-Nov-2007  ad branches: 1.1.4;
Atomic ops for sh3.
 1.1.4.3 23-Mar-2008  matt sync with HEAD
 1.1.4.2 09-Jan-2008  matt sync with HEAD
 1.1.4.1 29-Nov-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:21:07 +0000
 1.5.14.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.8.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.4.1 12-Nov-2014  snj Pull up following revision(s) (requested by martin in ticket #218):
common/lib/libc/arch/arm/atomic/Makefile.inc: revision 1.24-1.26
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.7
common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.18
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.7
common/lib/libc/atomic/atomic_and_16_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_and_8_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_16.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_32.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_8.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_cas_by_cas32.c: revision 1.4
common/lib/libc/atomic/atomic_op_namespace.h: revision 1.7
Add __sync_val_compare_and_swap_{1,2} aliases for _atomic_cas_{8,16}
--
Provide __atomic_compare_exchange_N (as needed for the C11 2011 <atomic>
ops) via the corresponding CAS.
--
Hook __atomic_compare_exchange_N into vax libc.
--
Provide __sync_and_and_fetch_2 and __sync_and_and_fetch_1 for pre-ARMv6,
they are needed for the C++ 2011 <atomic> stuff.
--
Add C++ 2011 <atomic> support functions.
--
Move the and_{16,8}_nv sources into the right (libc only) block.
--
Provide <atomic> C++ 2011 support functions for mips and sh3.
--
Provide C++ 2011 <atomic> support functions for hppa and arm.
--
Provide prototypes to fix build with clang.
 1.7.16.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.16.2 21-Apr-2020  martin Sync with HEAD
 1.7.16.1 10-Jun-2019  christos Sync with HEAD
 1.8.14.1 31-Jul-2023  martin Pull up following revision(s) (requested by riastradh in ticket #275):

common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.24
common/lib/libc/arch/m68k/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.15
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.9
common/lib/libc/atomic/atomic_is_lock_free.c: revision 1.1
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.9

libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.

For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.
 1.3 23-Apr-2006  uwe Now that bcopy.S doesn't contain the actual implemenation, move it
back to libc.
 1.2 22-Apr-2006  uwe Move the code into memcpy.S. Adjust the other files accordingly.
Same binaries are generated.
 1.1 22-Dec-2005  christos move from libc.
 1.3 04-Jul-2011  mrg branches: 1.3.44;
sh3 needs the __ffssi2 weak symbol, too.
 1.2 28-Apr-2008  martin Remove clause 3 and 4 from TNF licenses
 1.1 20-Dec-2005  christos branches: 1.1.26;
Merge libkern + libc common files. As requested by core.
 1.1.26.1 18-May-2008  yamt sync with head.
 1.3.44.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.44.1 21-Apr-2020  martin Sync with HEAD
 1.3 07-Feb-2024  msaitoh Remove ryo@'s mail addresses.
 1.2 22-Apr-2006  uwe branches: 1.2.94;
Move the code into memcpy.S. Adjust the other files accordingly.
Same binaries are generated.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.94.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Apr-2006  uwe branches: 1.2.94;
Move the code into memcpy.S. Adjust the other files accordingly.
Same binaries are generated.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.94.1 21-Apr-2020  martin Sync with HEAD
 1.2 16-Feb-2008  apb branches: 1.2.70;
For all files under src/common, as well as related man pages under
src/lib/libc, or related headers under src/sys/sys: change u_intNN_t to
uintNN_t.
 1.1 20-Dec-2005  christos branches: 1.1.6; 1.1.18;
Merge libkern + libc common files. As requested by core.
 1.1.18.1 23-Mar-2008  matt sync with HEAD
 1.1.6.1 18-Nov-2008  snj Pull up following revision(s) (requested by tsutsui in ticket #1224):
common/lib/libc/gen/bswap64.c: revision 1.2
common/lib/libprop/prop_data.c: revision 1.10
common/lib/libc/inet/inet_addr.c: revision 1.2
common/lib/libc/hash/rmd160/rmd160.c: revision 1.4
common/lib/libc/gen/bswap32.c: revision 1.2
common/lib/libc/gen/bswap16.c: revision 1.2
common/lib/libc/arch/alpha/gen/byte_swap_4.S: revision 1.3
common/lib/libc/arch/sh3/string/memset.S: revision 1.2
common/lib/libc/arch/alpha/gen/byte_swap_2.S: revision 1.3
common/lib/libc/md/md5c.c: revision 1.3
common/lib/libc/hash/sha1/sha1.c: revision 1.3
common/lib/libc/md/md4c.c: revision 1.3
sys/sys/rmd160.h: revision 1.2
sys/sys/sha2.h: revision 1.2
For all files under src/common, as well as related man pages under
src/lib/libc, or related headers under src/sys/sys: change u_intNN_t to
uintNN_t.
 1.2.70.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.70.1 21-Apr-2020  martin Sync with HEAD
 1.24 30-Mar-2023  riastradh libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.
For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.

XXX pullup-10
 1.23 28-Feb-2019  isaki branches: 1.23.14;
Add missing atomic_and_{8,16}_nv_cas.c for __sync_and_and_fetch_{1,2}.
XXX why is not only atomic_and_* symmetric unlike the others?
(in common/lib/libc/atomic/)
 1.22 29-May-2016  nakayama branches: 1.22.16;
Use sparc64 code always on 32-bit sparc64 kernels since %psr read/write
instructions used in sparc's atomic_cas.S don't exist in SPARC-V9.
 1.21 27-Feb-2016  joerg When using clang, do not use the sparc64 code as it doesn't provide the
full set of symbols and clang in -m32 defaults to sparcv8, not
sparcv8plus.
 1.20 27-Feb-2016  joerg Include C11 load/store helpers.
 1.19 18-Oct-2015  nakayama Fix 64-bit atomic operations on 32-bit kernel and userland, and add
support for missing __sync_*_8 builtins.
 1.18 13-Oct-2014  martin Add C++ 2011 <atomic> support functions.
 1.17 21-Feb-2014  martin branches: 1.17.4;
Provide the missing __sync_* ops for sparc.
 1.16 29-Jan-2014  martin Provide _atomic_cas_{16,8}_up
 1.15 28-Nov-2012  martin Provide an assembler version of _atomic_cas_up for sparc - the C code
does not compile to something usable in a RAS. See PR 38482.
 1.14 13-Mar-2009  abs branches: 1.14.6; 1.14.12;
Revert test from MACHINE_ARCH to MACHINE, as this can be called with
with MACHINE_ARCH=sparc MACHINE=sparc64 when building 32bit kernels for
ultrasparc hardware. Picked up by nakayama@ - thanks.
Add a comment explaining this odd case
 1.13 12-Mar-2009  abs Prefer MACHINE_ARCH to MACHINE in some tests
 1.12 10-Jan-2009  pooka branches: 1.12.2;
Include userland version of atomic cas routines in librump.
(the kernel version disables interrupts and therefore can't be used)
 1.11 04-Jan-2009  pooka allow inclusion of atomic ops in librump
 1.10 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.9 07-Mar-2008  ad oops
 1.8 07-Mar-2008  ad In libc, use the MI atomic_cas(). The assembly version is faster, but I
think there are PIC issues with it and I don't have a way to fix and test.
 1.7 11-Feb-2008  ad branches: 1.7.2;
Only build atomic ops for libkern/libc.
 1.6 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.5 22-Jan-2008  nakayama Search files in arch/sparc64/atomic before arch/sparc/atomic in case of
sparc64 32-bit kernel.

Make sparc64 32-bit kernel works again.
 1.4 22-Dec-2007  mrg branches: 1.4.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.
 1.3 29-Nov-2007  ad Use the CAS-based inc/dec variants, since these CPUs don't have atomic
add in hardware (does arm?).
 1.2 29-Nov-2007  ad sparc atomic ops
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc.
 1.4.2.3 23-Mar-2008  matt sync with HEAD
 1.4.2.2 09-Jan-2008  matt sync with HEAD
 1.4.2.1 22-Dec-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:21:07 +0000
 1.7.2.1 24-Mar-2008  keiichi sync with head.
 1.12.2.1 13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.14.12.2 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.14.12.1 25-Feb-2013  tls resync with head
 1.14.6.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.14.6.1 16-Jan-2013  yamt sync with (a bit old) head
 1.17.4.1 12-Nov-2014  snj Pull up following revision(s) (requested by martin in ticket #218):
common/lib/libc/arch/arm/atomic/Makefile.inc: revision 1.24-1.26
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.7
common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.18
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.7
common/lib/libc/atomic/atomic_and_16_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_and_8_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_16.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_32.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_8.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_cas_by_cas32.c: revision 1.4
common/lib/libc/atomic/atomic_op_namespace.h: revision 1.7
Add __sync_val_compare_and_swap_{1,2} aliases for _atomic_cas_{8,16}
--
Provide __atomic_compare_exchange_N (as needed for the C11 2011 <atomic>
ops) via the corresponding CAS.
--
Hook __atomic_compare_exchange_N into vax libc.
--
Provide __sync_and_and_fetch_2 and __sync_and_and_fetch_1 for pre-ARMv6,
they are needed for the C++ 2011 <atomic> stuff.
--
Add C++ 2011 <atomic> support functions.
--
Move the and_{16,8}_nv sources into the right (libc only) block.
--
Provide <atomic> C++ 2011 support functions for mips and sh3.
--
Provide C++ 2011 <atomic> support functions for hppa and arm.
--
Provide prototypes to fix build with clang.
 1.22.16.2 21-Apr-2020  martin Sync with HEAD
 1.22.16.1 10-Jun-2019  christos Sync with HEAD
 1.23.14.1 31-Jul-2023  martin Pull up following revision(s) (requested by riastradh in ticket #275):

common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.24
common/lib/libc/arch/m68k/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.15
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.9
common/lib/libc/atomic/atomic_is_lock_free.c: revision 1.1
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.9

libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.

For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_add.c was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_and.c was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc.
 1.13 21-Feb-2014  martin branches: 1.13.26;
Provide the missing __sync_* ops for sparc.
 1.12 12-Sep-2013  joerg Pass PICFLAGS down to cc-as-as and use __PIC__ to decide if it is small
vs big PIC mode. Retire -DPIC and -DBIGPIC.
 1.11 22-Feb-2011  pooka branches: 1.11.4; 1.11.10;
Add PIC variant for fetching the lock address.

librump.so now works on sparc
 1.10 13-Mar-2009  nakayama branches: 1.10.4;
Fix typo. (s/_HARKDERNEL/_HARDKERNEL/)
 1.9 12-Jan-2009  pooka branches: 1.9.2;
include sys/param.h for _HARDKERNEL instead of homegrown def.
 1.8 10-Jan-2009  pooka Include userland version of atomic cas routines in librump.
(the kernel version disables interrupts and therefore can't be used)
 1.7 25-May-2008  chs enable profiling of assembly functions.
 1.6 28-Apr-2008  martin branches: 1.6.2;
Remove clause 3 and 4 from TNF licenses
 1.5 10-Feb-2008  ad branches: 1.5.4;
Add atomic_cas_foo_ni().
 1.4 08-Dec-2007  ad branches: 1.4.4;
Add some goo to make this work in userspace.
 1.3 08-Dec-2007  ad objdump -d to the rescue. I think this should work now.
 1.2 29-Nov-2007  ad sparc atomic ops
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_cas.S was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc.
 1.4.4.3 23-Mar-2008  matt sync with HEAD
 1.4.4.2 09-Jan-2008  matt sync with HEAD
 1.4.4.1 08-Dec-2007  matt file atomic_cas.S was added on branch matt-armv6 on 2008-01-09 01:21:08 +0000
 1.5.4.2 04-Jun-2008  yamt sync with head
 1.5.4.1 18-May-2008  yamt sync with head.
 1.6.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.9.2.1 13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.10.4.1 05-Mar-2011  bouyer Sync with HEAD
 1.11.10.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.11.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.13.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.13.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.2 29-Jan-2014  martin branches: 1.2.26;
Provide _atomic_cas_{16,8}_up
 1.1 28-Nov-2012  martin branches: 1.1.2; 1.1.6;
Provide an assembler version of _atomic_cas_up for sparc - the C code
does not compile to something usable in a RAS. See PR 38482.
 1.1.6.3 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.6.2 25-Feb-2013  tls resync with head
 1.1.6.1 28-Nov-2012  tls file atomic_cas_up.S was added on branch tls-maxphys on 2013-02-25 00:24:00 +0000
 1.1.2.3 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.2.2 16-Jan-2013  yamt sync with (a bit old) head
 1.1.2.1 28-Nov-2012  yamt file atomic_cas_up.S was added on branch yamt-pagecache on 2013-01-16 05:25:53 +0000
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_dec.c was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_inc.c was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc.
 1.7 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.6 17-Jan-2011  joerg branches: 1.6.48;
Move SPARC and SPARC64 to modern CPP. Update UPDATING note for this and
Alpha.
 1.5 28-Apr-2008  martin Remove clause 3 and 4 from TNF licenses
 1.4 22-Jan-2008  nakayama branches: 1.4.4;
Use __arch64__ instead of __sparc64__, since -D__sparc64__ is appended
to build option of sparc64 32-bit kernel.
 1.3 22-Dec-2007  mrg branches: 1.3.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.
 1.2 29-Nov-2007  ad sparc atomic ops
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc.
 1.3.2.3 23-Mar-2008  matt sync with HEAD
 1.3.2.2 09-Jan-2008  matt sync with HEAD
 1.3.2.1 22-Dec-2007  matt file atomic_op_asm.h was added on branch matt-armv6 on 2008-01-09 01:21:08 +0000
 1.4.4.1 18-May-2008  yamt sync with head.
 1.6.48.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.48.1 21-Apr-2020  martin Sync with HEAD
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_or.c was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_swap.c was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc.
 1.9 06-Sep-2025  riastradh paravirt_membar_sync(9): New memory barrier.

For use in paravirtualized drivers which require store-before-load
ordering -- irrespective of whether the kernel is built for a single
processor, or whether the (virtual) machine is booted with a single
processor.

This is even required on architectures that don't even have a
store-before-load ordering barrier, like m68k; adding, e.g., a virtio
bus is _as if_ the architecture has been extended with relaxed memory
ordering when talking with that new bus. Such architectures need
some way to request the hypervisor enforce that ordering -- on m68k,
that's done by issuing a CASL instruction, which qemu maps to an
atomic r/m/w with sequential consistency ordering in the host.

PR kern/59618: occasional virtio block device lock ups/hangs
 1.8 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.7 09-Apr-2022  riastradh sparc/membar_ops: Upgrade membar_enter from R/RW to RW/RW.

This will be deprecated soon but let's avoid leaving rakes to trip on
with it arising from disagreement over the documentation (W/RW) and
implementation and usage (R/RW).
 1.6 09-Apr-2022  riastradh sparc: Fix membar_sync with LDSTUB.

membar_sync is required to be a full sequential consistency barrier,
equivalent to MEMBAR #StoreStore|LoadStore|StoreLoad|LoadLoad on
sparcv9. LDSTUB and SWAP are the only pre-v9 instructions that do
this and SWAP doesn't exist on all v7 hardware, so use LDSTUB.

Note: I'm having a hard time nailing down a reference for the
ordering implied by LDSTUB and SWAP. I'm _pretty sure_ SWAP has to
imply store-load ordering since the SPARCv8 manual recommends it for
Dekker's algorithm (which notoriously requires store-load ordering),
and the formal memory model treats LDSTUB and SWAP the same for
ordering. But the v8 and v9 manuals aren't clear.

GCC issues STBAR and LDSTUB, but (a) I don't see why STBAR is
necessary here, (b) STBAR doesn't exist on v7 so it'd be a pain to
use, and (c) from what I've heard (although again it's hard to nail
down authoritative references here) all actual SPARC hardware is TSO
or SC anyway so STBAR is a noop in all the silicon anyway.

Either way, certainly this is better than what we had before, which
was nothing implying ordering at all, just a store!
 1.5 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.4 25-May-2008  chs branches: 1.4.62;
enable profiling of assembly functions.
 1.3 28-Apr-2008  martin branches: 1.3.2;
Remove clause 3 and 4 from TNF licenses
 1.2 29-Nov-2007  ad branches: 1.2.4; 1.2.8;
Fix the aliases.
 1.1 29-Nov-2007  ad sparc atomic ops
 1.2.8.2 04-Jun-2008  yamt sync with head
 1.2.8.1 18-May-2008  yamt sync with head.
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 29-Nov-2007  matt file membar_ops.S was added on branch matt-armv6 on 2008-01-09 01:21:08 +0000
 1.3.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.4.62.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.62.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.4 12-Sep-2013  joerg branches: 1.4.26;
Pass PICFLAGS down to cc-as-as and use __PIC__ to decide if it is small
vs big PIC mode. Retire -DPIC and -DBIGPIC.
 1.3 04-Jul-2011  mrg branches: 1.3.2; 1.3.8;
add a weak alias from ffs to __ffssi2. sparc dynamic works with gcc 4.5 now.
 1.2 19-Dec-2009  pooka Don't export __ffstab, it's used only in this routine. This gets
rid of one of those evil common symbols in the __ namespace.

reviewed by martin
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.3.8.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.4.26.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.6 04-Jan-2009  pooka branches: 1.6.50;
allow inclusion of atomic ops in librump
 1.5 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.4 11-Feb-2008  ad Only build atomic ops for libkern/libc.
 1.3 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.2 28-Nov-2007  ad branches: 1.2.4;
Pull in sparc64 atomic ops from the thorpej-atomic branch.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc64.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:21:09 +0000
 1.6.50.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.50.1 21-Apr-2020  martin Sync with HEAD
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 18-Oct-2015  nakayama branches: 1.7.16;
Fix 64-bit atomic operations on 32-bit kernel and userland, and add
support for missing __sync_*_8 builtins.
 1.6 25-May-2008  chs enable profiling of assembly functions.
 1.5 28-Apr-2008  martin branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses
 1.4 09-Apr-2008  nakayama branches: 1.4.2;
Use %xcc for 64-bit comparisons.
 1.3 22-Dec-2007  mrg branches: 1.3.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.
 1.2 28-Nov-2007  ad Pull in sparc64 atomic ops from the thorpej-atomic branch.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_add.S was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc64.
 1.3.2.2 09-Jan-2008  matt sync with HEAD
 1.3.2.1 22-Dec-2007  matt file atomic_add.S was added on branch matt-armv6 on 2008-01-09 01:21:09 +0000
 1.4.2.2 04-Jun-2008  yamt sync with head
 1.4.2.1 18-May-2008  yamt sync with head.
 1.5.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.7.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.16.1 21-Apr-2020  martin Sync with HEAD
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 18-Oct-2015  nakayama branches: 1.7.16;
Fix 64-bit atomic operations on 32-bit kernel and userland, and add
support for missing __sync_*_8 builtins.
 1.6 25-May-2008  chs enable profiling of assembly functions.
 1.5 28-Apr-2008  martin branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses
 1.4 09-Apr-2008  nakayama branches: 1.4.2;
Use %xcc for 64-bit comparisons.
 1.3 22-Dec-2007  mrg branches: 1.3.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.
 1.2 28-Nov-2007  ad Pull in sparc64 atomic ops from the thorpej-atomic branch.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_and.S was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc64.
 1.3.2.2 09-Jan-2008  matt sync with HEAD
 1.3.2.1 22-Dec-2007  matt file atomic_and.S was added on branch matt-armv6 on 2008-01-09 01:21:10 +0000
 1.4.2.2 04-Jun-2008  yamt sync with head
 1.4.2.1 18-May-2008  yamt sync with head.
 1.5.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.7.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.16.1 21-Apr-2020  martin Sync with HEAD
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 18-Oct-2015  nakayama branches: 1.7.16;
Fix 64-bit atomic operations on 32-bit kernel and userland, and add
support for missing __sync_*_8 builtins.
 1.6 25-May-2008  chs enable profiling of assembly functions.
 1.5 28-Apr-2008  martin branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses
 1.4 10-Feb-2008  ad branches: 1.4.4;
Add atomic_cas_foo_ni().
 1.3 22-Dec-2007  mrg branches: 1.3.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.
 1.2 28-Nov-2007  ad Pull in sparc64 atomic ops from the thorpej-atomic branch.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_cas.S was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc64.
 1.3.2.3 23-Mar-2008  matt sync with HEAD
 1.3.2.2 09-Jan-2008  matt sync with HEAD
 1.3.2.1 22-Dec-2007  matt file atomic_cas.S was added on branch matt-armv6 on 2008-01-09 01:21:10 +0000
 1.4.4.2 04-Jun-2008  yamt sync with head
 1.4.4.1 18-May-2008  yamt sync with head.
 1.5.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.7.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.16.1 21-Apr-2020  martin Sync with HEAD
 1.9 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.8 18-Oct-2015  nakayama branches: 1.8.16;
Fix 64-bit atomic operations on 32-bit kernel and userland, and add
support for missing __sync_*_8 builtins.
 1.7 25-May-2008  chs enable profiling of assembly functions.
 1.6 28-Apr-2008  martin branches: 1.6.2;
Remove clause 3 and 4 from TNF licenses
 1.5 09-Apr-2008  nakayama branches: 1.5.2;
Use %xcc for 64-bit comparisons.
 1.4 22-Dec-2007  mrg branches: 1.4.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.
 1.3 30-Nov-2007  ad This is atomic_dec, not atomic_inc!
 1.2 28-Nov-2007  ad Pull in sparc64 atomic ops from the thorpej-atomic branch.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_dec.S was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc64.
 1.4.2.2 09-Jan-2008  matt sync with HEAD
 1.4.2.1 22-Dec-2007  matt file atomic_dec.S was added on branch matt-armv6 on 2008-01-09 01:21:11 +0000
 1.5.2.2 04-Jun-2008  yamt sync with head
 1.5.2.1 18-May-2008  yamt sync with head.
 1.6.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.8.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.16.1 21-Apr-2020  martin Sync with HEAD
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 18-Oct-2015  nakayama branches: 1.7.16;
Fix 64-bit atomic operations on 32-bit kernel and userland, and add
support for missing __sync_*_8 builtins.
 1.6 25-May-2008  chs enable profiling of assembly functions.
 1.5 28-Apr-2008  martin branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses
 1.4 09-Apr-2008  nakayama branches: 1.4.2;
Use %xcc for 64-bit comparisons.
 1.3 22-Dec-2007  mrg branches: 1.3.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.
 1.2 28-Nov-2007  ad Pull in sparc64 atomic ops from the thorpej-atomic branch.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_inc.S was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc64.
 1.3.2.2 09-Jan-2008  matt sync with HEAD
 1.3.2.1 22-Dec-2007  matt file atomic_inc.S was added on branch matt-armv6 on 2008-01-09 01:21:12 +0000
 1.4.2.2 04-Jun-2008  yamt sync with head
 1.4.2.1 18-May-2008  yamt sync with head.
 1.5.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.7.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.16.1 21-Apr-2020  martin Sync with HEAD
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 18-Oct-2015  nakayama branches: 1.7.16;
Fix 64-bit atomic operations on 32-bit kernel and userland, and add
support for missing __sync_*_8 builtins.
 1.6 17-Jan-2011  joerg Move SPARC and SPARC64 to modern CPP. Update UPDATING note for this and
Alpha.
 1.5 28-Apr-2008  martin Remove clause 3 and 4 from TNF licenses
 1.4 22-Jan-2008  nakayama branches: 1.4.4;
Use __arch64__ instead of __sparc64__, since -D__sparc64__ is appended
to build option of sparc64 32-bit kernel.
 1.3 22-Dec-2007  mrg branches: 1.3.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.
 1.2 28-Nov-2007  ad Pull in sparc64 atomic ops from the thorpej-atomic branch.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc64.
 1.3.2.3 23-Mar-2008  matt sync with HEAD
 1.3.2.2 09-Jan-2008  matt sync with HEAD
 1.3.2.1 22-Dec-2007  matt file atomic_op_asm.h was added on branch matt-armv6 on 2008-01-09 01:21:12 +0000
 1.4.4.1 18-May-2008  yamt sync with head.
 1.7.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.16.1 21-Apr-2020  martin Sync with HEAD
 1.9 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.8 18-Oct-2015  nakayama branches: 1.8.16;
Fix 64-bit atomic operations on 32-bit kernel and userland, and add
support for missing __sync_*_8 builtins.
 1.7 25-May-2008  chs enable profiling of assembly functions.
 1.6 28-Apr-2008  martin branches: 1.6.2;
Remove clause 3 and 4 from TNF licenses
 1.5 09-Apr-2008  nakayama branches: 1.5.2;
Use %xcc for 64-bit comparisons.
 1.4 12-Feb-2008  ad Correct a dodgy alias.
 1.3 22-Dec-2007  mrg branches: 1.3.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.
 1.2 28-Nov-2007  ad Pull in sparc64 atomic ops from the thorpej-atomic branch.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_or.S was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc64.
 1.3.2.3 23-Mar-2008  matt sync with HEAD
 1.3.2.2 09-Jan-2008  matt sync with HEAD
 1.3.2.1 22-Dec-2007  matt file atomic_or.S was added on branch matt-armv6 on 2008-01-09 01:21:12 +0000
 1.5.2.2 04-Jun-2008  yamt sync with head
 1.5.2.1 18-May-2008  yamt sync with head.
 1.6.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.8.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.8.16.1 21-Apr-2020  martin Sync with HEAD
 1.8 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.7 18-Oct-2015  nakayama branches: 1.7.16;
Fix 64-bit atomic operations on 32-bit kernel and userland, and add
support for missing __sync_*_8 builtins.
 1.6 25-May-2008  chs enable profiling of assembly functions.
 1.5 28-Apr-2008  martin branches: 1.5.2;
Remove clause 3 and 4 from TNF licenses
 1.4 09-Apr-2008  nakayama branches: 1.4.2;
Use %xcc for 64-bit comparisons.
 1.3 22-Dec-2007  mrg branches: 1.3.2;
- provide 32 bit versions of the sparc v9 atomic ops.
- use them on sparc64 32 bit kernels.

fixes issues on 32 bit sparc64 kernels that were using sparc v8 ops.
tested by rjs@.
 1.2 28-Nov-2007  ad Pull in sparc64 atomic ops from the thorpej-atomic branch.
 1.1 27-Apr-2007  thorpej branches: 1.1.2;
file atomic_swap.S was initially added on branch thorpej-atomic.
 1.1.2.1 27-Apr-2007  thorpej First cut at atomic ops for sparc64.
 1.3.2.2 09-Jan-2008  matt sync with HEAD
 1.3.2.1 22-Dec-2007  matt file atomic_swap.S was added on branch matt-armv6 on 2008-01-09 01:21:13 +0000
 1.4.2.2 04-Jun-2008  yamt sync with head
 1.4.2.1 18-May-2008  yamt sync with head.
 1.5.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.7.16.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.16.1 21-Apr-2020  martin Sync with HEAD
 1.10 06-Sep-2025  riastradh paravirt_membar_sync(9): New memory barrier.

For use in paravirtualized drivers which require store-before-load
ordering -- irrespective of whether the kernel is built for a single
processor, or whether the (virtual) machine is booted with a single
processor.

This is even required on architectures that don't even have a
store-before-load ordering barrier, like m68k; adding, e.g., a virtio
bus is _as if_ the architecture has been extended with relaxed memory
ordering when talking with that new bus. Such architectures need
some way to request the hypervisor enforce that ordering -- on m68k,
that's done by issuing a CASL instruction, which qemu maps to an
atomic r/m/w with sequential consistency ordering in the host.

PR kern/59618: occasional virtio block device lock ups/hangs
 1.9 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.8 09-Apr-2022  riastradh sparc64/membar_ops: Upgrade membar_enter from R/RW to RW/RW.

This will be deprecated soon but let's avoid leaving rakes to trip on
with it arising from disagreement over the documentation (W/RW) and
implementation and usage (R/RW).
 1.7 09-Apr-2022  riastradh sparc64: Fix membar_sync by issuing membar #StoreLoad.

In TSO this is the only memory barrier ever needed, and somehow we
got this wrong and instead issued an unnecessary membar #LoadLoad --
not needed even in PSO let alone in TSO.

XXX Apparently we may run userland programs with PSO or RMO, in which
case all of these membars need fixing:

PSO RMO
membar_consumer nop membar #LoadLoad
membar_producer membar #StoreStore membar #StoreStore
membar_enter nop membar #LoadLoad|LoadStore
membar_exit membar #StoreStore membar #LoadStore|StoreStore
membar_sync membar #StoreLoad|StoreStore
membar #...everything...

But at least this fixes the TSO case in which we run the kernel.
Also I'm not sure there's any non-TSO hardware out there in practice.
 1.6 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.5 25-May-2008  chs branches: 1.5.62;
enable profiling of assembly functions.
 1.4 28-Apr-2008  martin branches: 1.4.2;
Remove clause 3 and 4 from TNF licenses
 1.3 12-Jan-2008  ad branches: 1.3.4;
Don't put membar in the delay slot. thorpej@ says it may be problematic.
 1.2 29-Nov-2007  ad branches: 1.2.4;
Fix the aliases.
 1.1 28-Nov-2007  ad Memory barriers for sparc64. XXX Can the 'membar' go in retl delay slot?
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 29-Nov-2007  matt file membar_ops.S was added on branch matt-armv6 on 2008-01-09 01:21:13 +0000
 1.3.4.2 04-Jun-2008  yamt sync with head
 1.3.4.1 18-May-2008  yamt sync with head.
 1.4.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.5.62.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.62.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.3 12-Sep-2013  joerg branches: 1.3.26;
Pass PICFLAGS down to cc-as-as and use __PIC__ to decide if it is small
vs big PIC mode. Retire -DPIC and -DBIGPIC.
 1.2 19-Dec-2009  pooka branches: 1.2.6; 1.2.12;
Don't export __ffstab, it's used only in this routine. This gets
rid of one of those evil common symbols in the __ namespace.

reviewed by martin
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 17-Mar-2013  christos branches: 1.2.6; 1.2.12; 1.2.36;
Add RCSID
Fix out of date comment
 1.1 17-Mar-2013  christos Use a single copy of the source.
 1.2.36.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.36.1 21-Apr-2020  martin Sync with HEAD
 1.2.12.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.12.1 17-Mar-2013  yamt file memcpy.S was added on branch yamt-pagecache on 2014-05-22 11:26:30 +0000
 1.2.6.2 23-Jun-2013  tls resync from head
 1.2.6.1 17-Mar-2013  tls file memcpy.S was added on branch tls-maxphys on 2013-06-23 06:26:13 +0000
 1.3 13-Aug-2021  andvar fix typos in words "pointer" and s/fram /frame/
 1.2 17-Mar-2013  christos branches: 1.2.6; 1.2.12; 1.2.36;
add RCSID; use clr for pattern
 1.1 17-Mar-2013  christos Use a single copy of the source.
 1.2.36.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.36.1 21-Apr-2020  martin Sync with HEAD
 1.2.12.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.12.1 17-Mar-2013  yamt file memset.S was added on branch yamt-pagecache on 2014-05-22 11:26:30 +0000
 1.2.6.2 23-Jun-2013  tls resync from head
 1.2.6.1 17-Mar-2013  tls file memset.S was added on branch tls-maxphys on 2013-06-23 06:26:13 +0000
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 17-Mar-2013  christos branches: 1.1.6; 1.1.12; 1.1.36;
Use a single copy of the source.
 1.1.36.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.36.1 21-Apr-2020  martin Sync with HEAD
 1.1.12.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.1.12.1 17-Mar-2013  yamt file strmacros.h was added on branch yamt-pagecache on 2014-05-22 11:26:30 +0000
 1.1.6.2 23-Jun-2013  tls resync from head
 1.1.6.1 17-Mar-2013  tls file strmacros.h was added on branch tls-maxphys on 2013-06-23 06:26:13 +0000
 1.9 30-Mar-2023  riastradh libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.
For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.

XXX pullup-10
 1.8 28-Feb-2019  isaki branches: 1.8.14;
Add missing atomic_and_{8,16}_nv_cas.c for __sync_and_and_fetch_{1,2}.
XXX why is not only atomic_and_* symmetric unlike the others?
(in common/lib/libc/atomic/)
 1.7 12-Oct-2014  martin branches: 1.7.16;
Hook __atomic_compare_exchange_N into vax libc.
 1.6 22-Feb-2014  martin branches: 1.6.4;
Add the missing __sync_* ops to libc
 1.5 04-Jan-2009  pooka branches: 1.5.8; 1.5.14;
allow inclusion of atomic ops in librump
 1.4 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.3 11-Feb-2008  ad branches: 1.3.4;
Only build atomic ops for libkern/libc.
 1.2 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.1 03-Feb-2008  matt Add the .C versions for the atomic ops.
 1.3.4.2 23-Mar-2008  matt sync with HEAD
 1.3.4.1 11-Feb-2008  matt file Makefile.inc was added on branch matt-armv6 on 2008-03-23 00:12:43 +0000
 1.5.14.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.8.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.4.1 12-Nov-2014  snj Pull up following revision(s) (requested by martin in ticket #218):
common/lib/libc/arch/arm/atomic/Makefile.inc: revision 1.24-1.26
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.13
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.7
common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.18
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.7
common/lib/libc/atomic/atomic_and_16_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_and_8_nv_cas.c: revision 1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_16.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_32.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_c11_compare_exchange_cas_8.c: revision 1.1-1.2
common/lib/libc/atomic/atomic_cas_by_cas32.c: revision 1.4
common/lib/libc/atomic/atomic_op_namespace.h: revision 1.7
Add __sync_val_compare_and_swap_{1,2} aliases for _atomic_cas_{8,16}
--
Provide __atomic_compare_exchange_N (as needed for the C11 2011 <atomic>
ops) via the corresponding CAS.
--
Hook __atomic_compare_exchange_N into vax libc.
--
Provide __sync_and_and_fetch_2 and __sync_and_and_fetch_1 for pre-ARMv6,
they are needed for the C++ 2011 <atomic> stuff.
--
Add C++ 2011 <atomic> support functions.
--
Move the and_{16,8}_nv sources into the right (libc only) block.
--
Provide <atomic> C++ 2011 support functions for mips and sh3.
--
Provide C++ 2011 <atomic> support functions for hppa and arm.
--
Provide prototypes to fix build with clang.
 1.7.16.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.16.2 21-Apr-2020  martin Sync with HEAD
 1.7.16.1 10-Jun-2019  christos Sync with HEAD
 1.8.14.1 31-Jul-2023  martin Pull up following revision(s) (requested by riastradh in ticket #275):

common/lib/libc/arch/sparc/atomic/Makefile.inc: revision 1.24
common/lib/libc/arch/m68k/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/mips/atomic/Makefile.inc: revision 1.16
common/lib/libc/arch/hppa/atomic/Makefile.inc: revision 1.15
common/lib/libc/arch/vax/atomic/Makefile.inc: revision 1.9
common/lib/libc/atomic/atomic_is_lock_free.c: revision 1.1
common/lib/libc/arch/sh3/atomic/Makefile.inc: revision 1.9

libc: Define __atomic_is_lock_free.

Limited to architectures where it is actually needed by gcc for any
calls to stdatomic.h atomic_is_lock_free for now.

We should also add it to other architectures too, along with lockful
atomic r/m/w operations for sizes that can't be handled natively, but
that's a lot more work. It is also necessary for -fno-inline-atomics
but we're missing a lot of other symbols for that too, to be fixed.

For now, this should enable the OpenSSL build to complete on these
architectures again after I reverted a local change.
 1.3 25-Jan-2011  matt branches: 1.3.46;
Use END(sym)
use RCSID
 1.2 28-Apr-2008  martin branches: 1.2.20;
Remove clause 3 and 4 from TNF licenses
 1.1 31-Mar-2007  matt branches: 1.1.12;
Add these so kernels built -O0 will link.
 1.1.12.1 18-May-2008  yamt sync with head.
 1.2.20.1 08-Feb-2011  bouyer Sync with HEAD
 1.3.46.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.46.1 21-Apr-2020  martin Sync with HEAD
 1.3 25-Jan-2011  matt branches: 1.3.46;
Use END(sym)
use RCSID
 1.2 28-Apr-2008  martin branches: 1.2.20;
Remove clause 3 and 4 from TNF licenses
 1.1 31-Mar-2007  matt branches: 1.1.12;
Add these so kernels built -O0 will link.
 1.1.12.1 18-May-2008  yamt sync with head.
 1.2.20.1 08-Feb-2011  bouyer Sync with HEAD
 1.3.46.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.46.1 21-Apr-2020  martin Sync with HEAD
 1.3 25-Jan-2011  matt branches: 1.3.46;
Use END(sym)
use RCSID
 1.2 24-Dec-2005  matt branches: 1.2.48;
Don't include "DEFS.h" when <machine/asm.h> is enough.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.48.1 08-Feb-2011  bouyer Sync with HEAD
 1.3.46.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.46.1 21-Apr-2020  martin Sync with HEAD
 1.3 25-Jan-2011  matt branches: 1.3.46;
Use END(sym)
use RCSID
 1.2 24-Dec-2005  matt branches: 1.2.48;
Don't include "DEFS.h" when <machine/asm.h> is enough.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.48.1 08-Feb-2011  bouyer Sync with HEAD
 1.3.46.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.46.1 21-Apr-2020  martin Sync with HEAD
 1.3 25-Jan-2011  matt branches: 1.3.46;
Use END(sym)
use RCSID
 1.2 24-Dec-2005  matt branches: 1.2.48;
Don't include "DEFS.h" when <machine/asm.h> is enough.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.48.1 08-Feb-2011  bouyer Sync with HEAD
 1.3.46.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.46.1 21-Apr-2020  martin Sync with HEAD
 1.3 25-Jan-2011  matt branches: 1.3.46;
Use END(sym)
use RCSID
 1.2 24-Dec-2005  matt branches: 1.2.48;
Don't include "DEFS.h" when <machine/asm.h> is enough.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.48.1 08-Feb-2011  bouyer Sync with HEAD
 1.3.46.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.46.1 21-Apr-2020  martin Sync with HEAD
 1.3 25-Jan-2011  matt branches: 1.3.46;
Use END(sym)
use RCSID
 1.2 24-Dec-2005  matt branches: 1.2.48;
Don't include "DEFS.h" when <machine/asm.h> is enough.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.48.1 08-Feb-2011  bouyer Sync with HEAD
 1.3.46.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.46.1 21-Apr-2020  martin Sync with HEAD
 1.2 25-Jan-2011  matt branches: 1.2.46;
Use END(sym)
use RCSID
 1.1 12-Mar-2010  uwe branches: 1.1.4;
Add memset to libkern on vax, moving existing memset.S to src/common.

While kernel source code is supposed to use <machine/macros.h> via
<sys/param.h> and pick an inline asm version from there, gcc itself
can generate direct calls to memset() for structure initialization, so
the symbol must be provided.

Fixes INSTALL kernel, for which -Os caused gcc to use memset() call to
initialize a struct in uvm_fault_internal().
 1.1.4.1 08-Feb-2011  bouyer Sync with HEAD
 1.2.46.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.46.1 21-Apr-2020  martin Sync with HEAD
 1.6 04-Jan-2009  pooka branches: 1.6.50;
allow inclusion of atomic ops in librump
 1.5 29-Sep-2008  ad Allow atomic ops to be built as part of libpthread.
 1.4 11-Feb-2008  ad Only build atomic ops for libkern/libc.
 1.3 10-Feb-2008  ad Enable the atomic ops in userspace.
 1.2 28-Nov-2007  ad branches: 1.2.4;
x86 atomic ops.
 1.1 18-Apr-2007  thorpej branches: 1.1.2;
file Makefile.inc was initially added on branch thorpej-atomic.
 1.1.2.2 26-Apr-2007  thorpej *_nv() ops implemented in terms of CMPXCHG directly. This is much tighter
code than what GCC can genererate for the generic-in-C versions.
 1.1.2.1 18-Apr-2007  thorpej Build glue for amd64.
 1.2.4.3 23-Mar-2008  matt sync with HEAD
 1.2.4.2 09-Jan-2008  matt sync with HEAD
 1.2.4.1 28-Nov-2007  matt file Makefile.inc was added on branch matt-armv6 on 2008-01-09 01:21:14 +0000
 1.6.50.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.50.1 21-Apr-2020  martin Sync with HEAD
 1.32 06-Sep-2025  riastradh paravirt_membar_sync(9): New memory barrier.

For use in paravirtualized drivers which require store-before-load
ordering -- irrespective of whether the kernel is built for a single
processor, or whether the (virtual) machine is booted with a single
processor.

This is even required on architectures that don't even have a
store-before-load ordering barrier, like m68k; adding, e.g., a virtio
bus is _as if_ the architecture has been extended with relaxed memory
ordering when talking with that new bus. Such architectures need
some way to request the hypervisor enforce that ordering -- on m68k,
that's done by issuing a CASL instruction, which qemu maps to an
atomic r/m/w with sequential consistency ordering in the host.

PR kern/59618: occasional virtio block device lock ups/hangs
 1.31 16-Jul-2024  riastradh amd64: Fix performance regression in uniprocessor atomics/membars.

Back in 2022, I eliminated the MFENCE hotpatch in membar_sync because
it's essentially always more expensive than LOCK ADD with no benefit
for CPU/CPU store-before-load ordering. (It is relevant only for
non-temporal stores or write-combining memory.)

https://mail-index.netbsd.org/source-changes/2022/07/30/msg140047.html

But in that change, I made a mistake and _also_ eliminated the LOCK
hotpatch on uniprocessor amd64. And our assembler gas helpfully
interprets uppercase LOCK just like lowercase lock and assembles them
the same way, so I didn't notice.

This change restores the LOCK hotpatch, so that when booting on a
uniprocessor system (or a uniprocessor guest on a multicore host),
the LOCK prefix is replaced by NOP for a cheaper instruction.

Found by puzzling over how my explanation for PR kern/57199 could
possibly be correct when (on an amd64 guest) ddb x/i membar_sync kept
showing the lock prefix even in uniprocessor boots.
 1.30 16-Jul-2024  riastradh xen: Don't hotpatch away LOCK prefix in xen_mb, even on UP boots.

Both xen_mb and membar_sync are designed to provide store-before-load
ordering, but xen_mb has to provide it in synchronizing guest with
hypervisor, while membar_sync only has to provide it in synchronizing
one (guest) CPU with another (guest) CPU.

It is safe to hotpatch away the LOCK prefix in membar_sync on a
uniprocessor boot because membar_sync is only designed to coordinate
between normal memory on multiple CPUs, and is never necessary when
there's only one CPU involved.

But xen_mb is used to coordinate between the guest and the `device'
implemented by a hypervisor, which might be running on another
_physical_ CPU even if the NetBSD guest only sees one `CPU', i.e.,
one _virtual_ CPU. So even on `uniprocessor' boots, xen_mb must
still issue an instruction with store-before-load ordering on
multiprocessor systems, such as a LOCK ADD (or MFENCE, but MFENCE is
costlier for no benefit here).

No need to change xen_wmb (release ordering, load/store-before-store)
or xen_rmb (acquire ordering, load-before-load/store) because every
x86 store is a store-release and every x86 load is a load-acquire,
even on multiprocessor systems, so there's no hotpatching involved
anyway.

PR kern/57199
 1.29 30-Jul-2022  riastradh branches: 1.29.2; 1.29.8;
x86: Eliminate mfence hotpatch for membar_sync.

The more-compatible LOCK ADD $0,-N(%rsp) turns out to be cheaper
than MFENCE anyway. Let's save some space and maintenance and rip
out the hotpatching for it.
 1.28 09-Apr-2022  riastradh Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.
 1.27 09-Apr-2022  riastradh x86_64/membar_ops: Upgrade membar_enter from R/RW to RW/RW.

This will be deprecated soon but let's avoid leaving rakes to trip on
with it arising from disagreement over the documentation (W/RW) and
implementation and usage (R/RW).
 1.26 09-Apr-2022  riastradh x86: Add a note on membar_sync and mfence.
 1.25 09-Apr-2022  riastradh x86: Omit needless store in membar_producer/exit.

On x86, every store is a store-release, so there is no need for any
barrier. But this wasn't a barrier anyway; it was just a store,
which was redundant with the store of the return address to the stack
implied by CALL even if issuing a store made a difference.
 1.24 09-Apr-2022  riastradh x86: Every load is a load-acquire, so membar_consumer is a noop.

lfence is only needed for MD logic, such as operations on I/O memory
rather than normal cacheable memory, or special instructions like
RDTSC -- never for MI synchronization between threads/CPUs. No need
for hot-patching to do lfence here.

(The x86_lfence function might reasonably be patched on i386 to do
lfence for MD logic, but it isn't now and this doesn't change that.)
 1.23 06-Apr-2022  riastradh Nix trailing whitespace in files of membars, atomics, and lock stubs.

Will be touching many of these files soon for functional changes.

No functional change intended.
 1.22 26-Apr-2020  maxv Use the hotpatch framework for LFENCE/MFENCE.
 1.21 26-Apr-2020  maxv Remove unused argument in macro.
 1.20 26-Apr-2020  maxv Remove unused.
 1.19 26-Apr-2020  maxv Drop the hardcoded array, use the hotpatch section.
 1.18 17-Feb-2019  isaki Add missing export of atomic_or_64 (since rev1.1).
 1.17 22-May-2014  uebayasi branches: 1.17.24;
Put missing END() markers to set ELF symbol size.
 1.16 12-Jan-2011  joerg branches: 1.16.12; 1.16.24;
Allow use of traditional CPP to be set on a per platform base in sys.mk.
Honour this for dependency processing in bsd.dep.mk. Switch i386 and
amd64 assembly to use ISO C90 preprocessor concat and drop the
-traditional-cpp on this platform.
 1.15 26-Nov-2009  pooka Use strong alias within the kernel namespace regardless of if we're
dealing with a hard or soft kernel (kernel linker doesn't support
weak symbols).
 1.14 12-Jan-2009  pooka include sys/param.h for _HARDKERNEL instead of homegrown def.
 1.13 04-Jan-2009  pooka Do not use lockpatches with _RUMPKERNEL (non-PIC ... and pointless).
 1.12 25-May-2008  chs enable profiling of assembly functions.
 1.11 03-May-2008  yamt branches: 1.11.2;
rename END to ENDLABEL. i'll use END for other purpose. ok by Andrew Doran.
 1.10 28-Apr-2008  martin Remove clause 3 and 4 from TNF licenses
 1.9 10-Feb-2008  ad branches: 1.9.4;
Add atomic_cas_foo_ni().
 1.8 09-Feb-2008  ad membar_enter was doing the wrong thing. For x86 we can alias:

membar_enter -> membar_consumer
membar_exit -> membar_producer
 1.7 10-Dec-2007  ad branches: 1.7.4;
Fix _atomic_cas_64. Noted by bouyer@.
 1.6 09-Dec-2007  ad Add missing strong aliases; sure I did this before?
 1.5 29-Nov-2007  ad Fix ia32 -> amd64 thinko.
 1.4 29-Nov-2007  ad atomic_add_* takes signed integers, the others take unsigned.
 1.3 28-Nov-2007  ad A lock prefix on xchg is meaningless.
 1.2 28-Nov-2007  ad Fix up a few minor problems.
 1.1 28-Nov-2007  ad x86 atomic ops.
 1.7.4.3 23-Mar-2008  matt sync with HEAD
 1.7.4.2 09-Jan-2008  matt sync with HEAD
 1.7.4.1 10-Dec-2007  matt file atomic.S was added on branch matt-armv6 on 2008-01-09 01:21:14 +0000
 1.9.4.2 04-Jun-2008  yamt sync with head
 1.9.4.1 18-May-2008  yamt sync with head.
 1.11.2.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.16.24.1 10-Aug-2014  tls Rebase.
 1.16.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.17.24.3 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.17.24.2 21-Apr-2020  martin Sync with HEAD
 1.17.24.1 10-Jun-2019  christos Sync with HEAD
 1.29.8.1 02-Aug-2025  perseant Sync with HEAD
 1.29.2.1 20-Jul-2024  martin Pull up following revision(s) (requested by riastradh in ticket #764):

common/lib/libc/arch/i386/atomic/atomic.S: revision 1.37
sys/arch/xen/include/xenring.h: revision 1.8
sys/arch/i386/i386/cpufunc.S: revision 1.52
sys/arch/amd64/amd64/cpufunc.S: revision 1.68
sys/arch/xen/include/hypervisor.h: revision 1.60
common/lib/libc/arch/x86_64/atomic/atomic.S: revision 1.30

xen: Don't hotpatch away LOCK prefix in xen_mb, even on UP boots.

Both xen_mb and membar_sync are designed to provide store-before-load
ordering, but xen_mb has to provide it in synchronizing guest with
hypervisor, while membar_sync only has to provide it in synchronizing
one (guest) CPU with another (guest) CPU.

It is safe to hotpatch away the LOCK prefix in membar_sync on a
uniprocessor boot because membar_sync is only designed to coordinate
between normal memory on multiple CPUs, and is never necessary when
there's only one CPU involved.

But xen_mb is used to coordinate between the guest and the `device'
implemented by a hypervisor, which might be running on another
_physical_ CPU even if the NetBSD guest only sees one `CPU', i.e.,
one _virtual_ CPU. So even on `uniprocessor' boots, xen_mb must
still issue an instruction with store-before-load ordering on
multiprocessor systems, such as a LOCK ADD (or MFENCE, but MFENCE is
costlier for no benefit here).

No need to change xen_wmb (release ordering, load/store-before-store)
or xen_rmb (acquire ordering, load-before-load/store) because every
x86 store is a store-release and every x86 load is a load-acquire,
even on multiprocessor systems, so there's no hotpatching involved
anyway.

PR kern/57199
 1.1 17-Apr-2007  thorpej branches: 1.1.2;
file atomic_add.S was initially added on branch thorpej-atomic.
 1.1.2.3 26-Apr-2007  thorpej *_nv() ops implemented in terms of CMPXCHG directly. This is much tighter
code than what GCC can genererate for the generic-in-C versions.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 17-Apr-2007  thorpej First draft implementation of atomic ops for amd64
 1.1 17-Apr-2007  thorpej branches: 1.1.2;
file atomic_and.S was initially added on branch thorpej-atomic.
 1.1.2.3 26-Apr-2007  thorpej *_nv() ops implemented in terms of CMPXCHG directly. This is much tighter
code than what GCC can genererate for the generic-in-C versions.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 17-Apr-2007  thorpej First draft implementation of atomic ops for amd64
 1.1 17-Apr-2007  thorpej branches: 1.1.2;
file atomic_cas.S was initially added on branch thorpej-atomic.
 1.1.2.3 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.2 17-Apr-2007  thorpej Fix typo.
 1.1.2.1 17-Apr-2007  thorpej First draft implementation of atomic ops for amd64
 1.1 17-Apr-2007  thorpej branches: 1.1.2;
file atomic_dec.S was initially added on branch thorpej-atomic.
 1.1.2.3 26-Apr-2007  thorpej *_nv() ops implemented in terms of CMPXCHG directly. This is much tighter
code than what GCC can genererate for the generic-in-C versions.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 17-Apr-2007  thorpej First draft implementation of atomic ops for amd64
 1.1 17-Apr-2007  thorpej branches: 1.1.2;
file atomic_inc.S was initially added on branch thorpej-atomic.
 1.1.2.3 26-Apr-2007  thorpej *_nv() ops implemented in terms of CMPXCHG directly. This is much tighter
code than what GCC can genererate for the generic-in-C versions.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 17-Apr-2007  thorpej First draft implementation of atomic ops for amd64
 1.1 17-Apr-2007  thorpej branches: 1.1.2;
file atomic_op_asm.h was initially added on branch thorpej-atomic.
 1.1.2.2 17-Apr-2007  thorpej amd64 doesn't have opt_cputype.h
 1.1.2.1 17-Apr-2007  thorpej First draft implementation of atomic ops for amd64
 1.1 17-Apr-2007  thorpej branches: 1.1.2;
file atomic_or.S was initially added on branch thorpej-atomic.
 1.1.2.3 26-Apr-2007  thorpej *_nv() ops implemented in terms of CMPXCHG directly. This is much tighter
code than what GCC can genererate for the generic-in-C versions.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 17-Apr-2007  thorpej First draft implementation of atomic ops for amd64
 1.1 17-Apr-2007  thorpej branches: 1.1.2;
file atomic_swap.S was initially added on branch thorpej-atomic.
 1.1.2.2 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.1 17-Apr-2007  thorpej First draft implementation of atomic ops for amd64
 1.1 17-Apr-2007  thorpej branches: 1.1.2;
file membar_ops.S was initially added on branch thorpej-atomic.
 1.1.2.4 22-Apr-2007  thorpej Make sure namespace-cleansed aliases are avaialble for all atomic ops.
 1.1.2.3 17-Apr-2007  thorpej Tidy up a comment.
 1.1.2.2 17-Apr-2007  thorpej amd64 doesn't have opt_cputype.h
 1.1.2.1 17-Apr-2007  thorpej First draft implementation of atomic ops for amd64
 1.3 22-May-2014  uebayasi branches: 1.3.24;
Put missing END() markers to set ELF symbol size.
 1.2 04-Feb-2006  uwe branches: 1.2.56; 1.2.70;
libc wants __bswapNN, kernel wants bswapNN. That was not accounted
for during the merge of kernel and libc versions. Fix to match
e.g. i386 code.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.70.1 10-Aug-2014  tls Rebase.
 1.2.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.24.1 21-Apr-2020  martin Sync with HEAD
 1.3 22-May-2014  uebayasi branches: 1.3.24;
Put missing END() markers to set ELF symbol size.
 1.2 04-Feb-2006  uwe branches: 1.2.56; 1.2.70;
libc wants __bswapNN, kernel wants bswapNN. That was not accounted
for during the merge of kernel and libc versions. Fix to match
e.g. i386 code.
 1.1 20-Dec-2005  christos Merge libkern + libc common files. As requested by core.
 1.2.70.1 10-Aug-2014  tls Rebase.
 1.2.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.24.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-May-2014  uebayasi branches: 1.2.24;
Put missing END() markers to set ELF symbol size.
 1.1 14-Jan-2010  joerg branches: 1.1.12; 1.1.24;
Move AMD64's bswap64 implementation from libc to src/common and share it
with the kernel.
 1.1.24.1 10-Aug-2014  tls Rebase.
 1.1.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.24.1 21-Apr-2020  martin Sync with HEAD
 1.5 27-Jan-2020  ad x86 uses the C versions of bcmp() and memcmp() now.
 1.4 15-Jan-2020  ad Rewrite bcmp() & memcmp() to not use REP CMPS. Seems about 5-10x faster for
small strings on modern hardware.
 1.3 22-Mar-2014  jakllsch branches: 1.3.26; 1.3.30;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.2 12-Nov-2007  ad branches: 1.2.28; 1.2.34;
Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.
 1.1 20-Dec-2005  christos branches: 1.1.18;
Merge libkern + libc common files. As requested by core.
 1.1.18.1 09-Jan-2008  matt sync with HEAD
 1.2.34.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.28.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.30.1 29-Feb-2020  ad Sync with head.
 1.3.26.1 08-Apr-2020  martin Merge changes from current as of 20200406
 1.5 22-Mar-2014  jakllsch branches: 1.5.26;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.4 22-Nov-2009  dsl branches: 1.4.6; 1.4.12;
Align to the destination buffer.
This probably costs 1 clock (on modern cpus) in the normal case.
But gives a big benefit when the destination is misaligned.
In particular when the source has the same misalignment - although
that may not be a gain on Nehalem!
Fixes PR/35535
 1.3 21-Nov-2009  dsl Avoid doing two 'rep movs' operations.
 1.2 12-Nov-2007  ad Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.
 1.1 20-Dec-2005  christos branches: 1.1.18;
Merge libkern + libc common files. As requested by core.
 1.1.18.1 09-Jan-2008  matt sync with HEAD
 1.4.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.4 23-Nov-2013  jakllsch Remove x86_64 bzero.S; which since 2009 has only contained instructions
that it should be removed in 2010.
 1.3 01-Aug-2009  dsl branches: 1.3.6; 1.3.12;
Remove some long dependant instruction sequences (ie allow parallel code).
Since 'rep stos' will have a long setup time, avoid doing it more than once.
For misaligned (start address or length) write an unaligned word at both
ends of the buffer then aligned 'rep stosd' the middle.
Use the same code for bzero().
bzero.S is left being compiled for a while (empty) - to avoid issues with
duplicate symbols in libc.a after update builds.
 1.2 12-Nov-2007  ad Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.
 1.1 20-Dec-2005  christos branches: 1.1.18;
Merge libkern + libc common files. As requested by core.
 1.1.18.1 09-Jan-2008  matt sync with HEAD
 1.3.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.3.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.5 22-Mar-2014  jakllsch branches: 1.5.26;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.4 20-Jul-2009  christos branches: 1.4.6; 1.4.12;
Put back dsl's string changes, but fix memchr.S to use cmp so that the
condition code is set (and fix the comments 0x10->0x01). From Anon Ymous
We need a test for memchr(x, -1)...
 1.3 19-Jul-2009  christos revert changes that made new kernels hang in ACPI detection
 1.2 18-Jul-2009  dsl Remove a pointless _ALIGN_TEXT.
XXX ffs() ought to be a gcc inline asm.
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.4.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.4.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.5.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 17-Jul-2009  dsl Delete files that are no longer needed.
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.6 22-Mar-2014  jakllsch branches: 1.6.26;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.5 01-Aug-2009  dsl branches: 1.5.6; 1.5.12;
In the misaligned case, xor the read word with the target pattern
before making the unwanted bytes non-zero.
Means that memchr(buf, 0xff) is no longer a special case.
 1.4 20-Jul-2009  christos Put back dsl's string changes, but fix memchr.S to use cmp so that the
condition code is set (and fix the comments 0x10->0x01). From Anon Ymous
We need a test for memchr(x, -1)...
 1.3 19-Jul-2009  christos revert changes that made new kernels hang in ACPI detection
 1.2 18-Jul-2009  dsl A better memchr().
Always read aligned words, invalidating unwanted bytes in first word,
and checking that any match in the last word is before the buffer end.
No loops apart from the one through the data.
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.5.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.6 27-Jan-2020  ad x86 uses the C versions of bcmp() and memcmp() now.
 1.5 16-Jan-2020  ad Back out previous, it's broken.
 1.4 15-Jan-2020  ad Rewrite bcmp() & memcmp() to not use REP CMPS. Seems about 5-10x faster for
small strings on modern hardware.
 1.3 22-Mar-2014  jakllsch branches: 1.3.26; 1.3.30;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.2 12-Nov-2007  ad branches: 1.2.28; 1.2.34;
Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.
 1.1 20-Dec-2005  christos branches: 1.1.18;
Merge libkern + libc common files. As requested by core.
 1.1.18.1 09-Jan-2008  matt sync with HEAD
 1.2.34.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.28.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.30.1 29-Feb-2020  ad Sync with head.
 1.3.26.1 08-Apr-2020  martin Merge changes from current as of 20200406
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.1 20-Dec-2005  christos branches: 1.1.94;
Merge libkern + libc common files. As requested by core.
 1.1.94.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.1.94.1 21-Apr-2020  martin Sync with HEAD
 1.5 22-May-2014  pooka branches: 1.5.24;
fix build for _KERNEL
 1.4 22-May-2014  uebayasi Put missing END() markers to set ELF symbol size.
 1.3 01-Aug-2009  dsl branches: 1.3.12; 1.3.24;
Remove some long dependant instruction sequences (ie allow parallel code).
Since 'rep stos' will have a long setup time, avoid doing it more than once.
For misaligned (start address or length) write an unaligned word at both
ends of the buffer then aligned 'rep stosd' the middle.
Use the same code for bzero().
bzero.S is left being compiled for a while (empty) - to avoid issues with
duplicate symbols in libc.a after update builds.
 1.2 12-Nov-2007  ad Don't unconditionally clear the direction flag. The ABI says it must always
be clear when making a function call, and 'cld' takes about 50 clock cyles
on the P4.
 1.1 20-Dec-2005  christos branches: 1.1.18;
Merge libkern + libc common files. As requested by core.
 1.1.18.1 09-Jan-2008  matt sync with HEAD
 1.3.24.1 10-Aug-2014  tls Rebase.
 1.3.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.24.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.5.24.1 21-Apr-2020  martin Sync with HEAD
 1.2 17-Jul-2009  dsl Delete files that are no longer needed.
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.7 22-Mar-2014  jakllsch branches: 1.7.26;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.6 20-Jul-2009  christos branches: 1.6.6; 1.6.12;
Put back dsl's string changes, but fix memchr.S to use cmp (rather
than test) so that the condition code is set correctly (and fix the
comments: 0x10->0x01 and ^->&). From Anon Ymous

XXX: There are similar comment errors in some of the other string code.

XXX: We really need a regression test that includes misaligned memory
with searches designed to catch corner cases such as searching for 0,
-1, etc, and search length limit violations. Searching for 0 on
misaligned memory would have caught this problem.
 1.5 19-Jul-2009  christos revert changes that made new kernels hang in ACPI detection
 1.4 18-Jul-2009  dsl Shorten a dependency chain by using 'sbb, xor' (at a time when carry is set)
instead of 'mov, neg, dec'.
('mov, not' can't be used because it doesn't set the flags.)
 1.3 18-Jul-2009  dsl Replace with a version that:
1) doesn't do byte compares to find which byte matched
2) doesn't do byte compares if any top bits are set
3) doesn't use a loop when the input is misaligned
4) has less mispredicted branches
Passes regression tests and 'build.sh' doesn't explode (and more than usual).
 1.2 17-Jul-2009  dsl Change all archs so that strchr.[cS] and strrchr.[cS] exist and generate
duplicate symbols for index() and rindex().
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.6.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.6.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.7.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.7.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.2 22-Mar-2014  jakllsch branches: 1.2.26;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.1 20-Dec-2005  christos branches: 1.1.50; 1.1.56;
Merge libkern + libc common files. As requested by core.
 1.1.56.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.1.50.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.2.26.1 21-Apr-2020  martin Sync with HEAD
 1.8 30-Mar-2024  andvar s/Westley/Wesley/ in a book reference (in comments).
 1.7 08-Dec-2021  andvar s/efficent/efficient/ in comments.
 1.6 22-Mar-2014  jakllsch branches: 1.6.26;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.5 12-Jul-2009  dsl branches: 1.5.6; 1.5.12;
Add netbsd copyright.
Reorder a few instructions to interleave a dependency chain.
(I'm really not sure of the best order for those instructions!)
 1.4 12-Jul-2009  dsl Correct some comments
 1.3 11-Jul-2009  dsl After alg 2 triggers, mask with ~x (alg 3) to ignore bytes with top bit set.
Then use bit scan to work out which byte is zero.
If the source is misaligned read the aligned word and make the unwanted
(low order) bytes non-zero.
Passes regression test - which probably tests just enough cases.
 1.2 11-Jul-2009  dsl Change comments about algorithms, 99.6% for random data isn't 'rare' in my book!
(The 'rare' case is any byte values 0x80-0xff.)
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.5.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.5.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.6.26.1 21-Apr-2020  martin Sync with HEAD
 1.3 22-Mar-2014  jakllsch branches: 1.3.26;
For all x86_64 string assembly functions that don't overlap (i.e. every
one except memset and bzero) use END() so that symbol size information
is available.
 1.2 17-Jul-2009  dsl branches: 1.2.6; 1.2.12;
Change all archs so that strchr.[cS] and strrchr.[cS] exist and generate
duplicate symbols for index() and rindex().
 1.1 20-Dec-2005  christos branches: 1.1.36;
Merge libkern + libc common files. As requested by core.
 1.1.36.1 23-Jul-2009  jym Sync with HEAD.
 1.2.12.1 19-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.6.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.26.2 21-Apr-2020  martin Ooops, restore accidently removed files from merge mishap
 1.3.26.1 21-Apr-2020  martin Sync with HEAD

RSS XML Feed