aarch64: Implement __aarch64_casN_sync.

gcc generates calls to this symbol in programs that use
__sync_*_compare_and_swap, which require full sequential consistency
barriers, including store-before-load ordering on both sides of the
atomic; none of the release/acquire operations guarantee that, so we
have to insert explicit DMB instructions.

Note: gcc's own definition omits some of the DMB instructions, but I
can't prove that it's correct that way -- stores preceding the CAS
must complete before the load part of the CAS, and the store part of
the CAS must complete before loads following the CAS. Maybe there's
some way to prove that one of these orderings is guaranteed some
other way than a DMB but I'm not seeing it, and store-before-load
ordering is hard to understand.

Patch by skrll@ based on a patch by mrg@, soliloquy in commit message
by me.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

aarch64: Implement __aarch64_casN_sync.

gcc generates calls to this symbol in programs that use
__sync_*_compare_and_swap, which require full sequential consistency
barriers, including store-before-load ordering on both sides of the
atomic; none of the release/acquire operations guarantee that, so we
have to insert explicit DMB instructions.

Note: gcc's own definition omits some of the DMB instructions, but I
can't prove that it's correct that way -- stores preceding the CAS
must complete before the load part of the CAS, and the store part of
the CAS must complete before loads following the CAS. Maybe there's
some way to prove that one of these orderings is guaranteed some
other way than a DMB but I'm not seeing it, and store-before-load
ordering is hard to understand.

Patch by skrll@ based on a patch by mrg@, soliloquy in commit message
by me.

whitespace

be consistent about comparing loaded value against expected old value
register ordering

Fix some register usage

remove stray 'w'

Whitespace

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

Provide all the LSE operation fuctions. The use of LSE instructions is
currently disabled.

Do previous differently as the API is different.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

Comment nit

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

Comment nit

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

Comment nit

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.3.2;
Sort STRONG_ALIAS's in the same manner as ATOMIC_OP_ALIAS's.
No functional changes.

Export _atomic_cas_64 as atomic_cas_64_ni.

Note that _atomic_cas_64 is already exported as atomic_cas_{ulong,prt}_ni.

Fix build error of test/lib/atomic/t_atomic_cas, which is successfully
passed on RPI3B+ now.

branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

Comment nit

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

One more s/pte/ptr/

Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows

{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

typo in comment s/pte/ptr/

Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows

{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

typo in comment s/pte/ptr/

Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows

{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

typo in comment s/pte/ptr/

Fix the logic operation for atomic_nand_{8,16,32,64}

From the gcc docs the operations are as follows

{ tmp = *ptr; *ptr = ~(tmp & value); return tmp; } // nand
{ tmp = ~(*ptr & value); *ptr = tmp; return *ptr; } // nand

yes, this is really rather strange.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Comment nit

branches: 1.3.2;
- atomic_*_{8,16}_nv() must return a new value, not an old value.
- use "dmb sy" for atomic_*{8,16}_nv() in the same way as atomic_*{32,64}_nv().

fix atomic_sub_*(). it was (delta - *ptr), should be (*ptr - delta).
changing shared macro doesn't effect other atomic_ops because
(*ptr [+|^] delta) and (delta [+|^] *ptr) have same result.

atomic_sub_*() haven't used because non standard API?

branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

__sync_or_and_fetch_8 should return new value... make it do that.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Part I of ad@'s performance improvements for aarch64

- Remove memory barriers from the atomic ops. I don't understand why those
are there. Is it some architectural thing, or for a CPU bug, or just
over-caution maybe? They're not needed for correctness.

- Have unlikely conditional branches go forwards to help the static branch
predictor.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

Do previous differently as the API is different.

Provide some more operations that are part of compiler lse.S. This is
incomplete, but at least covers all the atomic_swap ops and allows the
aa64 kernel to link with gcc 10.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...
From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.

aarch64/membar_ops: Fix wrong symbol end.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove htonll and ntohll as symbols from aarch64 libc.

Other architectures do not define them, and so we don't provide a
function declaration in any header.

This means a package may detect it with a link-test and then fail
due to the missing declaration, like sysutils/collectd currently does.

Done this way as aarch64 has not had a release yet. Discussed with releng.

branches: 1.1.4; 1.1.26; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove unused assembly source files

Fix typo/pasteo in aarch64 clzdi2() END()

branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove unused assembly source files

branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove unused assembly source files

branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove ryo@'s mail addresses.

port-arm/57388: Minor bug fix in bcopy.S

Use correct register to check alignment of destination in backwards copy.

Patch from Antoni Pokusinski. Thanks.

branches: 1.2.8;
Fixed to not use the "br" instruction. Branch Target Identification (BTI) doesn't like "br".

requested by maxv@

branches: 1.1.4;

Working / new versions from Ryo Shimizu

avoid reading from out of range that may cause access fault.

branches: 1.2.2; 1.2.4;

Working / new versions from Ryo Shimizu

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.2.4;

Working / new versions from Ryo Shimizu

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4;

Working / new versions from Ryo Shimizu

Fixed to not use the "br" instruction. Branch Target Identification (BTI) doesn't like "br".

requested by maxv@

branches: 1.2.4;
* aarch64/memset.S didn't work! fixed some bugs.
* maximum size of DCZID_EL0:BS (2048) supported.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Re-do previous aarch64eb strlen fix more simply and correctly.

Fix a broken corner case of strlen()/strnlen() on aarch64eb

Previously a string such as "\x1\x1\x1\x1\x1\x1\x1" would count as
0 instead of 7 on BE.

strnlen(s, (size_t)-1) returned -1. it must return the length of s.

branches: 1.2.2; 1.2.4;
aarch64/strlen.S didn't work. fixed some bugs.

branches: 1.1.4;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

branches: 1.1.4; 1.1.26;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.