libc: New functions c8rtomb(3) and mbrtoc8(3).

New in C23, for converting from UTF-8 to locale-dependent multibyte
sequences (c8rtomb) or vice versa (mbrtoc8), along with the new type
char8_t.

Conditional on either:
- _NETBSD_SOURCE
- _ISOC23_SOURCE
- __STDC_VERSION__ >= 202311L

(Riding the libc minor bump from this morning for the UTF-16/UTF-32
versions from C11.)

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

uchar.h: New header file for C11 (and C++11) compliance.

Implementation of the new functions mbrtoc16, c16rtomb, mbrtoc32, and
c32rtomb to come later. Updates for C23 to come later.

PR lib/52374: <uchar.h> missing

tests/lib/libc/locale/Makefile: Sort.

No functional change intended.

Preparation for PR lib/52374.

branches: 1.14.2;
Don't use fmtcheck for strfmon format strings. It does not work. Fix a broken
test.

branches: 1.13.10;
PR/54414: Valery Ushakov: add a test for wcsrtombs(3) doesn't update the
source argument on conversion error

branches: 1.12.4;
Add missing strfmon_l. Noticed by Bruno Haible. Add test case.

Add missing files from last commit:

Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.

branches: 1.10.2;
Add a simple collation test. This test is expected to fail on HEAD since
we do not yet have a working implementation of wcscoll.

branches: 1.9.2;
Add tests for btowc(3)/wctob(3) and enable compilation of the test for
digittoint(3).

The digittoint(3) test is skipped since we don't provide that function yet.

One of the test cases for btowc(3) is also skipped, since it tests conversion
to Unicode---whereas our wchar_t representation is locale-dependent.

Add test cases for sprintf/sscanf/strto{d,l} and the is* and isw* ctype functions, for single-byte encodings

Add simple test case for toupper/tolower

Add mbsnrtowcs and wcsnrtombs. Approved by core.

regression tests for wide char i/o. Currently there are failures.

branches: 1.4.6;
Add test cases for strcspn, strpbrk, strspn, wcscspn, wcspbrk and
wcsspn.

branches: 1.3.2;
Rename two test files to get functional scope (and avoid confusion
with ctype(3)). No functional change.

Fix build with stack smash protection enabled.

atf-ify the various locale tests

Add missing files from last commit:

Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.

branches: 1.1.2;
file ducet_test.h was initially added on branch perseant-stdc-iso10646.

Separate the C/POSIX locale test from the rest; make it more thorough
and more correct. This fixes a problem reported by martin@ when the
test is compiled with -funsigned-char.

Add ISO10646 versions of these tests, conditional on __STDC_ISO_10646__ .
Also make the tests a bit more verbose, to aid debugging when they fail.

branches: 1.1.2;
Add tests for btowc(3)/wctob(3) and enable compilation of the test for
digittoint(3).

The digittoint(3) test is skipped since we don't provide that function yet.

One of the test cases for btowc(3) is also skipped, since it tests conversion
to Unicode---whereas our wchar_t representation is locale-dependent.

branches: 1.6.2; 1.6.6;
c32rtomb(3): Use conversion state to handle shift sequences.

For conversion of Unicode scalar values to coding systems requiring
shift sequences, such as ISO-2022-JP, _citrus_iconv_convert will
always produce:

1. a shift sequence from the initial state to some nondefault state,
like from US-ASCII to JIS X 0208
2. the encoding of the desired characater
3. a shift sequence restoring the initial state

This is unnecessary if the output is already in the state needed to
encoded the desired character. For example, this method produces
seven bytes to encode each YEN SIGN in ISO-2022-JP -- and fourteen,
to encode two consecutive ones -- even though the shift sequence is
only three bytes long and once shifted YEN SIGN takes only one byte.

Instead, convert the Unicode scalar value to a locale-dependent wide
character and encode that, by composing

- _citrus_iconv_convert
=> gives us a multibyte encoding of the character from the initial
state (and restoring the initial state afterward)
- mbrtowc with initial conversion state
=> gives us the single wide character representation
XXX If combining characters are possible here, this may fail.
- wcrtomb with caller's conversion tsate
=> gives us a state-dependent multibyte encoding of the character

XXX Is there a cheaper way to convert from Unicode scalar value to
locale-dependent wide character? It is not obvious to me from the
largely undocumented Citrus machinery, but it would obviously be
better than this somewhat circuitous Rube Goldberg contraption of
chained multibyte APIs.

PR lib/58612: c8rtomb/c16rtomb/c32rtomb yield suboptimal shift
sequences

t_c8rtomb, t_c16rtomb: Simplify comment.

ESC $ B is technically rather the JIS X 0208-1983 shift sequence, but
since I don't see any way to provoke the JIS X 0208-1978 shift
sequence to come flying out of this conversion (ESC $ @), and I'm not
sure there's any difference in the interpretation, let's just say JIS
X 0208.

PR lib/58612: c8rtomb/c16rtomb/c32rtomb yield suboptimal shift
sequences

c8rtomb(3), c16rtomb(3), c32rtomb(3): Test stateful shift sequences.

PR lib/58612: c8rtomb/c16rtomb/c32rtomb yield suboptimal shift
sequences

c8rtomb(3), c16rtomb(3): Fix NUL handling.

PR lib/58615: incomplete c8rtomb, c16rtomb handles NUL termination
wrong

c8rtomb(3), c16rtomb(3): Add tests for incomplete NUL termination.

PR lib/58615: incomplete c8rtomb, c16rtomb handles NUL termination
wrong

libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

branches: 1.1.2; 1.1.6;
libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

branches: 1.7.2; 1.7.6;
c32rtomb(3): Use conversion state to handle shift sequences.

For conversion of Unicode scalar values to coding systems requiring
shift sequences, such as ISO-2022-JP, _citrus_iconv_convert will
always produce:

1. a shift sequence from the initial state to some nondefault state,
like from US-ASCII to JIS X 0208
2. the encoding of the desired characater
3. a shift sequence restoring the initial state

This is unnecessary if the output is already in the state needed to
encoded the desired character. For example, this method produces
seven bytes to encode each YEN SIGN in ISO-2022-JP -- and fourteen,
to encode two consecutive ones -- even though the shift sequence is
only three bytes long and once shifted YEN SIGN takes only one byte.

Instead, convert the Unicode scalar value to a locale-dependent wide
character and encode that, by composing

- _citrus_iconv_convert
=> gives us a multibyte encoding of the character from the initial
state (and restoring the initial state afterward)
- mbrtowc with initial conversion state
=> gives us the single wide character representation
XXX If combining characters are possible here, this may fail.
- wcrtomb with caller's conversion tsate
=> gives us a state-dependent multibyte encoding of the character

XXX Is there a cheaper way to convert from Unicode scalar value to
locale-dependent wide character? It is not obvious to me from the
largely undocumented Citrus machinery, but it would obviously be
better than this somewhat circuitous Rube Goldberg contraption of
chained multibyte APIs.

PR lib/58612: c8rtomb/c16rtomb/c32rtomb yield suboptimal shift
sequences

t_c8rtomb, t_c16rtomb: Simplify comment.

ESC $ B is technically rather the JIS X 0208-1983 shift sequence, but
since I don't see any way to provoke the JIS X 0208-1978 shift
sequence to come flying out of this conversion (ESC $ @), and I'm not
sure there's any difference in the interpretation, let's just say JIS
X 0208.

PR lib/58612: c8rtomb/c16rtomb/c32rtomb yield suboptimal shift
sequences

c8rtomb(3): Fix digit error in shift sequence test.

PR lib/58612: c8rtomb/c16rtomb/c32rtomb yield suboptimal shift
sequences

c8rtomb(3), c16rtomb(3), c32rtomb(3): Test stateful shift sequences.

PR lib/58612: c8rtomb/c16rtomb/c32rtomb yield suboptimal shift
sequences

c8rtomb(3), c16rtomb(3): Fix NUL handling.

PR lib/58615: incomplete c8rtomb, c16rtomb handles NUL termination
wrong

c8rtomb(3), c16rtomb(3): Add tests for incomplete NUL termination.

PR lib/58615: incomplete c8rtomb, c16rtomb handles NUL termination
wrong

libc: New functions c8rtomb(3) and mbrtoc8(3).

New in C23, for converting from UTF-8 to locale-dependent multibyte
sequences (c8rtomb) or vice versa (mbrtoc8), along with the new type
char8_t.

Conditional on either:
- _NETBSD_SOURCE
- _ISOC23_SOURCE
- __STDC_VERSION__ >= 202311L

(Riding the libc minor bump from this morning for the UTF-16/UTF-32
versions from C11.)

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

Rename two test files to get functional scope (and avoid confusion
with ctype(3)). No functional change.

atf-ify the various locale tests

Rename two test files to get functional scope (and avoid confusion
with ctype(3)). No functional change.

atf-ify the various locale tests

fix various typos in comment, documentation and log messages.

Add tests for btowc(3)/wctob(3) and enable compilation of the test for
digittoint(3).

The digittoint(3) test is skipped since we don't provide that function yet.

One of the test cases for btowc(3) is also skipped, since it tests conversion
to Unicode---whereas our wchar_t representation is locale-dependent.

Add test cases for sprintf/sscanf/strto{d,l} and the is* and isw* ctype functions, for single-byte encodings

Add missing files from last commit:

Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.

branches: 1.1.2;
file t_ducet.c was initially added on branch perseant-stdc-iso10646.

Add ISO10646 versions of these tests, conditional on __STDC_ISO_10646__ .
Also make the tests a bit more verbose, to aid debugging when they fail.

branches: 1.4.4; 1.4.20;
fix comment typos pointed out by uebayasi

- fix funopen usage
- some more checks
- remove a bogus test case (bad_eucJP_getwc) PR/47660 (Julio Merino)
- add XXX comments

branches: 1.2.4;
Mark two routinely-broken tests as expected failures referencing PR lib/47660.

regression tests for wide char i/o. Currently there are failures.

branches: 1.3.2; 1.3.6;
mbrtoc32(3): Use conversion state to handle shift sequences.

PR lib/58618: mbrtocN(3) fails to keep shift state

mbrtoc8(3), mbrtoc16(3): Test consuming shift sequences with state.

This has the side effect of testing mbrtoc32(3) because they are both
defined in terms of it.

PR lib/58618: mbrtocN(3) fails to keep shift state

libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

branches: 1.1.2; 1.1.6;
libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

branches: 1.3.2; 1.3.6;
mbrtoc32(3): Use conversion state to handle shift sequences.

PR lib/58618: mbrtocN(3) fails to keep shift state

mbrtoc8(3), mbrtoc16(3): Test consuming shift sequences with state.

This has the side effect of testing mbrtoc32(3) because they are both
defined in terms of it.

PR lib/58618: mbrtocN(3) fails to keep shift state

libc: New functions c8rtomb(3) and mbrtoc8(3).

New in C23, for converting from UTF-8 to locale-dependent multibyte
sequences (c8rtomb) or vice versa (mbrtoc8), along with the new type
char8_t.

Conditional on either:
- _NETBSD_SOURCE
- _ISOC23_SOURCE
- __STDC_VERSION__ >= 202311L

(Riding the libc minor bump from this morning for the UTF-16/UTF-32
versions from C11.)

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

Add ISO10646 versions of these tests, conditional on __STDC_ISO_10646__ .
Also make the tests a bit more verbose, to aid debugging when they fail.

branches: 1.1.34;
Rename two test files to get functional scope (and avoid confusion
with ctype(3)). No functional change.

branches: 1.2.2;
include string.h for memset

branches: 1.1.2; 1.1.6;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

adapt mbstowcs_basic test for unicode table update

reformat so it's easier to find which result data belongs to which input

branches: 1.2.16;
Add ISO10646 versions of these tests, conditional on __STDC_ISO_10646__ .
Also make the tests a bit more verbose, to aid debugging when they fail.

branches: 1.1.34;
Rename two test files to get functional scope (and avoid confusion
with ctype(3)). No functional change.

After a comedy of errors, move t_mbtowc to its final resting place.

Add a member to the test data structure that indicates whether the given
encoding is state-dependent, and test the results of wctomb(NULL, '\0') and
mbtowc(NULL, NULL, 0) against this instead of against each other.

atf-ify the various locale tests

s/diferent/different/

Since the C standard allows for intermediate floating results to contain
more precision bits than the data type expects, but (kind of obviously)
does not allow such values to be stored in memory, expecting the value
returned from strtod() (an intermediate result) to be identical (that is,
equal) to a stored value is incorrect.

So instead go back to checking that the two numbers are very very close.
See comments added to the test for more explanation.

Revert 1.4 (perhaps temporarily) and add even more diagnostics to those
added in 1.3 to see if it is possible to determine why the strict equality
test fails on i386, yet succeeds elsewhere.

When comparing doubles (any floating point values) which have been
computed using different methods, don't expect to achieve identical
results (here, one constant is perhaps converted to binary from a string by
a cross compiler, the other is converted at run time). Allow them to
have a small difference (for now, small is < 1e-7 - the constant is ~ 1e5,
so this is 12 orders of magnitude less) before failing (and include the
actual difference in the error message if it does fail.)

Add some diagnostics to the strto test, so I can see why this
fails on i386 (on qemu) - will probably keep them when done.

Add ISO10646 versions of these tests, conditional on __STDC_ISO_10646__ .
Also make the tests a bit more verbose, to aid debugging when they fail.

Change t_sprintf to an expected failure, since we don't respect the empty
thousands separator of the C/POSIX locale (PR standards/52282).

branches: 1.1.2;
Add test cases for sprintf/sscanf/strto{d,l} and the is* and isw* ctype functions, for single-byte encodings

Don't use fmtcheck for strfmon format strings. It does not work. Fix a broken
test.

PR/57633: Jose Luis Duran: Add strfmon tests from FreeBSD

Add testing for pad resetting (Jose Luis Duran)

s/diferent/different/

Update this test to expect the output that is supposed to be produced
by strfmon() rather than the output the old buggy implementation used
to produce.

branches: 1.1.2;
Add missing strfmon_l. Noticed by Bruno Haible. Add test case.

s/diferent/different/

branches: 1.1.4;
Add simple test case for toupper/tolower

branches: 1.3.2; 1.3.6;
tests/t_uchar: fix copy-and-paste typo

tests/lib/libc/locale/t_uchar: Test for char8_t, mbrtoc8, c8rtomb.

PR lib/58752: various header files test _*_SOURCE macros but don't
include sys/featuretest.h

PR lib/52374: <uchar.h> missing

uchar.h: New header file for C11 (and C++11) compliance.

Implementation of the new functions mbrtoc16, c16rtomb, mbrtoc32, and
c32rtomb to come later. Updates for C23 to come later.

PR lib/52374: <uchar.h> missing

branches: 1.1.2;
Add a simple collation test. This test is expected to fail on HEAD since
we do not yet have a working implementation of wcscoll.

branches: 1.1.4;
Add test cases for strcspn, strpbrk, strspn, wcscspn, wcspbrk and
wcsspn.

branches: 1.1.4;
Add test cases for strcspn, strpbrk, strspn, wcscspn, wcspbrk and
wcsspn.

branches: 1.1.6;
PR/54414: Valery Ushakov: add a test for wcsrtombs(3) doesn't update the
source argument on conversion error

branches: 1.1.4;
Add test cases for strcspn, strpbrk, strspn, wcscspn, wcspbrk and
wcsspn.

VAX doesn't have the test cases, so stub the body as well.

Add ISO10646 versions of these tests, conditional on __STDC_ISO_10646__ .
Also make the tests a bit more verbose, to aid debugging when they fail.

branches: 1.3.34;
Undo previous, Checking for vax is more appropriate.

no more ifdef vax

atf-ify the various locale tests

Add ISO10646 versions of these tests, conditional on __STDC_ISO_10646__ .
Also make the tests a bit more verbose, to aid debugging when they fail.

branches: 1.4.2;
Add a member to the test data structure that indicates whether the given
encoding is state-dependent, and test the results of wctomb(NULL, '\0') and
mbtowc(NULL, NULL, 0) against this instead of against each other.

Don't size an array using MB_CUR_MAX while one locale is in effect and
then use it with another locale having a larger MB_CUR_MAX. This
should fix the t_wctomb:wcrtomb_state test failures seen on i386.

branches: 1.2.2; 1.2.8;
Turn warns on for all tests and fix all the bugs.

branches: 1.1.2;
atf-ify the various locale tests

fix various typos in comment, documentation and log messages.

Add ISO10646 versions of these tests, conditional on __STDC_ISO_10646__ .
Also make the tests a bit more verbose, to aid debugging when they fail.

branches: 1.1.2;
Add test cases for sprintf/sscanf/strto{d,l} and the is* and isw* ctype functions, for single-byte encodings