ctype(3): New environment variable LIBC_ALLOWCTYPEABUSE.

If set, this does not force the ctype(3) functions to crash when
passed invalid inputs -- instead, they will return nonsense results,
and possibly print warnings to stderr, as is their right in
implementing undefined behaviour.

The nature of the nonsense results is unspecified. Currently, is*()
will always return true (even if that leads to mutually contradictory
conclusions, like isalpha and isdigit, or isgraph and isblank), and
tolower/toupper() will always return EOF. But perhaps in the future
the results may be randomized.

This way, if an application like firefox crashes on ctype abuse, you
can opt to accept the consequences of nonsense results instead by
running `env LIBC_ALLOWCTYPEABUSE= firefox' until the application is
fixed.

PR lib/58208: ctype(3) provides poor runtime feedback of abuse

branches: 1.68.2;
mbsnrtowcs(3), wcsnrtombs(3): Add man pages for these mouthfuls.

Based on patch drafted by abhinav@, with a little additional tidying
by me.

PR standards/52343: wcsnrtombs missing man-page

libc: New functions c8rtomb(3) and mbrtoc8(3).

New in C23, for converting from UTF-8 to locale-dependent multibyte
sequences (c8rtomb) or vice versa (mbrtoc8), along with the new type
char8_t.

Conditional on either:
- _NETBSD_SOURCE
- _ISOC23_SOURCE
- __STDC_VERSION__ >= 202311L

(Riding the libc minor bump from this morning for the UTF-16/UTF-32
versions from C11.)

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

branches: 1.65.6; 1.65.8;
Add missing man pages (from FreeBSD)

branches: 1.64.20; 1.64.28;
Rename fixup to update_global to better reflect the purpose.
Remove the various Current pointers, they are implementation internals
that shouldn't be leaked. Remove _CATEGORY_DEFAULT related handling and
directly pick the implemenation pointer from the C locale.
Merge the C locale data into global_locale.c with the exception of
_DefaultRuneLocale. Mark that one hidden. Move _RUNE_LOCALE and
_CITRUS_CTYPE into multibyte.h and pick up the correct rl_citrus_ctype
for non-default locales.

Provide explicit LC_C_LOCALE accessor and drop the various NULL checks.
Provide LC_GLOBAL_LOCALE in a way that works with all locale functions.
Merge constant data used by the initial global locale and the C locale.
Drop function call layer for _current_locale() and directly return the
locale reference, not a pointer to it. Use protected access for global
variables, so that libc references can avoid the GOT overhead.

Make that to process PATH_LOCALE the first time _find_category is
called. Use this to implement newlocale and provide duplocale/freelocale
as well. Based on patches by Takehiko Nozaki with simplications and fix
for the init order by myself.

Provide a const copy of global_locale for libc-internal use.
This will be used by *_l when a NULL pointer is given.

branches: 1.60.6;
Remove code for CITRUS!=yes.

branches: 1.59.2;
annotate that some functions use non-literal format strings.

1. refactoring new locale-db(RuneCT10) loading method with mmap(2).
2. remove unused field from _RuneLocale.
3. localeio(CITRUS=no) can read new locale-db(RuneCT10) now.

1. MB_LEN_MAX switch MD to MI.
2. unfortunately hppa's MB_LEN_MAX is defined incorrectly 6 instead of 32
so we have to add more setlocale(3) __RENAME func, __setlocale50.
3. move setlocale1.c and setlocale32.c to lib/libc/compat/locale/*
prepareing for next libc major crunk.
4. bump libc minor version.

refactoring old locale-db(BSDCTYPE) loading method with mmap(2).

1. fix build problem CITRUS=no
2. make CITRUS={yes,no} knob friendly with libc/compat.

merge christos-time_t

Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.52.4; 1.52.6;
Add support for additional locale categories: LC_MESSAGES, LC_MONETARY,
LC_NUMERIC.

The code used to load LC_TIME was refactored in to a more general routine.
This common routine is now used to load LC_TIME along with the newly added
categories.

Changes discussed with/reviewed by christos.

branches: 1.51.10; 1.51.12;
Add support for LC_TIME, from Joachim Kuebart, through PR lib/10877

PR/30809 added manpages of wcswcs(3) and wcscoll(3), wcsxfrm(3)
(wcs?width and wprintf are already in repository).

fix lib/33476:
added following manpages (derived from FreeBSD),
wcwidth(3) wcswidth(3)
wcstol(3) wcstoll(3) wcstoimax(3)
wcstoul(3) wcstoull(3) wcstoumax(3)
and update doc/TODO.i18n.

add c99 wcstof(3) and wcstold(3).
requested by skrll AT netbsd DOT org, thanks.

bump libc minor 138 -> 139.

Add man page for wcsftime from FreeBSD.

PR/29826: J. T. Conklin: Add wcsftime(3) (from FreeBSD)

branches: 1.45.6;
add wcstoll, wcstoimax, wcstoull and wcstoumax. (lib/20656)
- these are defined in C99 (wcsto[ui]max) and SUSv3.
- these implementations are written by sigsegv@s25.xrea.com,
derived from NetBSD's wcstol/wcstoul.
- I just added some comments.

changes related to lib/20576 and lib/20577.
- add wcstok() and wcswcs(),
- add wcstok(3),
- move btowc() and wctob() into each ctype modules,
- bump ctype interface ABI version,
- bump i18n modules' minor and
- update sets.
and fix a stupid bug in _RUNE_ISCACHED macro.

add manual pages for iswctype(), towctrans(), wctype() and wctrans().

- add btowc.3 / wctob.3.
- fix an unappropriate comment in wctob.c

- add wctob() function. (I forgot it at the last commit.)
- fix btowc(); I quite misunderstood about it.

add some ISO C 1995 I18N functions and types:
btowc, wctrans, towctrans, wcscoll, wcsxfrm, wctype_t and wctrans_t.

add manpages for mb*/wc* functions.

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

add a function to handle alias names.
(primarily for nls, encoding and locale names)

add wcstod/wcstol/wcstoul.

Remove UTF-2. UTF-2 is superceded by UTF-8.
see also:
http://mail-index.netbsd.org/tech-userlevel/2001/03/30/0000.html
http://mail-index.netbsd.org/tech-userlevel/2001/03/30/0001.html
http://mail-index.netbsd.org/tech-userlevel/2001/03/30/0002.html

branches: 1.34.2;
(finally!) enable multibyte LC_CTYPE locale support.

the functionality will be available for dynamic-linked binary only, due to
limitation with dlopen() from within statically-linked binaries.
(statically-linked binary can still enjoy singlebyte LC_CTYPE locale support)

from citrus project.

split setlocale.c for smaller footprint (do not pull things in too much).
have magic number check in locale modules.
rename: lib/libc/locale/setlocale_sb.c -> setlocale1.c

NOTE: this commit does not enable multibyte locale support, yet.

move __mb_cur_max to dedicated *.o, so that we do not pull in
locale stuff just by calling printf().

re-enable citrus (rune)locale support.

XXX please clean and recompile libc if you get this revision. UPDATE=yes
may leave obsolete *.o.

byte default, use (non-citrus) singlebyte locale. if we compile
ld.elf_so with citrus locale, it bombs.
XXX this is shortterm workaround. if you are willing to test citrus locale,
use RUNE=yes in libc/locale/Makefile.inc.

move wcs* and wmem* from lib/libc/locale to lib/libc/string.

add wcsl{cat,cpy} - outside of ISO/IEC 9899:1999,
but i believe it should be there.

ad manpage for wide character string manipulation (wmem*/wcs*)

manpage for tow*.

manpage for isw*.

add multibyte locale converters (just for use by testers at this moment,
to be built as dynamically loadable binary)

support multibytes in isw*/tow*.

first import of citrus LC_CTYPE locale. singlebyte only at this moment.
shot-term TODO:
- enable multibyte locales. this includes loadable locale modules
(#ifdef DLRUNE).
- populate LC_CTYPE definition files.
- other functions such as iconv()
- make libc functions more friendly with multibyte.
- multilingualize other LC_xx
- torture tests

XXX don't forget to "make depend" when you rebuild libc, multibyte.c is
moved from lib/libc/stdlib to lib/libc/locale.

add dummy wcwidth() and wcswidth(). wcwidth() will be updated later.

have dummy wctype.h and isw*/tow*. part of libc minor 68 -> 69 crank.
equivalent to is*/to* at this moment. we'll put wchar_t code in the
near future.

add wcs* functions. consier it as part of libc minor crank 68 -> 69.
add decl of WEOF into wchar.h. ISO/IEC 9899:1999(E)..

add wmem* functions, from citrus. bump libc shlib minor.

Preparation for the future introduction of multibyte locale.
- MB_LEN_MAX is increased to 32.
- To ensure binary compatibility for old executables
under multibyte locale, versioned setlocale is added.
- __mb_len_cur definision is added in setlocale.c
and enable it in stdlib.h .
It is also important for multibyte locale stuffs,
but I just forgot.

branches: 1.16.8; 1.16.10;
use ${ARCHDIR} instead of ${.CURDIR}/arch/${MACHINE_ARCH} where appropriate
libc builds on pmax again

Link localeconv(3) to setlocale(3).

add setlocale.3 man page

Add support for localized character sets (a.k.a. LC_CTYPE).

Thanks go to Matthias Scheler <tron@lyssa.owl.de> for contributing his initial
work in PR/3592, and to Christos Zoulas for refining it!

RCSid police.

specify man pages the new way.

Minor tweaks to prepare for checkin of real i18n code.

branches: 1.9.2;
Added a nl_langinfo(3) man page.

Add nl_langinfo(), a SysV-X/Open function that provides access to some
of the locale info.

Some of the POSIX.2 utilities require this information, and I thought it
was better to use an existing interface rather than rolling my own.

fix typo

New localeconv() implementation that constructs it's return value from
the current numeric and current monetary locale data.

Compile the default locale data for LC_NUMERIC, LC_MONETARY, and LC_TIME.

update for .S and MACHINE_ARCH

make an arch directory in libc, and move all machine dirs into it

Add RCS identifiers.

branches: 1.1.1;
Initial revision

Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.10.32; 1.10.34;
considering huge character set(ucs, gb18030),
use not linear but binary search.
this change derrived from recent FreeBSD's rune.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

- support for the additional code area of zh_CN.GB18030 locale,
- make sure that __nbrune_t is unsigned 32bit clean and
- fix wrong copyright notice in the last commit.
this patch is contributed by Takehiko NOZAKI <sigsegv at s25 dot xrea dot com>.

no point to compare wint_t and EOF.

add some ISO C 1995 I18N functions and types:
btowc, wctrans, towctrans, wcscoll, wcsxfrm, wctype_t and wctrans_t.

Fix signed/unsigned comparison warnings.

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

branches: 1.3.2;
avoid exposure of libc internal symbols into public header (wctype.h).

__RCSID

support multibytes in isw*/tow*.

add some ISO C 1995 I18N functions and types:
btowc, wctrans, towctrans, wcscoll, wcsxfrm, wctype_t and wctrans_t.

Fix signed/unsigned comparison warnings.

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

branches: 1.3.2;
avoid exposure of libc internal symbols into public header (wctype.h).

__RCSID

support multibytes in isw*/tow*.

add some ISO C 1995 I18N functions and types:
btowc, wctrans, towctrans, wcscoll, wcsxfrm, wctype_t and wctrans_t.

Fix signed/unsigned comparison warnings.

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

branches: 1.3.2;
avoid exposure of libc internal symbols into public header (wctype.h).

__RCSID

support multibytes in isw*/tow*.

Catch up with all the lint warnings since exit on warning was disabled.
Disable 'missing header declaration' and 'nested extern' warnings for now.

split setlocale.c for smaller footprint (do not pull things in too much).
have magic number check in locale modules.
rename: lib/libc/locale/setlocale_sb.c -> setlocale1.c

NOTE: this commit does not enable multibyte locale support, yet.

move __mb_cur_max to dedicated *.o, so that we do not pull in
locale stuff just by calling printf().

branches: 1.1.2;
add a header used by wcstoul/wcstol.
i forgot it at last commit.

Rename fixup to update_global to better reflect the purpose.
Remove the various Current pointers, they are implementation internals
that shouldn't be leaked. Remove _CATEGORY_DEFAULT related handling and
directly pick the implemenation pointer from the C locale.
Merge the C locale data into global_locale.c with the exception of
_DefaultRuneLocale. Mark that one hidden. Move _RUNE_LOCALE and
_CITRUS_CTYPE into multibyte.h and pick up the correct rl_citrus_ctype
for non-default locales.

branches: 1.6.44; 1.6.50;
Add missing __RCSID()

netbsd.org->NetBSD.org

fix typo; something called 'yesexpr' shouldn't have a 'n' in it

RCSid police.

Winning Strategies has placed this code into the Public Domain

Minor tweaks to prepare for checkin of real i18n code.

Rename fixup to update_global to better reflect the purpose.
Remove the various Current pointers, they are implementation internals
that shouldn't be leaked. Remove _CATEGORY_DEFAULT related handling and
directly pick the implemenation pointer from the C locale.
Merge the C locale data into global_locale.c with the exception of
_DefaultRuneLocale. Mark that one hidden. Move _RUNE_LOCALE and
_CITRUS_CTYPE into multibyte.h and pick up the correct rl_citrus_ctype
for non-default locales.

branches: 1.8.44; 1.8.50;
Add missing __RCSID()

netbsd.org->NetBSD.org

Add casts to please lint. Reviewed by christos.

C99: add new parameters int_p_cs_precedes, int_n_cs_precedes,
int_p_sep_by_space, int_n_sep_by_space, int_p_sign_posn and
int_n_sign_posn to monetary locale information.

RCSid police.

Winning Strategies has placed this code into the Public Domain

Minor tweaks to prepare for checkin of real i18n code.

Default Locale Data

Rename fixup to update_global to better reflect the purpose.
Remove the various Current pointers, they are implementation internals
that shouldn't be leaked. Remove _CATEGORY_DEFAULT related handling and
directly pick the implemenation pointer from the C locale.
Merge the C locale data into global_locale.c with the exception of
_DefaultRuneLocale. Mark that one hidden. Move _RUNE_LOCALE and
_CITRUS_CTYPE into multibyte.h and pick up the correct rl_citrus_ctype
for non-default locales.

branches: 1.6.44; 1.6.50;
Add missing __RCSID()

netbsd.org->NetBSD.org

RCSid police.

Winning Strategies has placed this code into the Public Domain

Minor tweaks to prepare for checkin of real i18n code.

Default Locale Data

Rename fixup to update_global to better reflect the purpose.
Remove the various Current pointers, they are implementation internals
that shouldn't be leaked. Remove _CATEGORY_DEFAULT related handling and
directly pick the implemenation pointer from the C locale.
Merge the C locale data into global_locale.c with the exception of
_DefaultRuneLocale. Mark that one hidden. Move _RUNE_LOCALE and
_CITRUS_CTYPE into multibyte.h and pick up the correct rl_citrus_ctype
for non-default locales.

Store time-specific locale data directly in the locale.
Ad dstrftime_l, strftime_lz and strptime_l.

branches: 1.10.2; 1.10.6; 1.10.26;
Add support for additional locale categories: LC_MESSAGES, LC_MONETARY,
LC_NUMERIC.

The code used to load LC_TIME was refactored in to a more general routine.
This common routine is now used to load LC_TIME along with the newly added
categories.

Changes discussed with/reviewed by christos.

branches: 1.9.10; 1.9.12;
Add support for LC_TIME, from Joachim Kuebart, through PR lib/10877

Add missing __RCSID()

netbsd.org->NetBSD.org

branches: 1.6.2;
XSH5/C99: in the C/POSIX locale, change the day of month format used in d_t_fmt
(locale's appropriate date and time representation) from %d (single digits
are preceded by 0) to %e (single digits are preceded by a blank).

branches: 1.5.4; 1.5.8;
RCSid police.

Winning Strategies has placed this code into the Public Domain

Minor tweaks to prepare for checkin of real i18n code.

Fixed major bogon: What happened to Wednesday!

Default Locale Data

branches: 1.4.22;
Provide explicit LC_C_LOCALE accessor and drop the various NULL checks.
Provide LC_GLOBAL_LOCALE in a way that works with all locale functions.
Merge constant data used by the initial global locale and the C locale.
Drop function call layer for _current_locale() and directly return the
locale reference, not a pointer to it. Use protected access for global
variables, so that libc references can avoid the GOT overhead.

Add wcstof_l, wcstod_l and wcstold_l.

branches: 1.2.6; 1.2.12;
Observe the following spelling:
- wide character (noun)
- wide-character (adjective)

Inspired by jmc@OpenBSD.

add c99 wcstof(3) and wcstold(3).
requested by skrll AT netbsd DOT org, thanks.

bump libc minor 138 -> 139.

Provide explicit LC_C_LOCALE accessor and drop the various NULL checks.
Provide LC_GLOBAL_LOCALE in a way that works with all locale functions.
Merge constant data used by the initial global locale and the C locale.
Drop function call layer for _current_locale() and directly return the
locale reference, not a pointer to it. Use protected access for global
variables, so that libc references can avoid the GOT overhead.

Add support for wcstoimax_l and friends.

branches: 1.4.2;
Update old-style definitions to ANSI, remove a couple of register
definitions along the way. Fixed gcc 4.1 build (thank you vax)

branches: 1.3.44;
WARNS=4

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

add wcstoll, wcstoimax, wcstoull and wcstoumax. (lib/20656)
- these are defined in C99 (wcsto[ui]max) and SUSv3.
- these implementations are written by sigsegv@s25.xrea.com,
derived from NetBSD's wcstol/wcstoul.
- I just added some comments.

Provide explicit LC_C_LOCALE accessor and drop the various NULL checks.
Provide LC_GLOBAL_LOCALE in a way that works with all locale functions.
Merge constant data used by the initial global locale and the C locale.
Drop function call layer for _current_locale() and directly return the
locale reference, not a pointer to it. Use protected access for global
variables, so that libc references can avoid the GOT overhead.

Add support for wcstoimax_l and friends.

branches: 1.4.2;
Update old-style definitions to ANSI, remove a couple of register
definitions along the way. Fixed gcc 4.1 build (thank you vax)

branches: 1.3.44;
WARNS=4

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

add wcstoll, wcstoimax, wcstoull and wcstoumax. (lib/20656)
- these are defined in C99 (wcsto[ui]max) and SUSv3.
- these implementations are written by sigsegv@s25.xrea.com,
derived from NetBSD's wcstol/wcstoul.
- I just added some comments.

branches: 1.17.40;
1. split runetype_local.h -> runetype_file.h
and remove renameing _Rune* -> _NBRune* namespace protection.

FreeBSD traditionaly exposes struct _Rune* in runetype.h
which included by ctype.h. it may cause conflicting type error
in our cross build process, former we use renaming namespace
to avoid this problem, now i reworked more resonable way.

2. merge rune_local.h to runetype_local.h, and remove it.
3. split bsdctype.h -> bsdctype_{file,local}.h

remove duplicated towctrans internal function.

more split ctype.h -> sys/ctype_inline.h, sys/ctype_bits.h

rune.h is not public, so merge with rune_local.h

PR lib/40411, nuke wctrans_init(). lazy initialization is not good idea.
statically initialized _DefaultRuneLocale.rl_wctrans field.
so we can re-const-ify _DefaultRuneLocale.

pkgsrc/shells/standalone-tcsh should be rebuild,
because it's staticaly linked binary.

try a different approach since allocating a new RuneLocale is complicated.
Don't make DefaultRuneLocale const, so that we can write to it, and undo
previous changes.

use a separate variable to copy the locale.

make this compile [now that I am on a machine that can compile :-)]

PR/40411: Hisashi T Fujinaka: standalone-tcsh dumps core when tab completing

The new rune code abuses __UNCONST to lazily initialize _RuneLocale. If
that happens to be the _DefaultRuneLocale which is const, the program will
core-dump because it will attempt to write to read-only memory. Catch this
and clone a copy of the locale and return it. The reason we don't see
everything core-dumping is because our shared library code probably loads
the text segment COW so it works (Is there an mprotect missing somewhere?).
But on statically linked binaries this is not the case and we die.

XXX[1]: Vet the code so that we are sure that there is no more of that
happening trying to get rid of much of the __UNCONST'ing.
XXX[2]: This needs to be fixed for 5.0 and all related fixes.
XXX[3]: There is one place in the code where _DefaultRuneLocale is treated
specially, does the patch break things?

merge christos-time_t

Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.6.32; 1.6.34;
considering huge character set(ucs, gb18030),
use not linear but binary search.
this change derrived from recent FreeBSD's rune.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

- support for the additional code area of zh_CN.GB18030 locale,
- make sure that __nbrune_t is unsigned 32bit clean and
- fix wrong copyright notice in the last commit.
this patch is contributed by Takehiko NOZAKI <sigsegv at s25 dot xrea dot com>.

changes for cross build on FreeBSD box.
- rename some macros and types defined in runetype.h.
- move declarations of some global symbols to rune.h from runetype.h.
- make sure the target sources use rune.h and mklocale use runetype.h.
I tested them on FreeBSD-4.6.2 box (./build.sh -m i386 tools).

no point to compare wint_t and EOF.

add some ISO C 1995 I18N functions and types:
btowc, wctrans, towctrans, wcscoll, wcsxfrm, wctype_t and wctrans_t.

branches: 1.9.40;
remove duplicated towctrans internal function.

sign-compare fixes

branches: 1.7.2;
PR lib/40411, nuke wctrans_init(). lazy initialization is not good idea.
statically initialized _DefaultRuneLocale.rl_wctrans field.
so we can re-const-ify _DefaultRuneLocale.

pkgsrc/shells/standalone-tcsh should be rebuild,
because it's staticaly linked binary.

try a different approach since allocating a new RuneLocale is complicated.
Don't make DefaultRuneLocale const, so that we can write to it, and undo
previous changes.

PR/40411: Hisashi T Fujinaka: standalone-tcsh dumps core when tab completing

The new rune code abuses __UNCONST to lazily initialize _RuneLocale. If
that happens to be the _DefaultRuneLocale which is const, the program will
core-dump because it will attempt to write to read-only memory. Catch this
and clone a copy of the locale and return it. The reason we don't see
everything core-dumping is because our shared library code probably loads
the text segment COW so it works (Is there an mprotect missing somewhere?).
But on statically linked binaries this is not the case and we die.

XXX[1]: Vet the code so that we are sure that there is no more of that
happening trying to get rid of much of the __UNCONST'ing.
XXX[2]: This needs to be fixed for 5.0 and all related fixes.
XXX[3]: There is one place in the code where _DefaultRuneLocale is treated
specially, does the patch break things?

merge christos-time_t

Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.2.38; 1.2.40;
- support for the additional code area of zh_CN.GB18030 locale,
- make sure that __nbrune_t is unsigned 32bit clean and
- fix wrong copyright notice in the last commit.
this patch is contributed by Takehiko NOZAKI <sigsegv at s25 dot xrea dot com>.

add some ISO C 1995 I18N functions and types:
btowc, wctrans, towctrans, wcscoll, wcsxfrm, wctype_t and wctrans_t.

branches: 1.9.40;
1. split runetype_local.h -> runetype_file.h
and remove renameing _Rune* -> _NBRune* namespace protection.

FreeBSD traditionaly exposes struct _Rune* in runetype.h
which included by ctype.h. it may cause conflicting type error
in our cross build process, former we use renaming namespace
to avoid this problem, now i reworked more resonable way.

2. merge rune_local.h to runetype_local.h, and remove it.
3. split bsdctype.h -> bsdctype_{file,local}.h

remove duplicated towctrans internal function.

more split ctype.h -> sys/ctype_inline.h, sys/ctype_bits.h

rune.h is not public, so merge with rune_local.h

PR lib/40411, nuke wctrans_init(). lazy initialization is not good idea.
statically initialized _DefaultRuneLocale.rl_wctrans field.
so we can re-const-ify _DefaultRuneLocale.

pkgsrc/shells/standalone-tcsh should be rebuild,
because it's staticaly linked binary.

try a different approach since allocating a new RuneLocale is complicated.
Don't make DefaultRuneLocale const, so that we can write to it, and undo
previous changes.

PR/40411: Hisashi T Fujinaka: standalone-tcsh dumps core when tab completing

The new rune code abuses __UNCONST to lazily initialize _RuneLocale. If
that happens to be the _DefaultRuneLocale which is const, the program will
core-dump because it will attempt to write to read-only memory. Catch this
and clone a copy of the locale and return it. The reason we don't see
everything core-dumping is because our shared library code probably loads
the text segment COW so it works (Is there an mprotect missing somewhere?).
But on statically linked binaries this is not the case and we die.

XXX[1]: Vet the code so that we are sure that there is no more of that
happening trying to get rid of much of the __UNCONST'ing.
XXX[2]: This needs to be fixed for 5.0 and all related fixes.
XXX[3]: There is one place in the code where _DefaultRuneLocale is treated
specially, does the patch break things?

branches: 1.2.2;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

remove duplicated towctrans internal function.

branches: 1.2.2;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

Remove code for CITRUS!=yes.

branches: 1.4.8;
merge christos-time_t

Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.2.32; 1.2.34;
A little libc namespace housekeeping exercise:
* Make vfprintf_unlocked() an internal function, c.f. __svfscanf_unlocked().
* Add internal names for arc4random(), endnetpath(), fhstatvfs(),
fstatvfs(), mkstemp(), shquote(), statvfs(), taddr2uaddr(), uaddr2taddr(),
uuid_create_nil(), uuid_is_nil(), and wcwidth().
* Include namespace.h where supposed to.

branches: 1.1.2;
add a function to handle alias names.
(primarily for nls, encoding and locale names)

merge christos-time_t

Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.1.2; 1.1.42; 1.1.44;
add a function to handle alias names.
(primarily for nls, encoding and locale names)

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

sync with latest Citrus XPG4DL.

- mbsrtowcs/wcsrtombs
- when conversion stopped with terminating-null, set *src null-pointer.
- if dst isn't null, ignore len.
- fix how to treat ___mbrtowc
- mbrtowc
- fix problem that terminating null charactor wasn't stored.
- fix problem with stateful encoding.
(when src points to escape sequence + '\0', mbstate wasn't initialized)
- fix return value when restarted.
XXX euctw and big5 are not tested.
- iso2022-jp
- fix mbrtowc with imcomplete sequence.
- etc
- make sure mbstate is initialized. (___rune_initstate is called)
primarily to fix mbsinit for iso2022.
- sync citrus Id

branches: 1.7.2;
Include string.h for mem*() prototypes.

split setlocale.c for smaller footprint (do not pull things in too much).
have magic number check in locale modules.
rename: lib/libc/locale/setlocale_sb.c -> setlocale1.c

NOTE: this commit does not enable multibyte locale support, yet.

sprinkle in _DIAGASSERT() as appropriate

put _RuneLocale pointer into mbstate_t for better thread safeness

switch locale module API, from s{get,put}rune to wcrtomb/mbrtowc.
handle intermediate mbstate_t more correctly.
no externally-visible changes here.

add multibyte locale converters (just for use by testers at this moment,
to be built as dynamically loadable binary)

branches: 1.1.2;
file big5.c was initially added on branch minoura-xpg4dl.

Remove code for CITRUS!=yes.

branches: 1.10.4;
Remove support for the old BSDCTYPE format.

fix cross build breakage mklocale(1).
move inline finction that uses sys/ctype_bits.h to runetype_misc.h.

1. refactoring new locale-db(RuneCT10) loading method with mmap(2).
2. remove unused field from _RuneLocale.
3. localeio(CITRUS=no) can read new locale-db(RuneCT10) now.

1. split runetype_local.h -> runetype_file.h
and remove renameing _Rune* -> _NBRune* namespace protection.

FreeBSD traditionaly exposes struct _Rune* in runetype.h
which included by ctype.h. it may cause conflicting type error
in our cross build process, former we use renaming namespace
to avoid this problem, now i reworked more resonable way.

2. merge rune_local.h to runetype_local.h, and remove it.
3. split bsdctype.h -> bsdctype_{file,local}.h

uint8_t -> unsigned char, int16_t -> short.

refactoring old locale-db(BSDCTYPE) loading method with mmap(2).

1. fix build problem CITRUS=no
2. make CITRUS={yes,no} knob friendly with libc/compat.

1. hide _CTYPE_PRIVATE section in ctype.h, move them to private header ctype_local.h.
2. do not use _CTYPE_NUM_CHARS macro to read data from LC_CTYPE(old BSDCTYPE style) database.
because 1<<CHAR_BIT is MD, so i added MI macro _CTYPE_CACHE_SIZE(1<<8).
3. remove _NB_CACHED_RUNE macro, use _CTYPE_CACHE_SIZE instead.

branches: 1.2.2;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

1. split runetype_local.h -> runetype_file.h
and remove renameing _Rune* -> _NBRune* namespace protection.

FreeBSD traditionaly exposes struct _Rune* in runetype.h
which included by ctype.h. it may cause conflicting type error
in our cross build process, former we use renaming namespace
to avoid this problem, now i reworked more resonable way.

2. merge rune_local.h to runetype_local.h, and remove it.
3. split bsdctype.h -> bsdctype_{file,local}.h

s/_CTYPE_NUM_CHARS/_CTYPE_CACHE_SIZE/

refactoring old locale-db(BSDCTYPE) loading method with mmap(2).

1. fix build problem CITRUS=no
2. make CITRUS={yes,no} knob friendly with libc/compat.

branches: 1.2.2;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

Remove code for CITRUS!=yes.

branches: 1.2.4;
Remove support for the old BSDCTYPE format.

1. split runetype_local.h -> runetype_file.h
and remove renameing _Rune* -> _NBRune* namespace protection.

FreeBSD traditionaly exposes struct _Rune* in runetype.h
which included by ctype.h. it may cause conflicting type error
in our cross build process, former we use renaming namespace
to avoid this problem, now i reworked more resonable way.

2. merge rune_local.h to runetype_local.h, and remove it.
3. split bsdctype.h -> bsdctype_{file,local}.h

Remove code for CITRUS!=yes.

branches: 1.2.6;
1. refactoring new locale-db(RuneCT10) loading method with mmap(2).
2. remove unused field from _RuneLocale.
3. localeio(CITRUS=no) can read new locale-db(RuneCT10) now.

1. split runetype_local.h -> runetype_file.h
and remove renameing _Rune* -> _NBRune* namespace protection.

FreeBSD traditionaly exposes struct _Rune* in runetype.h
which included by ctype.h. it may cause conflicting type error
in our cross build process, former we use renaming namespace
to avoid this problem, now i reworked more resonable way.

2. merge rune_local.h to runetype_local.h, and remove it.
3. split bsdctype.h -> bsdctype_{file,local}.h

Lots of grammar and wording changes from jmc@openbsd, with some
additional consistency changes by me.

Use
.In header.h
instead of
.Fd #include \*[Lt]header.h\*[Gt]
Much easier to read and write, and supported by groff for ages.
Okayed by ross.

Use Dv in some places, and sort SEE ALSO.

- add btowc.3 / wctob.3.
- fix an unappropriate comment in wctob.c

changes related to lib/20576 and lib/20577.
- add wcstok() and wcswcs(),
- add wcstok(3),
- move btowc() and wctob() into each ctype modules,
- bump ctype interface ABI version,
- bump i18n modules' minor and
- update sets.
and fix a stupid bug in _RUNE_ISCACHED macro.

add #include <string.h> for memset().

- add wctob() function. (I forgot it at the last commit.)
- fix btowc(); I quite misunderstood about it.

add some ISO C 1995 I18N functions and types:
btowc, wctrans, towctrans, wcscoll, wcsxfrm, wctype_t and wctrans_t.

branches: 1.11.2; 1.11.6;
c8rtomb(3), c16rtomb(3), c32rtomb(3): Fix null string output case.

This ignores c8/c16/c32, produces no output anywhere, and just resets
ps to the initial conversion state.

Also just use 0 in the example, not '\0' or L'\0'. This works for
C11, which prefers '\0' and L'\0', for and C23, which introduced the
new u8'\0', u'\0' (UTF-16), and U'\0' (UTF-32).

c8rtomb(3), c16rtomb(3), c32rtomb(3): Attempt to simplify language.

c8rtomb(3), c16rtomb(3), c32rtomb(3): Suggest MB_LEN_MAX in example.

This way it avoids variable-length arrays, by always allocating the
maximum space that could be occupied by MB_CUR_MAX.

c8rtomb(3), c16rtomb(3): Fix NUL handling.

PR lib/58615: incomplete c8rtomb, c16rtomb handles NUL termination
wrong

c16rtomb(3), c32rtomb(3), mbrtoc16(3), mbrtoc32(3): xref c8 versions.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

c16rtomb(3), c32rtomb(3): Attempt a deturgidification pass.

Limit the jargon around surrogates.

PR lib/52374: <uchar.h> missing

c16rtomb(3), c32rtomb(3): No state overlap with mbrtoc8 or c8rtomb.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

c16rtomb(3), c32rtomb(3): Specify what happens if ps is null.

PR lib/52374: <uchar.h> missing

c16rtomb(3), c32rtomb(3): brush up markup

c16rtomb(3), c32rtomb(3): Fix more \n in man pages.

Also, tighten an assertion: we left room for a NUL byte at the end.

PR lib/52374: <uchar.h> missing

libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

branches: 1.9.2; 1.9.6;
c16rtomb(3): Omit needless cast.

Now that lint has been taught to propagate the range in this kind of
expression (PR toolchain/58728), we can omit the cast -- the result is
guaranteed not to the truncated.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb
PR toolchain/58728: lint objects to __SHIFTIN/__BITS of values in
known-limited range without cast

c16rtomb, c8rtomb: Omit needless casts. Mark remaining lint issues.

These casts were introduced to pacify lint. lint has now been fixed
to handle __SHIFTOUT/__BITS more robustly so some of them are no
longer needed to pacify lint.

But lint still objects without some semantically unnecessary casts --
mark those.

- In c8rtomb, c32 is guaranteed to be at most 24-bit -- actually, at
most 21-bit -- but I'm not sure how to prove this to lint without a
transitive analysis of the state machine transitions and knowledge
of the only allowed initial state.

- In c16rtomb, the shiftouts are always 9-bit and so the shiftin
should never overflow, but lint hasn't figured this out yet.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb
PR toolchain/58728: __SHIFTIN/__BITS of values in known-limited range
without cast

c16rtomb, c32rtomb, mbrtoc8: fix page numbers in comments

c8rtomb(3), c16rtomb(3): Fix NUL handling.

PR lib/58615: incomplete c8rtomb, c16rtomb handles NUL termination
wrong

libc: Add _l variants of the cNrtomb and mbrtocN functions.

These accept an explicit locale parameter, rather than using the
current locale.

Visible under _NETBSD_SOURCE, not exposed otherwise.

NOTE: This adds libc symbols. Riding the libc minor bump for the
non-_l variants of these from two days ago -- hope that's not pushing
it too far.

PR lib/58613: c*rtomb, mbrtoc* should have locale-parametric _l
variants

pass lint, XXX see lint bug.

libc: c32rtomb and mbrtoc32 are used internally, so weak-alias them.

PR lib/52374: <uchar.h> missing

libc: Use the more idiomatic alignof from stdalign.h.

No functional change intended.

PR lib/52374: <uchar.h> missing

libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

branches: 1.11.2; 1.11.6;
c8rtomb(3), c16rtomb(3), c32rtomb(3): Fix null string output case.

This ignores c8/c16/c32, produces no output anywhere, and just resets
ps to the initial conversion state.

Also just use 0 in the example, not '\0' or L'\0'. This works for
C11, which prefers '\0' and L'\0', for and C23, which introduced the
new u8'\0', u'\0' (UTF-16), and U'\0' (UTF-32).

c8rtomb(3), c16rtomb(3), c32rtomb(3): Attempt to simplify language.

c8rtomb(3), c16rtomb(3), c32rtomb(3): Suggest MB_LEN_MAX in example.

This way it avoids variable-length arrays, by always allocating the
maximum space that could be occupied by MB_CUR_MAX.

c8rtomb(3), c16rtomb(3): Fix NUL handling.

PR lib/58615: incomplete c8rtomb, c16rtomb handles NUL termination
wrong

c16rtomb(3), c32rtomb(3), mbrtoc16(3), mbrtoc32(3): xref c8 versions.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

c16rtomb(3), c32rtomb(3): Attempt a deturgidification pass.

Limit the jargon around surrogates.

PR lib/52374: <uchar.h> missing

c16rtomb(3), c32rtomb(3): No state overlap with mbrtoc8 or c8rtomb.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

c16rtomb(3), c32rtomb(3): Specify what happens if ps is null.

PR lib/52374: <uchar.h> missing

c16rtomb(3), c32rtomb(3): brush up markup

c16rtomb(3), c32rtomb(3): Fix more \n in man pages.

Also, tighten an assertion: we left room for a NUL byte at the end.

PR lib/52374: <uchar.h> missing

libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

branches: 1.6.2; 1.6.6;
c16rtomb, c32rtomb, mbrtoc8: fix page numbers in comments

c32rtomb(3): Fix type of wcrtomb_l return value.

This was from `int wctomb_l(...)' in an earlier draft and I didn't
update it to size_t when I changed the draft to wcrtomb_l. Caught by
lint.

`wc_len' mirrors `mb_len' in the complementary code in mbrtoc32(3) to
avoid clash with standard C function mblen(3).

PR lib/58612: c8rtomb/c16rtomb/c32rtomb yield suboptimal shift
sequences

c32rtomb(3): Use conversion state to handle shift sequences.

For conversion of Unicode scalar values to coding systems requiring
shift sequences, such as ISO-2022-JP, _citrus_iconv_convert will
always produce:

1. a shift sequence from the initial state to some nondefault state,
like from US-ASCII to JIS X 0208
2. the encoding of the desired characater
3. a shift sequence restoring the initial state

This is unnecessary if the output is already in the state needed to
encoded the desired character. For example, this method produces
seven bytes to encode each YEN SIGN in ISO-2022-JP -- and fourteen,
to encode two consecutive ones -- even though the shift sequence is
only three bytes long and once shifted YEN SIGN takes only one byte.

Instead, convert the Unicode scalar value to a locale-dependent wide
character and encode that, by composing

- _citrus_iconv_convert
=> gives us a multibyte encoding of the character from the initial
state (and restoring the initial state afterward)
- mbrtowc with initial conversion state
=> gives us the single wide character representation
XXX If combining characters are possible here, this may fail.
- wcrtomb with caller's conversion tsate
=> gives us a state-dependent multibyte encoding of the character

XXX Is there a cheaper way to convert from Unicode scalar value to
locale-dependent wide character? It is not obvious to me from the
largely undocumented Citrus machinery, but it would obviously be
better than this somewhat circuitous Rube Goldberg contraption of
chained multibyte APIs.

PR lib/58612: c8rtomb/c16rtomb/c32rtomb yield suboptimal shift
sequences

libc: Add _l variants of the cNrtomb and mbrtocN functions.

These accept an explicit locale parameter, rather than using the
current locale.

Visible under _NETBSD_SOURCE, not exposed otherwise.

NOTE: This adds libc symbols. Riding the libc minor bump for the
non-_l variants of these from two days ago -- hope that's not pushing
it too far.

PR lib/58613: c*rtomb, mbrtoc* should have locale-parametric _l
variants

libc: c32rtomb and mbrtoc32 are used internally, so weak-alias them.

PR lib/52374: <uchar.h> missing

libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

branches: 1.2.2; 1.2.6;
c32rtomb(3): Use conversion state to handle shift sequences.

For conversion of Unicode scalar values to coding systems requiring
shift sequences, such as ISO-2022-JP, _citrus_iconv_convert will
always produce:

1. a shift sequence from the initial state to some nondefault state,
like from US-ASCII to JIS X 0208
2. the encoding of the desired characater
3. a shift sequence restoring the initial state

This is unnecessary if the output is already in the state needed to
encoded the desired character. For example, this method produces
seven bytes to encode each YEN SIGN in ISO-2022-JP -- and fourteen,
to encode two consecutive ones -- even though the shift sequence is
only three bytes long and once shifted YEN SIGN takes only one byte.

Instead, convert the Unicode scalar value to a locale-dependent wide
character and encode that, by composing

- _citrus_iconv_convert
=> gives us a multibyte encoding of the character from the initial
state (and restoring the initial state afterward)
- mbrtowc with initial conversion state
=> gives us the single wide character representation
XXX If combining characters are possible here, this may fail.
- wcrtomb with caller's conversion tsate
=> gives us a state-dependent multibyte encoding of the character

XXX Is there a cheaper way to convert from Unicode scalar value to
locale-dependent wide character? It is not obvious to me from the
largely undocumented Citrus machinery, but it would obviously be
better than this somewhat circuitous Rube Goldberg contraption of
chained multibyte APIs.

PR lib/58612: c8rtomb/c16rtomb/c32rtomb yield suboptimal shift
sequences

libc: New C11 functions mbrtoc16, mbrtoc32, c16rtomb, c32rtomb.

The mbrtoc16/32 functions read mulitbyte strings according to the
current locale into UTF-16/32 code unit sequences; the c16/32rtomb
functions write UTF-16/32 code unit sequences into multibyte strings
according to the current locale. The `r' means restartable: they
work incrementally and pick up where they left off.

NOTE: This bumps the libc minor version, since it adds new symbols.

PR lib/52374: <uchar.h> missing

branches: 1.9.2; 1.9.6;
c8rtomb(3), c16rtomb(3), c32rtomb(3): Fix null string output case.

This ignores c8/c16/c32, produces no output anywhere, and just resets
ps to the initial conversion state.

Also just use 0 in the example, not '\0' or L'\0'. This works for
C11, which prefers '\0' and L'\0', for and C23, which introduced the
new u8'\0', u'\0' (UTF-16), and U'\0' (UTF-32).

c8rtomb(3), c16rtomb(3), c32rtomb(3): Attempt to simplify language.

c8rtomb(3), c16rtomb(3), c32rtomb(3): Suggest MB_LEN_MAX in example.

This way it avoids variable-length arrays, by always allocating the
maximum space that could be occupied by MB_CUR_MAX.

c8rtomb(3), c16rtomb(3): Fix NUL handling.

PR lib/58615: incomplete c8rtomb, c16rtomb handles NUL termination
wrong

c8rtomb(3): Clarify prose and fix example in caveat.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

c8rtomb(3): Fix possible error descriptions.

The argument c8 can't be a surrogate code point itself (they're in
the range [0xd800,0xdfff], beyond 8-bit values), but the bits of a
surrogate code point could be forced into the UTF-8 format, which is
also invalid.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

c8rtomb(3): Specify what happens when ps is null.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

c8rtomb(3), mbrtoc8(3): brush up markup

libc: New functions c8rtomb(3) and mbrtoc8(3).

New in C23, for converting from UTF-8 to locale-dependent multibyte
sequences (c8rtomb) or vice versa (mbrtoc8), along with the new type
char8_t.

Conditional on either:
- _NETBSD_SOURCE
- _ISOC23_SOURCE
- __STDC_VERSION__ >= 202311L

(Riding the libc minor bump from this morning for the UTF-16/UTF-32
versions from C11.)

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

branches: 1.9.2; 1.9.6;
c8rtomb: use uint8_t for 8-bit values

On amd64 at least, uint_fast8_t is an alias to unsigned int, which
produced a lint warning that 'unsigned int << 24' could lose some bits,
when in fact the expression was 'unsigned char << 24'.

Fixes the last remaining part of PR toolchain/58728.

c8rtomb: declare array of constants as const

No binary change using GCC, as it had detected that all access to these
arrays was read-only and thus placed them in the read-only section.

c16rtomb, c8rtomb: Omit needless casts. Mark remaining lint issues.

These casts were introduced to pacify lint. lint has now been fixed
to handle __SHIFTOUT/__BITS more robustly so some of them are no
longer needed to pacify lint.

But lint still objects without some semantically unnecessary casts --
mark those.

- In c8rtomb, c32 is guaranteed to be at most 24-bit -- actually, at
most 21-bit -- but I'm not sure how to prove this to lint without a
transitive analysis of the state machine transitions and knowledge
of the only allowed initial state.

- In c16rtomb, the shiftouts are always 9-bit and so the shiftin
should never overflow, but lint hasn't figured this out yet.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb
PR toolchain/58728: __SHIFTIN/__BITS of values in known-limited range
without cast

c8rtomb(3): Nix __CTASSERT after case label.

I put this in to make it (machine-verifiably) clear that zeroing the
state is the same as returning to the initial conversion state, as
the standard requires, but this is causing build trouble (and will
likely cause more trouble if pulled up) because some definitions of
__CTASSERT make a declaration which is forbidden after a label, so
let's remove it.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

c8rtomb(3), c16rtomb(3): Fix NUL handling.

PR lib/58615: incomplete c8rtomb, c16rtomb handles NUL termination
wrong

libc: Add _l variants of the cNrtomb and mbrtocN functions.

These accept an explicit locale parameter, rather than using the
current locale.

Visible under _NETBSD_SOURCE, not exposed otherwise.

NOTE: This adds libc symbols. Riding the libc minor bump for the
non-_l variants of these from two days ago -- hope that's not pushing
it too far.

PR lib/58613: c*rtomb, mbrtoc* should have locale-parametric _l
variants

pass lint, XXX see lint bug.

c8rtomb(3), mbrtoc8(3): Use namespace.h to get private aliases.

This way applications defining the symbols c32rtomb or mbrtoc32 won't
clobber our private definitions, which are slightly more constrained
about their use of mbstate_t than is obvious from the interface
contract.

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

libc: New functions c8rtomb(3) and mbrtoc8(3).

New in C23, for converting from UTF-8 to locale-dependent multibyte
sequences (c8rtomb) or vice versa (mbrtoc8), along with the new type
char8_t.

Conditional on either:
- _NETBSD_SOURCE
- _ISOC23_SOURCE
- __STDC_VERSION__ >= 202311L

(Riding the libc minor bump from this morning for the UTF-16/UTF-32
versions from C11.)

PR standards/58601: uchar.h C23 compliance: char8_t, mbrtoc8, c8rtomb

Provide explicit LC_C_LOCALE accessor and drop the various NULL checks.
Provide LC_GLOBAL_LOCALE in a way that works with all locale functions.
Merge constant data used by the initial global locale and the C locale.
Drop function call layer for _current_locale() and directly return the
locale reference, not a pointer to it. Use protected access for global
variables, so that libc references can avoid the GOT overhead.

Store time-specific locale data directly in the locale.
Ad dstrftime_l, strftime_lz and strptime_l.

Provide a const copy of global_locale for libc-internal use.
This will be used by *_l when a NULL pointer is given.

branches: 1.1.2;
file collate.c was initially added on branch minoura-xpg4dl.

branches: 1.1.2; 1.1.40;
file collate.h was initially added on branch minoura-xpg4dl.

branches: 1.1.2;
file collate_local.h was initially added on branch perseant-stdc-iso10646.

branches: 1.1.2;
file collate_locale.c was initially added on branch perseant-stdc-iso10646.

branches: 1.1.2;
file collcmp.c was initially added on branch minoura-xpg4dl.

Extend ctype classification table to 16bit. Based on patch by
Takehiko Nozaki, with changes to compile fail when using the old names
and to exploit __BUILD_LEGACY

branches: 1.4.6; 1.4.12;
1. split runetype_local.h -> runetype_file.h
and remove renameing _Rune* -> _NBRune* namespace protection.

FreeBSD traditionaly exposes struct _Rune* in runetype.h
which included by ctype.h. it may cause conflicting type error
in our cross build process, former we use renaming namespace
to avoid this problem, now i reworked more resonable way.

2. merge rune_local.h to runetype_local.h, and remove it.
3. split bsdctype.h -> bsdctype_{file,local}.h

remove redundant guard.

rework bulid breakage problem on linux.

ldef.h: s/1<<8/_CTYPE_CACHE_SIZE/
ctype_local.h: use limits.h instead of machine/limits.h
runetype_local.h: don't use HAVE_NBTOOL_CONFIG as evil way.

1. hide _CTYPE_PRIVATE section in ctype.h, move them to private header ctype_local.h.
2. do not use _CTYPE_NUM_CHARS macro to read data from LC_CTYPE(old BSDCTYPE style) database.
because 1<<CHAR_BIT is MD, so i added MI macro _CTYPE_CACHE_SIZE(1<<8).
3. remove _NB_CACHED_RUNE macro, use _CTYPE_CACHE_SIZE instead.

refactoring old locale-db(BSDCTYPE) loading method with mmap(2).

more split ctype.h -> sys/ctype_inline.h, sys/ctype_bits.h

1. hide _CTYPE_PRIVATE section in ctype.h, move them to private header ctype_local.h.
2. do not use _CTYPE_NUM_CHARS macro to read data from LC_CTYPE(old BSDCTYPE style) database.
because 1<<CHAR_BIT is MD, so i added MI macro _CTYPE_CACHE_SIZE(1<<8).
3. remove _NB_CACHED_RUNE macro, use _CTYPE_CACHE_SIZE instead.

Close file handles in error cases. Found by cppcheck.

Remove 3rd and 4th clauses in christos' license. OK christos.

merge christos-time_t

Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.7.26; 1.7.28;
WARNS=4

Add missing __RCSID()

include <string.h> for mem*

branches: 1.4.8;
back out the #ifdef _DIAGNOSTIC argument checks; too many people complained.
_DIAGASSERT() is still retained.

* use _DIAGASSERT() to check pointer arguments against NULL and file
descriptors against -1 (as appropriate).
* add actual checks which to detect stuff that would trigger_DIAGASSERT(),
and attempt to return a sane error condition.
* knf some code
* remove some `register' decls.

the first two items result in the addition of code similar to the
following in various functions:

_DIAGASSERT(path != NULL)
#ifdef _DIAGNOSTIC
if (path == NULL) {
errno = EFAULT;
return (-1);
}
#endif

delint

Add support for localized character sets (a.k.a. LC_CTYPE).

Thanks go to Matthias Scheler <tron@lyssa.owl.de> for contributing his initial
work in PR/3592, and to Christos Zoulas for refining it!

refactoring old locale-db(BSDCTYPE) loading method with mmap(2).

Remove 3rd and 4th clauses in christos' license. OK christos.

merge christos-time_t

Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.1.54; 1.1.56;
Add support for localized character sets (a.k.a. LC_CTYPE).

Thanks go to Matthias Scheler <tron@lyssa.owl.de> for contributing his initial
work in PR/3592, and to Christos Zoulas for refining it!

Provide explicit LC_C_LOCALE accessor and drop the various NULL checks.
Provide LC_GLOBAL_LOCALE in a way that works with all locale functions.
Merge constant data used by the initial global locale and the C locale.
Drop function call layer for _current_locale() and directly return the
locale reference, not a pointer to it. Use protected access for global
variables, so that libc references can avoid the GOT overhead.

Rename struct _locale_impl_t to struct _locale, since it will end up as
locale_t later.

branches: 1.3.2;
Use C89 Prototypes.
Remove use of __P

branches: 1.2.2; 1.2.10;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.1.2;
file ducet_collation_data.h was initially added on branch perseant-stdc-iso10646.

don't use __link_set_* stuff for locale implementation.
patch reviewed by matt@, thanks.

branches: 1.2.2; 1.2.10;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

Rename struct _locale_impl_t to struct _locale, since it will end up as
locale_t later.

branches: 1.3.2;
don't use __link_set_* stuff for locale implementation.
patch reviewed by matt@, thanks.

branches: 1.2.2; 1.2.10;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

duplocale.3: fix typo

branches: 1.3.8;
Remove reference to multibyte(3) and trailing commas.

Remove FreeBSD portions that are not relevant to our implementation (pointed
out by joerg@)

Add missing man pages (from FreeBSD)

branches: 1.1.2; 1.1.8;
Make that to process PATH_LOCALE the first time _find_category is
called. Use this to implement newlocale and provide duplocale/freelocale
as well. Based on patches by Takehiko Nozaki with simplications and fix
for the init order by myself.

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

sync with latest Citrus XPG4DL.

- mbsrtowcs/wcsrtombs
- when conversion stopped with terminating-null, set *src null-pointer.
- if dst isn't null, ignore len.
- fix how to treat ___mbrtowc
- mbrtowc
- fix problem that terminating null charactor wasn't stored.
- fix problem with stateful encoding.
(when src points to escape sequence + '\0', mbstate wasn't initialized)
- fix return value when restarted.
XXX euctw and big5 are not tested.
- iso2022-jp
- fix mbrtowc with imcomplete sequence.
- etc
- make sure mbstate is initialized. (___rune_initstate is called)
primarily to fix mbsinit for iso2022.
- sync citrus Id

fix a bit bug. this bug normally causes no problem.

branches: 1.7.2;
split setlocale.c for smaller footprint (do not pull things in too much).
have magic number check in locale modules.
rename: lib/libc/locale/setlocale_sb.c -> setlocale1.c

NOTE: this commit does not enable multibyte locale support, yet.

sprinkle in _DIAGASSERT() as appropriate

put _RuneLocale pointer into mbstate_t for better thread safeness

switch locale module API, from s{get,put}rune to wcrtomb/mbrtowc.
handle intermediate mbstate_t more correctly.
no externally-visible changes here.

avoid duplicated free() on initialization failure

add multibyte locale converters (just for use by testers at this moment,
to be built as dynamically loadable binary)

branches: 1.1.2;
file euc.c was initially added on branch minoura-xpg4dl.

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

sync with latest Citrus XPG4DL.

- mbsrtowcs/wcsrtombs
- when conversion stopped with terminating-null, set *src null-pointer.
- if dst isn't null, ignore len.
- fix how to treat ___mbrtowc
- mbrtowc
- fix problem that terminating null charactor wasn't stored.
- fix problem with stateful encoding.
(when src points to escape sequence + '\0', mbstate wasn't initialized)
- fix return value when restarted.
XXX euctw and big5 are not tested.
- iso2022-jp
- fix mbrtowc with imcomplete sequence.
- etc
- make sure mbstate is initialized. (___rune_initstate is called)
primarily to fix mbsinit for iso2022.
- sync citrus Id

branches: 1.9.2;
Include string.h for mem*() prototypes.

split setlocale.c for smaller footprint (do not pull things in too much).
have magic number check in locale modules.
rename: lib/libc/locale/setlocale_sb.c -> setlocale1.c

NOTE: this commit does not enable multibyte locale support, yet.

sprinkle in _DIAGASSERT() as appropriate

put _RuneLocale pointer into mbstate_t for better thread safeness

sync citrus Id part with citrus repository.

switch locale module API, from s{get,put}rune to wcrtomb/mbrtowc.
handle intermediate mbstate_t more correctly.
no externally-visible changes here.

indentation

add multibyte locale converters (just for use by testers at this moment,
to be built as dynamically loadable binary)

branches: 1.1.2;
file euctw.c was initially added on branch minoura-xpg4dl.

libc/locale: Nix needless char casts.

These appear to have been added to pacify a spurious warning by lint,
which no longer warns about it anyway.

No functional change intended. No binary change to aarch64 libc.

Prompted by code inspection around PR lib/58319.

cast CHAR_MAX to (char) for unsigned character machines

branches: 1.5.6;
more split ctype.h -> sys/ctype_inline.h, sys/ctype_bits.h

branches: 1.4.2;
merge christos-time_t

1. reworking PR lib/40317:
libnbcompat already contains empty fparseln.lo
so previous fix doesn't work correctly.
i've just added broken fparseln check to configure script.

2. reworking cross build breakage under FreeBSD/MacOS X.
FreeBSD/MacOS X still have public /usr/include/runetype.h
derived from 4.4BSD-Lite. so i renamed out private header from
src/lib/libc/locale/runetype.h to src/lib/libc/locale/runetype_local.h
to solve this problems.

3. fix build breakage when CITRUS=no was set.

define __UNCONST for OSes that don't have it when building tools/mklocale

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.2.2;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

Remove reference to multibyte(3) and trailing commas.

Remove FreeBSD portions that are not relevant to our implementation (pointed
out by joerg@)

Add missing man pages (from FreeBSD)

branches: 1.2.2; 1.2.8;
Provide explicit LC_C_LOCALE accessor and drop the various NULL checks.
Provide LC_GLOBAL_LOCALE in a way that works with all locale functions.
Merge constant data used by the initial global locale and the C locale.
Drop function call layer for _current_locale() and directly return the
locale reference, not a pointer to it. Use protected access for global
variables, so that libc references can avoid the GOT overhead.

Make that to process PATH_LOCALE the first time _find_category is
called. Use this to implement newlocale and provide duplocale/freelocale
as well. Based on patches by Takehiko Nozaki with simplications and fix
for the init order by myself.

Catch up with all the lint warnings since exit on warning was disabled.
Disable 'missing header declaration' and 'nested extern' warnings for now.

Add bunch of missing includes of namespace.h in libc

The NetBSD Standard C Library uses internally some of its functions with
a mangled symbol name, usually "_symbol". The internal functions shall not
use the global (public) symbols.

This change eliminates usage of the global changes of the following symbols:
- strlcat -> _strlcat
- sysconf -> __sysconf
- closedir -> _closedir
- fparseln -> _fparseln
- kill -> _kill
- mkstemp -> _mkstemp
- reallocarr -> _reallocarr
- strcasecmp -> _strcasecmp
- strncasecmp -> _strncasecmp
- strptime -> _strptime
- strtok_r -> _strtok_r
- sysctl -> _sysctl
- dlopen -> __dlopen
- dlclose -> __dlclose
- dlsym -> __dlsym

Sponsored by <The NetBSD Foundation>

Rename struct _locale_impl_t to struct _locale, since it will end up as
locale_t later.

branches: 1.4.2;
don't use __link_set_* stuff for locale implementation.
patch reviewed by matt@, thanks.

branches: 1.3.6;
fix lib/42124: setlocale(3) never returns NULL with none existing locale.
reported by kambe-san, thanks!

branches: 1.2.2;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

don't use __link_set_* stuff for locale implementation.
patch reviewed by matt@, thanks.

branches: 1.3.6;
fix static linking setlocale(3) problem, reported by Aran Clauson
at current-users, thanks a lot!

branches: 1.2.2; 1.2.4;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

Rename struct _locale_impl_t to struct _locale, since it will end up as
locale_t later.

branches: 1.3.2;
don't use __link_set_* stuff for locale implementation.
patch reviewed by matt@, thanks.

branches: 1.2.2; 1.2.10;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

Redo l10n support in the strerror family.

Instead of opening the message catelog whenever strerror is called,
keep track of the translations in the locale cache. For the C locale,
the builtin sys_errlist is used directly. Other locales will open
the catalog file on the first strerror call and build a translation
table, so that further calls in this locale can just use an array
lookup.

libc/locale: Nix needless size_t casts.

No functional change intended. No binary change to aarch64 libc.

Prompted by code inspection around PR lib/58319.

libc/locale: Nix needless char casts.

These appear to have been added to pacify a spurious warning by lint,
which no longer warns about it anyway.

No functional change intended. No binary change to aarch64 libc.

Prompted by code inspection around PR lib/58319.

libc/locale/global_locale.c: Nix confusing NBCHAR_MAX.

NBCHAR_MAX is used only in the usr.bin/mklocale build, but that
doesn't use global_locale.c. This was added when inserting char
casts, but using the mklocale's hook just to insert a char cast is
unnecessarily confusing -- and the char casts are unnecessary anyway.

No functional change intended. No binary change to aarch64 libc.

Raised in PR lib/58319.

branches: 1.25.10; 1.25.26;
Go back to just using normal visibility for the locale symbols. Without
an actual specifier like dllimport, protected visibility is unusable.

undo the dso protected for clang

Undo the __dso_protected hackery.

disable dso protected to work around binutils bug.

Redo the locale cache to be constant. It now contains the localeconv()
data and which LC_MONETARY and LC_NUMERIC values it is derived from.
In newlocale(3) and setlocale(3), check for the existing entries and on
miss, create a new entry. This is currently not using a lock for the
list as the worst case is a small memory leak.

Extract the data for nl_langinfo directly from the corresponding
part implementation using a category/offset table.

Remove most LC_CTYPE specific parts of locale.cache.

Rename fixup to update_global to better reflect the purpose.
Remove the various Current pointers, they are implementation internals
that shouldn't be leaked. Remove _CATEGORY_DEFAULT related handling and
directly pick the implemenation pointer from the C locale.
Merge the C locale data into global_locale.c with the exception of
_DefaultRuneLocale. Mark that one hidden. Move _RUNE_LOCALE and
_CITRUS_CTYPE into multibyte.h and pick up the correct rl_citrus_ctype
for non-default locales.

Provide explicit LC_C_LOCALE accessor and drop the various NULL checks.
Provide LC_GLOBAL_LOCALE in a way that works with all locale functions.
Merge constant data used by the initial global locale and the C locale.
Drop function call layer for _current_locale() and directly return the
locale reference, not a pointer to it. Use protected access for global
variables, so that libc references can avoid the GOT overhead.

Store time-specific locale data directly in the locale.
Ad dstrftime_l, strftime_lz and strptime_l.

Rename struct _locale_impl_t to struct _locale, since it will end up as
locale_t later.

Extend ctype classification table to 16bit. Based on patch by
Takehiko Nozaki, with changes to compile fail when using the old names
and to exploit __BUILD_LEGACY

branches: 1.13.2;
cast CHAR_MAX to (char) for unsigned character machines

branches: 1.12.2;
Remove code for CITRUS!=yes.

branches: 1.11.6;
1. refactoring new locale-db(RuneCT10) loading method with mmap(2).
2. remove unused field from _RuneLocale.
3. localeio(CITRUS=no) can read new locale-db(RuneCT10) now.

1. split runetype_local.h -> runetype_file.h
and remove renameing _Rune* -> _NBRune* namespace protection.

FreeBSD traditionaly exposes struct _Rune* in runetype.h
which included by ctype.h. it may cause conflicting type error
in our cross build process, former we use renaming namespace
to avoid this problem, now i reworked more resonable way.

2. merge rune_local.h to runetype_local.h, and remove it.
3. split bsdctype.h -> bsdctype_{file,local}.h

more split ctype.h -> sys/ctype_inline.h, sys/ctype_bits.h

make _locale_impl_t.cache as pointer to maintain easily binary compatibility in future.
no ABI change was made.

rune.h is not public, so merge with rune_local.h

1. hide _CTYPE_PRIVATE section in ctype.h, move them to private header ctype_local.h.
2. do not use _CTYPE_NUM_CHARS macro to read data from LC_CTYPE(old BSDCTYPE style) database.
because 1<<CHAR_BIT is MD, so i added MI macro _CTYPE_CACHE_SIZE(1<<8).
3. remove _NB_CACHED_RUNE macro, use _CTYPE_CACHE_SIZE instead.

fix tpo.

_locale_cache_t: change ldata and items to pointer for future binary compatibility.

PR lib/40411, nuke wctrans_init(). lazy initialization is not good idea.
statically initialized _DefaultRuneLocale.rl_wctrans field.
so we can re-const-ify _DefaultRuneLocale.

pkgsrc/shells/standalone-tcsh should be rebuild,
because it's staticaly linked binary.

branches: 1.2.2;
merge christos-time_t

branches: 1.1.2;
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

sync with latest Citrus XPG4DL.

- mbsrtowcs/wcsrtombs
- when conversion stopped with terminating-null, set *src null-pointer.
- if dst isn't null, ignore len.
- fix how to treat ___mbrtowc
- mbrtowc
- fix problem that terminating null charactor wasn't stored.
- fix problem with stateful encoding.
(when src points to escape sequence + '\0', mbstate wasn't initialized)
- fix return value when restarted.
XXX euctw and big5 are not tested.
- iso2022-jp
- fix mbrtowc with imcomplete sequence.
- etc
- make sure mbstate is initialized. (___rune_initstate is called)
primarily to fix mbsinit for iso2022.
- sync citrus Id

branches: 1.11.2;
make tags[] and flags[] const

split setlocale.c for smaller footprint (do not pull things in too much).
have magic number check in locale modules.
rename: lib/libc/locale/setlocale_sb.c -> setlocale1.c

NOTE: this commit does not enable multibyte locale support, yet.

sprinkle in _DIAGASSERT() as appropriate

put _RuneLocale pointer into mbstate_t for better thread safeness

sync citrus Id part with citrus repository.

switch locale module API, from s{get,put}rune to wcrtomb/mbrtowc.
handle intermediate mbstate_t more correctly.
no externally-visible changes here.

arg mistake to memset

overhaul.
(1) alloate two mbstate_t, for input/output, per a file stream.
(2) correct inbound escape sequence processing.
cope with incomplete string better. keep intermediate state for single shifts.
let C0/C1 go through.

XXX for (1), we should expand FILE structure, but is possible only with
libc major # bump.

avoid duplicated free() on initialization failure

add multibyte locale converters (just for use by testers at this moment,
to be built as dynamically loadable binary)

branches: 1.1.2;
file iso2022.c was initially added on branch minoura-xpg4dl.

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

sync with latest Citrus XPG4DL.

- mbsrtowcs/wcsrtombs
- when conversion stopped with terminating-null, set *src null-pointer.
- if dst isn't null, ignore len.
- fix how to treat ___mbrtowc
- mbrtowc
- fix problem that terminating null charactor wasn't stored.
- fix problem with stateful encoding.
(when src points to escape sequence + '\0', mbstate wasn't initialized)
- fix return value when restarted.
XXX euctw and big5 are not tested.
- iso2022-jp
- fix mbrtowc with imcomplete sequence.
- etc
- make sure mbstate is initialized. (___rune_initstate is called)
primarily to fix mbsinit for iso2022.
- sync citrus Id

branches: 1.4.2;
sync citrus Id part with citrus repository.

switch locale module API, from s{get,put}rune to wcrtomb/mbrtowc.
handle intermediate mbstate_t more correctly.
no externally-visible changes here.

add multibyte locale converters (just for use by testers at this moment,
to be built as dynamically loadable binary)

branches: 1.1.2;
file iso2022.h was initially added on branch minoura-xpg4dl.

Lots of grammar and wording changes from jmc@openbsd, with some
additional consistency changes by me.

Use St -isoC-9X instead of St -isoC9X, to be in sync with groff.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

Use
.In header.h
instead of
.Fd #include \*[Lt]header.h\*[Gt]
Much easier to read and write, and supported by groff for ages.
Okayed by ross.

add CAVEATS.

Use .St -isoC99 to refer to ISO C99.

Generate <>& symbolically.

branches: 1.2.2;
place return value

manpage for isw*.

s/wcttype/wctype/

Lots of grammar and wording changes from jmc@openbsd, with some
additional consistency changes by me.

Use
.In header.h
instead of
.Fd #include \*[Lt]header.h\*[Gt]
Much easier to read and write, and supported by groff for ages.
Okayed by ross.

Typo and grammar fixes. Sort SEE ALSO. Use more mdoc.

Improve English, sort SEE ALSO.
XXX: list of valid charclasses should be added.

Drop trailing space.

add manual pages for iswctype(), towctrans(), wctype() and wctrans().

Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.16.4; 1.16.6;
SUSv3 says, wcwidth(wc) returns:
1) wc is nul wide-character, return 0.
2) wc is printable wide-character, return column width.
3) else, return -1.
but our implementation, case 3) returns 0. it's wrong!

branches: 1.15.26;
A little libc namespace housekeeping exercise:
* Make vfprintf_unlocked() an internal function, c.f. __svfscanf_unlocked().
* Add internal names for arc4random(), endnetpath(), fhstatvfs(),
fstatvfs(), mkstemp(), shquote(), statvfs(), taddr2uaddr(), uaddr2taddr(),
uuid_create_nil(), uuid_is_nil(), and wcwidth().
* Include namespace.h where supposed to.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

- support for the additional code area of zh_CN.GB18030 locale,
- make sure that __nbrune_t is unsigned 32bit clean and
- fix wrong copyright notice in the last commit.
this patch is contributed by Takehiko NOZAKI <sigsegv at s25 dot xrea dot com>.

Include <string.h> for strcmp(3) prototype.

iswctype: return 0 if charclass is 0.

add some ISO C 1995 I18N functions and types:
btowc, wctrans, towctrans, wcscoll, wcsxfrm, wctype_t and wctrans_t.

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

wcwidth's argument is wchar_t. not wint_t.

branches: 1.7.2;
de-lint.

avoid exposure of libc internal symbols into public header (wctype.h).

support multibytes in isw*/tow*.

add dummy wcwidth() and wcswidth(). wcwidth() will be updated later.

correct return type for tow*.

have dummy wctype.h and isw*/tow*. part of libc minor 68 -> 69 crank.
equivalent to is*/to* at this moment. we'll put wchar_t code in the
near future.

branches: 1.1.2;
file iswctype.c was initially added on branch minoura-xpg4dl.

libc/locale: Nix needless size_t casts.

No functional change intended. No binary change to aarch64 libc.

Prompted by code inspection around PR lib/58319.

branches: 1.13.22;
Provide explicit LC_C_LOCALE accessor and drop the various NULL checks.
Provide LC_GLOBAL_LOCALE in a way that works with all locale functions.
Merge constant data used by the initial global locale and the C locale.
Drop function call layer for _current_locale() and directly return the
locale reference, not a pointer to it. Use protected access for global
variables, so that libc references can avoid the GOT overhead.

Add wcwidth_l, wcswidth_l and the wctype.h family of *_l functions.

branches: 1.11.6; 1.11.12;
1. split runetype_local.h -> runetype_file.h
and remove renameing _Rune* -> _NBRune* namespace protection.

FreeBSD traditionaly exposes struct _Rune* in runetype.h
which included by ctype.h. it may cause conflicting type error
in our cross build process, former we use renaming namespace
to avoid this problem, now i reworked more resonable way.

2. merge rune_local.h to runetype_local.h, and remove it.
3. split bsdctype.h -> bsdctype_{file,local}.h

stop to renaming _CTYPE_* -> _RUNETYPE_*.

remove duplicated towctrans internal function.

more split ctype.h -> sys/ctype_inline.h, sys/ctype_bits.h

rune.h is not public, so merge with rune_local.h

1. {wctype,wctrans,mbstate}_t: switch MD to MI like other
libc implementation (such as *BSD and glibc2).

2. don't typedef void * wc{type,trans}_t, suggested by soda@-san.
it may pass through compiler type check, it's harmful.
so i introduce dummy struct __tag_wc{type,trans}_t(iconv_t already does).

no ABI change was made.

PR lib/40411, nuke wctrans_init(). lazy initialization is not good idea.
statically initialized _DefaultRuneLocale.rl_wctrans field.
so we can re-const-ify _DefaultRuneLocale.

pkgsrc/shells/standalone-tcsh should be rebuild,
because it's staticaly linked binary.