branches: 1.15.18;
Fix our iconv version for the issues that apply to us from CVE-2014-3951
(which are the:
- Consistently pass around context information using a simple pointer.
This fixes some dereferencing bugs in Chinese character set conversions.
- Fix Simplified Chinese character set conversions by switching around the
fields of an internal struct so it corresponds with the way variables of
this type are initialised.
part)
Patch taken from FreeBSD and mutilated to fit.
FreeBSD credits: Manuel Mausz (reporter), Tijl Coosemans (report handler)

branches: 1.14.4;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.13.4; 1.13.10;
Make intermediate size variable size_t like the rest to avoid
unnecessary casting.

branches: 1.12.6;
remove unused include, locale.h.

branches: 1.11.16; 1.11.18;
1. add iconv support for following encodings:
Chinese Simplefied
HZ, HZ8 - 7/8bit stateful encoding, see RFC1842,1843. (libHZ)
zW - 7bit stateful encoding, see RFC1842. (libZW)
2. add citrus_prop.[ch] - parser for encoding module's init parameter strings.

added Chinese Tradisional Big5 family,
Big5-2003, Big5-ETen, Big5-IBM, Big-5E, Big-5+.

``Big5 is now the alias of Big5-ETen,
if you want Unicode.org's obsolete mappings, use Big5-IBM instead.

fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

- move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.

introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.

fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.

- change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

branches: 1.1.2;
file citrus_big5_data.h was initially added on branch perseant-stdc-iso10646.

branches: 1.1.2;
file citrus_big5_k2u.h was initially added on branch perseant-stdc-iso10646.

branches: 1.1.2;
file citrus_big5_u2k.h was initially added on branch perseant-stdc-iso10646.

branches: 1.5.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.4.6;
remove unused variable, pointed by nonaka-san, thanks.

branches: 1.3.2; 1.3.6;
remove unused include, locale.h.

branches: 1.2.10; 1.2.12;
fix typo.

add new encoding support to iconv(3):
- RISCOS-LATIN1
- DEC-MCS
- DEC-HANYU(libDECHanyu)

add new encoding support to iconv(3):
- RISCOS-LATIN1
- DEC-MCS
- DEC-HANYU(libDECHanyu)

branches: 1.17.20;
revert previous, it causes other problem and I cannot easily debug it.

PR/47602: Christos Zoulas: getwc() modifies input instead of returning EILSEQ.
Waited for almost a year for feedback and there was none.

Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.14.8; 1.14.14;
merge christos-time_t

Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.

branches: 1.12.4; 1.12.6;
remove unused include, locale.h.

branches: 1.11.18; 1.11.20;
Coverity CID 2461: Bail out quickly instead of accessing uninitialized variables

fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().

fix a problem on wc->mb conversion for G2 plane.
This is reported by MORIYAMA Masayuki <msyk _at_ mtg.biglobe.ne.jp> and
"NARUSE, Yui" <naruse _at_ airemix.com>, and fixed by MORIYAMA-san.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

- move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.

introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.

fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.

- change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

branches: 1.1.2;
file citrus_euc_data.h was initially added on branch perseant-stdc-iso10646.

branches: 1.1.2;
file citrus_euc_k2u.h was initially added on branch perseant-stdc-iso10646.

branches: 1.1.2;
file citrus_euc_u2k.h was initially added on branch perseant-stdc-iso10646.

branches: 1.12.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.11.2; 1.11.6; 1.11.26;
remove unused include, locale.h.

branches: 1.10.18; 1.10.20;
fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().

add csmapper:CNS11643-1,2 and esdb:ISO-2022-CN,
integrate esdb:EUC-TW, locale:zh_TW.eucTW.

don't use uninitialized variable ('chlenbak').
the problem reported by Markus Illenseer on tech-userlevel@
and analyzed by enami tsugutomo.

correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

- move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.

introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.

fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.

- change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.

branches: 1.8.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.7.2; 1.7.6; 1.7.26;
remove unused include, locale.h.

branches: 1.6.18; 1.6.20;
MB_CUR_MAX should be 2 when MODE_2BYTE flag set.

fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().

correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

No need in default label if default case is not processed.

add support for zh_CN.GB18030 locale.
(lib/20835 by sigsegv at s25 dot xrea dot com)
XXX: charclass for GBK compatible area is not supported.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

add support for zh_CN.GB18030 locale.
(lib/20835 by sigsegv at s25 dot xrea dot com)
XXX: charclass for GBK compatible area is not supported.

branches: 1.4.18;
Fix our iconv version for the issues that apply to us from CVE-2014-3951
(which are the:
- Consistently pass around context information using a simple pointer.
This fixes some dereferencing bugs in Chinese character set conversions.
- Fix Simplified Chinese character set conversions by switching around the
fields of an internal struct so it corresponds with the way variables of
this type are initialised.
part)
Patch taken from FreeBSD and mutilated to fit.
FreeBSD credits: Manuel Mausz (reporter), Tijl Coosemans (report handler)

branches: 1.3.4;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.2.2; 1.2.6; 1.2.26;
remove unused include, locale.h.

branches: 1.1.16; 1.1.18;
1. add iconv support for following encodings:
Chinese Simplefied
HZ, HZ8 - 7/8bit stateful encoding, see RFC1842,1843. (libHZ)
zW - 7bit stateful encoding, see RFC1842. (libZW)
2. add citrus_prop.[ch] - parser for encoding module's init parameter strings.

1. add iconv support for following encodings:
Chinese Simplefied
HZ, HZ8 - 7/8bit stateful encoding, see RFC1842,1843. (libHZ)
zW - 7bit stateful encoding, see RFC1842. (libZW)
2. add citrus_prop.[ch] - parser for encoding module's init parameter strings.

branches: 1.3.38;
Actually return something deterministic

add iconv_t cache.
- separate shared part and context part from iconv_t.
- add cache of shared part.
and, bump shlib_version minor of i18n modules because iconv i/f is changed.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

branches: 1.16.34;
Remove unused variables. From cppcheck via Henning Petersen in PR 45997.

branches: 1.15.42;
avoid infinity loop, iso2022 module's stdenc_get_state_desc_generic()
never return _STDENC_SDGEN_INITIAL.

1. add iconv support for followint encoding:
Vietnamese
TCVN 8bit Viet Nam National Standard
VISCII 8bit RFC1456
VIQR 7bit RFC1456(libVIQR)
Unicode Escape (GNU libiconv compatibility)
C99, JAVA (libUES)
2. fix iconv_std module:
add special treatment for POSIX Defect Report #288 case.

Coverity CID 2473: Fix uninitialized variable reference.

change #include <sys/endian.h> => #include <machine/endian.h> so that
it's (more) consistent in the tree; this, along with changing tools/compat's
autoconf detection from AC_CHECK_FUNCS to AC_CHECK_DECLS makes the vast
majority of htobe16 and friends' redefinition errors bite the dust.
Tested with -current and FreeBSD.

fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().

White space nit- don't put a space before/after increment/decrement
operators.

constify.

constify.

_citrus_iconv_std_iconv_convert:
- allow 0 byte input. PR/23889 (Valeriy E. Ushakov)
- remove redundant E2BIG check, which should be done by
each encoding modules anyway.

fix the problem about resetting the state of iconv().
iconv() was not conforming to the SUSv3 at this point.
reported by Bruno Haible.

separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."

add iconv_t cache.
- separate shared part and context part from iconv_t.
- add cache of shared part.
and, bump shlib_version minor of i18n modules because iconv i/f is changed.

cosmetic change.

correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

add iconv_t cache.
- separate shared part and context part from iconv_t.
- add cache of shared part.
and, bump shlib_version minor of i18n modules because iconv i/f is changed.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

branches: 1.23.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.22.2; 1.22.8;
revert r1.21, still problem exists for posix2008 mbsnrtowcs(not yet commited),
but i have no time to investigate t_mbrtowc failure.

update string pointer when input is partial escape sequence or multibyte.

Remove tautology.

branches: 1.19.6;
remove unused include, locale.h.

branches: 1.18.6; 1.18.8;
fix lib/37290
- don't call abort(3) when there's no suitable charset found.
- use iso-8859-1(or INIT1 if specified) for C1 control char.

lib/36938 mbtowc misbehaving after invalid char sequence
- make sure to initialize mbtowc's internal state.
- add regression test.

branches: 1.16.10;
if INIT0 specified, use it instead of ASCII.

Coverity CID 2464: Don't use uninitialized variables; exit with error quickly.

fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().

fix typo in _DIAGASSERT

constify.

comment on values from 0x80000000 to 0xffffffff

safer realloc

use bounded string op

correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

- move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.

introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.

drop __packed__ attribute from _ISO2022State.

fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.

- change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

branches: 1.1.2;
file citrus_iso2022_data.h was initially added on branch perseant-stdc-iso10646.

branches: 1.1.2;
file citrus_iso2022_k2u.h was initially added on branch perseant-stdc-iso10646.

branches: 1.1.2;
file citrus_iso2022_u2k.h was initially added on branch perseant-stdc-iso10646.

The correct fix for lib/51000 is not to remove the apparently dead assignment
but to add the missing usage.

From https://bitbucket.org/tnozaki/netbsd/commits/24b35fdce250
Thanks to Takehiko Nozaki

branches: 1.6.10; 1.6.16;
PR 51000 David Binderman: comment out unused assignment

Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.4.2; 1.4.6; 1.4.26;
remove unused include, locale.h.

branches: 1.3.6; 1.3.8;
remove invalid range check.

branches: 1.2.4;
make del(\x7f) pass through.

add new iconv module libJOHAB,
this supports S.Korean character encoding scheme ``JOHAB''.

add new iconv module libJOHAB,
this supports S.Korean character encoding scheme ``JOHAB''.

use NONIDENTICAL instead of ILSEQ if the reverse conversion of ISO646.
(fix for lib/22135)

separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."

sync with the last change for citrus_hash.h.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

sync with the last change for citrus_hash.h.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

Fix typos in comments, mainly s/calcurate/calculate/.

branches: 1.12.14;
Avoid left-shifting a negative number in the non-compat case, too.

Correct Undefined Behavior in libc/citrus

Unportable left shift reported with MKSANITIZER=yes USE_SANITIZER=undefined:

# nm /usr/lib/libc.so|grep sanit
/public/src.git/lib/libc/citrus/modules/citrus_mapper_std.c:173:8: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'

Sponsored by <The NetBSD Foundation>

branches: 1.10.38;
fix memory leak, pointed by nonaka-san(again^3).

branches: 1.9.2;
Use boolean AND instead of bitwise one in _DIAGASSERT.
From Henning Petersen in PR 45518.

correct typo in _DIAGASSERT() and some KNF.
pointed by uebayashi-san, thanks!

mapper_std iconv module and mkcsmapper(1) now can treat
plain-row-col charset and 4byte code(like GB18030) as SRC_ZONE.

Coveriry CID 2472: If the number of bits is invalid, return immediately
instead of accessing uninitialized variables.

change #include <sys/endian.h> => #include <machine/endian.h> so that
it's (more) consistent in the tree; this, along with changing tools/compat's
autoconf detection from AC_CHECK_FUNCS to AC_CHECK_DECLS makes the vast
majority of htobe16 and friends' redefinition errors bite the dust.
Tested with -current and FreeBSD.

constify.

separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."

sync with the last change for citrus_hash.h.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

mapper_std iconv module and mkcsmapper(1) now can treat
plain-row-col charset and 4byte code(like GB18030) as SRC_ZONE.

separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

mapper_std iconv module and mkcsmapper(1) now can treat
plain-row-col charset and 4byte code(like GB18030) as SRC_ZONE.

separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."

fix lib/22023 - there were references to uninitialized variables.

sync with the last change for citrus_hash.h.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.

branches: 1.14.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.13.2; 1.13.6; 1.13.26;
remove unused include, locale.h.

branches: 1.12.10; 1.12.12;
iconv: add following CCS/CES support.
- CNS11643-[3-7] <-> UCS:BMP/SIP (EUC-TW, ISO-2022-CN-EXT)
- HKSCS <-> UCS:BMP/SIP (Big5-HKSCS)
- JISX0213-[1-2] <-> UCS:BMP/SIP (EUC-JIS-2004,Shift_JIS-2004,ISO-2022-JP-2004)

add missing _citrus_MSKanji_stdenc_get_state_desc_generic() function.
pointed out by Patrick Welche <prlw1 _at_ newn _dot_ cam _dot_ ac _dot_ uk>

make sure that this module can handle all private/vendor-defined
character area.
This is reported by MORIYAMA Masayuki <msyk _at_ mtg.biglobe.ne.jp> and
"NARUSE, Yui" <naruse _at_ airemix.com>, and fixed by MORIYAMA-san.

fix typo in _DIAGASSERT

_citrus_MSKanji_wcrtomb_priv: return E2BIG correctly.

correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

- move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.

introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.

fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.

- change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

branches: 1.1.2;
file citrus_mskanji_data.h was initially added on branch perseant-stdc-iso10646.

branches: 1.1.2;
file citrus_mskanji_k2u.h was initially added on branch perseant-stdc-iso10646.

branches: 1.1.2;
file citrus_mskanji_u2k.h was initially added on branch perseant-stdc-iso10646.

lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.

branches: 1.4.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.3.6;
Remove unused variables. From cppcheck via Henning Petersen in PR 45997.

branches: 1.2.6;
Mark function as static and give it an explicit return type.

1. add iconv support for followint encoding:
Vietnamese
TCVN 8bit Viet Nam National Standard
VISCII 8bit RFC1456
VIQR 7bit RFC1456(libVIQR)
Unicode Escape (GNU libiconv compatibility)
C99, JAVA (libUES)
2. fix iconv_std module:
add special treatment for POSIX Defect Report #288 case.

1. add iconv support for followint encoding:
Vietnamese
TCVN 8bit Viet Nam National Standard
VISCII 8bit RFC1456
VIQR 7bit RFC1456(libVIQR)
Unicode Escape (GNU libiconv compatibility)
C99, JAVA (libUES)
2. fix iconv_std module:
add special treatment for POSIX Defect Report #288 case.

lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.

branches: 1.12.34;
Remove unused variables. From cppcheck via Henning Petersen in PR 45997.

branches: 1.11.6;
fix byte order mark related bug introduced by previous commit,
reported by Sverre Froyen via current-user, thanks!

1. fix wrong byte order mark of utf-16, reported by NARUSE Yui -san.
patch provided by tshiozak@ -san.

2. don't eat 0xfeff/0xfffe if they don't appear at the first of bytestream.
noticed y tshiozak@ -san, patch provied by me.

thanks a lot.

branches: 1.9.4; 1.9.6;
remove unused include, locale.h.

branches: 1.8.2; 1.8.4;
add BOM to utf-16/32 stream in case that endian is not specified,
like other iconv implementation, GNU libiconv, glibc2 iconv, perl iconv.

branches: 1.7.8; 1.7.14;
don't pass through surrogate character(0xD800 - 0xDFFF).

Coverity 2462: Bail out quickly instead of accessing uninitialized variables.

change #include <sys/endian.h> => #include <machine/endian.h> so that
it's (more) consistent in the tree; this, along with changing tools/compat's
autoconf detection from AC_CHECK_FUNCS to AC_CHECK_DECLS makes the vast
majority of htobe16 and friends' redefinition errors bite the dust.
Tested with -current and FreeBSD.

fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().

$NetBSD $ -> $NetBSD$

correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.

branches: 1.6.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.5.42; 1.5.48;
cleanup code

Coverity CID 1439: Prevent array index out of bounds access.

fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().

anonymous union between chlen and _UTF7StatePrive has
compilation problem with gcc295.
this union attempt to make mbsinit(3) handle multibyte state correctly,
but it's useless as far as we use utf-7 only iconv interface.
so i eliminate ctype feature.

patch contributed by Joerg Sonnenberg(who porting Citrus to DragonFlyBSD).
and yamt-san gave me advice, thanks a lot.

add new citrus iconv module UTF-7.
thanks advice, yamt-san.

add new citrus iconv module UTF-7.
thanks advice, yamt-san.

branches: 1.18.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.17.2; 1.17.6; 1.17.26;
remove unused include, locale.h.

branches: 1.16.10; 1.16.12;
disallow illegal utf-8 byte sequence and surrogate chars (RFC3629).
5-6 byte sequence(0x110000 - 0x7FFFFFFF) are still available
for backward compatibility.

Coverity CID 1440: off by one in array count.

Coverity CID 2463: Bail out instead of accessing uninitialized variables.

fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().

UTF8EncodingInfo is an empty struct; remove noop code and DIAGASSERT.
From Joerg Sonnenberger

constify.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

fix the problem if EILSEQ is caused in mbrtowc(3) under *.UTF-8 locales.
(lib/20873 by khym at azeotrope dot org)

- move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.

introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.

fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.

- change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.

branches: 1.2.2;
don't check variable s0 before assign.

refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)

branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.

branches: 1.6.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.5.6;
remove unused variable, pointed by nonaka-san, thanks.

branches: 1.4.2; 1.4.6;
remove unused include, locale.h.

branches: 1.3.16; 1.3.18;
fix memory leak.

change uint32_t to size_t; fixes build issue on 64-bit platforms.

1. add iconv support for followint encoding:
Vietnamese
TCVN 8bit Viet Nam National Standard
VISCII 8bit RFC1456
VIQR 7bit RFC1456(libVIQR)
Unicode Escape (GNU libiconv compatibility)
C99, JAVA (libUES)
2. fix iconv_std module:
add special treatment for POSIX Defect Report #288 case.

1. add iconv support for followint encoding:
Vietnamese
TCVN 8bit Viet Nam National Standard
VISCII 8bit RFC1456
VIQR 7bit RFC1456(libVIQR)
Unicode Escape (GNU libiconv compatibility)
C99, JAVA (libUES)
2. fix iconv_std module:
add special treatment for POSIX Defect Report #288 case.

lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.

branches: 1.5.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.

branches: 1.4.2; 1.4.6; 1.4.26;
remove unused include, locale.h.

branches: 1.3.16; 1.3.18;
don't read input string more than MB_LEN_MAX(maybe redundant escape sequence).

don't throw EILSEQ when byte sequence is "zW ".

1. add iconv support for following encodings:
Chinese Simplefied
HZ, HZ8 - 7/8bit stateful encoding, see RFC1842,1843. (libHZ)
zW - 7bit stateful encoding, see RFC1842. (libZW)
2. add citrus_prop.[ch] - parser for encoding module's init parameter strings.

1. add iconv support for following encodings:
Chinese Simplefied
HZ, HZ8 - 7/8bit stateful encoding, see RFC1842,1843. (libHZ)
zW - 7bit stateful encoding, see RFC1842. (libZW)
2. add citrus_prop.[ch] - parser for encoding module's init parameter strings.