Home | History | Annotate | only in /src/lib/libc/citrus/modules
History log of /src/lib/libc/citrus/modules
RevisionDateAuthorComments
 1.15 24-Jun-2014  spz branches: 1.15.18;
Fix our iconv version for the issues that apply to us from CVE-2014-3951
(which are the:
- Consistently pass around context information using a simple pointer.
This fixes some dereferencing bugs in Chinese character set conversions.
- Fix Simplified Chinese character set conversions by switching around the
fields of an internal struct so it corresponds with the way variables of
this type are initialised.
part)
Patch taken from FreeBSD and mutilated to fit.
FreeBSD credits: Manuel Mausz (reporter), Tijl Coosemans (report handler)
 1.14 28-May-2013  joerg branches: 1.14.4;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.13 23-May-2011  joerg branches: 1.13.4; 1.13.10;
Make intermediate size variable size_t like the rest to avoid
unnecessary casting.
 1.12 14-Jun-2008  tnozaki branches: 1.12.6;
remove unused include, locale.h.
 1.11 22-Nov-2006  tnozaki branches: 1.11.16; 1.11.18;
1. add iconv support for following encodings:
Chinese Simplefied
HZ, HZ8 - 7/8bit stateful encoding, see RFC1842,1843. (libHZ)
zW - 7bit stateful encoding, see RFC1842. (libZW)
2. add citrus_prop.[ch] - parser for encoding module's init parameter strings.
 1.10 19-Jun-2006  tnozaki added Chinese Tradisional Big5 family,
Big5-2003, Big5-ETen, Big5-IBM, Big-5E, Big-5+.

``Big5 is now the alias of Big5-ETen,
if you want Unicode.org's obsolete mappings, use Big5-IBM instead.
 1.9 29-Oct-2005  tshiozak fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().
 1.8 07-Aug-2003  agc Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.
 1.7 26-Jun-2003  tshiozak correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.
 1.6 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.5 28-Mar-2002  yamt - move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.
 1.4 28-Mar-2002  yamt introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.
 1.3 27-Mar-2002  yamt fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.
 1.2 27-Mar-2002  yamt - change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.3 25-Apr-2002  nathanw Catch up to -current.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_big5.c was added on branch nathanw_sa on 2002-03-22 20:42:02 +0000
 1.11.18.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.11.16.1 17-Jun-2008  yamt sync with head.
 1.12.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.12.6.1 14-Jun-2008  tnozaki file citrus_big5.c was added on branch christos-time_t on 2008-06-14 16:01:08 +0000
 1.13.10.2 20-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.13.10.1 23-Jun-2013  tls resync from head
 1.13.4.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.14.4.1 10-Aug-2014  tls Rebase.
 1.15.18.5 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.15.18.4 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.15.18.3 31-Jul-2017  perseant Make the compilation conditional on __STDC_ISO_10646__, so that applications
that need one of the East Asian locales but don't want __STDC_ISO_10646__
don't have to pay the penalty for it.
 1.15.18.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.15.18.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.2 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_big5.h was added on branch nathanw_sa on 2002-03-22 20:42:02 +0000
 1.1 21-Jul-2017  perseant branches: 1.1.2;
file citrus_big5_data.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.1 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.1 20-Jan-2018  perseant branches: 1.1.2;
file citrus_big5_k2u.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.2 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.1.2.1 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.1 20-Jan-2018  perseant branches: 1.1.2;
file citrus_big5_u2k.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.2 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.1.2.1 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.5 28-May-2013  joerg branches: 1.5.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.4 19-Nov-2011  tnozaki branches: 1.4.6;
remove unused variable, pointed by nonaka-san, thanks.
 1.3 14-Jun-2008  tnozaki branches: 1.3.2; 1.3.6;
remove unused include, locale.h.
 1.2 24-Apr-2007  tnozaki branches: 1.2.10; 1.2.12;
fix typo.
 1.1 01-Apr-2007  tnozaki add new encoding support to iconv(3):
- RISCOS-LATIN1
- DEC-MCS
- DEC-HANYU(libDECHanyu)
 1.2.12.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.2.10.1 17-Jun-2008  yamt sync with head.
 1.3.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.3.6.1 14-Jun-2008  tnozaki file citrus_dechanyu.c was added on branch christos-time_t on 2008-06-14 16:01:08 +0000
 1.3.2.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.2.1 17-Apr-2012  yamt sync with head
 1.4.6.1 23-Jun-2013  tls resync from head
 1.5.22.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.5.22.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.1 01-Apr-2007  tnozaki add new encoding support to iconv(3):
- RISCOS-LATIN1
- DEC-MCS
- DEC-HANYU(libDECHanyu)
 1.17 18-Jan-2014  christos branches: 1.17.20;
revert previous, it causes other problem and I cannot easily debug it.
 1.16 16-Jan-2014  christos PR/47602: Christos Zoulas: getwc() modifies input instead of returning EILSEQ.
Waited for almost a year for feedback and there was none.
 1.15 28-May-2013  joerg Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.14 11-Jan-2009  christos branches: 1.14.8; 1.14.14;
merge christos-time_t
 1.13 02-Jan-2009  tnozaki Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES} db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).

[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].

[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().

[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<

old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.
 1.12 14-Jun-2008  tnozaki branches: 1.12.4; 1.12.6;
remove unused include, locale.h.
 1.11 19-Mar-2006  christos branches: 1.11.18; 1.11.20;
Coverity CID 2461: Bail out quickly instead of accessing uninitialized variables
 1.10 29-Oct-2005  tshiozak fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().
 1.9 18-Oct-2005  tshiozak fix a problem on wc->mb conversion for G2 plane.
This is reported by MORIYAMA Masayuki <msyk _at_ mtg.biglobe.ne.jp> and
"NARUSE, Yui" <naruse _at_ airemix.com>, and fixed by MORIYAMA-san.
 1.8 07-Aug-2003  agc Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.
 1.7 26-Jun-2003  tshiozak correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.
 1.6 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.5 28-Mar-2002  yamt - move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.
 1.4 28-Mar-2002  yamt introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.
 1.3 27-Mar-2002  yamt fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.
 1.2 27-Mar-2002  yamt - change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.3 25-Apr-2002  nathanw Catch up to -current.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_euc.c was added on branch nathanw_sa on 2002-03-22 20:42:02 +0000
 1.11.20.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.11.18.1 17-Jun-2008  yamt sync with head.
 1.12.6.2 04-Jan-2009  christos merge with head.
 1.12.6.1 14-Jun-2008  christos file citrus_euc.c was added on branch christos-time_t on 2009-01-04 17:02:19 +0000
 1.12.4.1 15-Jan-2009  snj Pull up following revision(s) (requested by tnozaki in ticket #222):
distrib/sets/lists/base/mi: revision 1.788
distrib/utils/libhack/Makefile.inc: revision 1.23
distrib/utils/libhack/Makefile: revision 1.22
etc/mtree/NetBSD.dist: revision 1.387
include/locale.h: revision 1.15
lib/libc/citrus/Makefile.inc: revision 1.6
lib/libc/citrus/citrus_aliasname_local.h: revision 1.1
lib/libc/citrus/citrus_bcs.h: revision 1.5
lib/libc/citrus/citrus_bcs_strtol.c: revision 1.1
lib/libc/citrus/citrus_bcs_strtoul.c: revision 1.1
lib/libc/citrus/citrus_csmapper.c: revision 1.9
lib/libc/citrus/citrus_fix_grouping.h: revision 1.1
lib/libc/citrus/citrus_lc_ctype.c: revision 1.1
lib/libc/citrus/citrus_lc_messages.c: revision 1.1
lib/libc/citrus/citrus_lc_messages.h: revision 1.1
lib/libc/citrus/citrus_lc_monetary.c: revision 1.1
lib/libc/citrus/citrus_lc_monetary.h: revision 1.1
lib/libc/citrus/citrus_lc_numeric.c: revision 1.1
lib/libc/citrus/citrus_lc_numeric.h: revision 1.1
lib/libc/citrus/citrus_lc_template.h: revision 1.1
lib/libc/citrus/citrus_lc_template_decl.h: revision 1.1
lib/libc/citrus/citrus_lc_time.c: revision 1.1
lib/libc/citrus/citrus_lc_time.h: revision 1.1
lib/libc/citrus/citrus_module.c: revision 1.8
lib/libc/citrus/citrus_namespace.h: revision 1.7
lib/libc/citrus/modules/citrus_euc.c: revision 1.13
lib/libc/gen/Makefile.inc: revision 1.166
lib/libc/gen/isctype.c: revision 1.17 via patch
lib/libc/gen/tolower_.c: revision 1.10
lib/libc/gen/toupper_.c: revision 1.10
lib/libc/iconv/Makefile.inc: revision 1.3
lib/libc/iconv/iconv.c: revision 1.7
lib/libc/locale/Makefile.inc: revision 1.53
lib/libc/locale/___runetype_mb.c: file removal
lib/libc/locale/_wctrans.c: revision 1.7
lib/libc/locale/_wctrans_local.h: revision 1.3
lib/libc/locale/_wctype.c: revision 1.1
lib/libc/locale/_wctype_local.h: revision 1.1
lib/libc/locale/aliasname.c: revision 1.3
lib/libc/locale/aliasname_local.h: revision 1.2
lib/libc/locale/bsdctype.c: revision 1.1
lib/libc/locale/bsdctype.h: revision 1.1
lib/libc/locale/ctypeio.c: revision 1.8
lib/libc/locale/ctypeio.h: revision 1.2
lib/libc/locale/current_locale.c: revision 1.1
lib/libc/locale/dummy_lc_collate.c: revision 1.1
lib/libc/locale/dummy_lc_template.h: revision 1.1
lib/libc/locale/fix_grouping.c: revision 1.1
lib/libc/locale/fix_grouping.h: revision 1.1
lib/libc/locale/generic_lc_all.c: revision 1.1
lib/libc/locale/generic_lc_template.h: revision 1.1
lib/libc/locale/generic_lc_template_decl.h: revision 1.1
lib/libc/locale/global_locale.c: revision 1.1
lib/libc/locale/iswctype.c: file removal
lib/libc/locale/iswctype_mb.c: revision 1.1
lib/libc/locale/iswctype_sb.c: revision 1.6
lib/libc/locale/lcmessages.c: file removal
lib/libc/locale/lcmessages.h: file removal
lib/libc/locale/lcmonetary.c: file removal
lib/libc/locale/lcmonetary.h: file removal
lib/libc/locale/lcnumeric.c: file removal
lib/libc/locale/lcnumeric.h: file removal
lib/libc/locale/lctime.c: file removal
lib/libc/locale/lctime.h: file removal
lib/libc/locale/localeconv.c: revision 1.15
lib/libc/locale/localeio.c: revision 1.2
lib/libc/locale/localeio.h: revision 1.2
lib/libc/locale/localeio_lc_ctype.c: revision 1.1
lib/libc/locale/localeio_lc_messages.c: revision 1.1
lib/libc/locale/localeio_lc_monetary.c: revision 1.1
lib/libc/locale/localeio_lc_numeric.c: revision 1.1
lib/libc/locale/localeio_lc_time.c: revision 1.1
lib/libc/locale/multibyte.h: revision 1.4
lib/libc/locale/multibyte_amd1.c: revision 1.6
lib/libc/locale/multibyte_c90.c: revision 1.5
lib/libc/locale/nb_lc_messages_misc.h: revision 1.1
lib/libc/locale/nb_lc_monetary_misc.h: revision 1.1
lib/libc/locale/nb_lc_numeric_misc.h: revision 1.1
lib/libc/locale/nb_lc_template.h: revision 1.1
lib/libc/locale/nb_lc_template_decl.h: revision 1.1
lib/libc/locale/nb_lc_time_misc.h: revision 1.1
lib/libc/locale/nl_langinfo.c: revision 1.12
lib/libc/locale/rune.c: revision 1.31
lib/libc/locale/rune.h: revision 1.13
lib/libc/locale/rune_local.h: revision 1.11
lib/libc/locale/runeglue.c: revision 1.14
lib/libc/locale/runetable.c: revision 1.18
lib/libc/locale/setlocale.c: revision 1.55 via patch
lib/libc/locale/setlocale_local.h: revision 1.1
lib/libc/locale/setrunelocale.c: file removal
lib/libc/nls/Makefile.inc: revision 1.9
lib/libc/nls/catopen.c: revision 1.26
lib/libc/string/Makefile.inc: revision 1.68
lib/libc/string/wcscmp.c: revision 1.6
lib/libc/string/wcsncmp.c: revision 1.6
lib/libc/string/wcswidth.c: file removal
lib/libc/string/wmemcmp.c: revision 1.4
regress/lib/libc/locale/Makefile: revision 1.4
regress/lib/libc/locale/ctype1/Makefile: revision 1.4
regress/lib/libc/locale/ctype1/en_US.UTF-8.exp.uue: revision 1.2
regress/lib/libc/locale/ctype1/ja_JP.ISO-2022-JP.exp.uue: revision 1.1
regress/lib/libc/locale/ctype1/ja_JP.ISO-2022-JP.in.uue: revision 1.1
regress/lib/libc/locale/ctype1/ja_JP.ISO2022-JP.exp.uue: file removal
regress/lib/libc/locale/ctype1/ja_JP.ISO2022-JP.in.uue: file removal
regress/lib/libc/locale/ctype1/ja_JP.SJIS.exp.uue: revision 1.2
regress/lib/libc/locale/ctype1/ja_JP.eucJP.exp.uue: revision 1.2
regress/lib/libc/locale/ctype2/Makefile: revision 1.5
regress/lib/libc/locale/ctype2/ja_JP.ISO-2022-JP-2.in.uue: revision 1.1
regress/lib/libc/locale/ctype2/ja_JP.ISO2022-JP2.in.uue: file removal
regress/lib/libc/locale/ctype3/Makefile: revision 1.5
regress/lib/libc/locale/mbtowc/Makefile: revision 1.3
regress/lib/libc/locale/mbtowc/ja_JP.ISO-2022-JP: revision 1.1
regress/lib/libc/locale/mbtowc/ja_JP.ISO2022-JP: file removal
regress/lib/libc/locale/wcstod/wcstod_test.c: revision 1.2
share/locale/Makefile.locale: revision 1.1
share/locale/Makefile: revision 1.5
share/locale/ctype/Makefile: revision 1.28
share/locale/locale.alias: revision 1.11
share/locale/messages/Makefile: revision 1.5
share/locale/messages/en_US.ISO8859-1.src: file removal
share/locale/messages/en_US.US-ASCII.src: revision 1.1
share/locale/messages/ja_JP.ISO-2022-JP.src: revision 1.1
share/locale/messages/ja_JP.ct.src: revision 1.1
share/locale/messages/sr_ME.ISO8859-2.src: revision 1.1
share/locale/messages/sr_ME.ISO8859-5.src: revision 1.1
share/locale/messages/sr_YU.ISO8859-2.src: file removal
share/locale/messages/sr_YU.ISO8859-5.src: file removal
share/locale/messages/sr_YU.UTF-8.src: file removal
share/locale/messages/zh_CN.GB18030.src: file removal
share/locale/messages/zh_TW.eucTW.src: revision 1.1
share/locale/monetary/Makefile: revision 1.5
share/locale/monetary/af_ZA.ISO8859-1.src: revision 1.2
share/locale/monetary/am_ET.UTF-8.src: revision 1.2
share/locale/monetary/be_BY.CP1131.src: file removal
share/locale/monetary/be_BY.CP1251.src: revision 1.2
share/locale/monetary/be_BY.ISO8859-5.src: revision 1.2
share/locale/monetary/be_BY.UTF-8.src: revision 1.2
share/locale/monetary/bg_BG.CP1251.src: revision 1.2
share/locale/monetary/bg_BG.UTF-8.src: revision 1.2
share/locale/monetary/ca_ES.ISO8859-1.src: revision 1.2
share/locale/monetary/cs_CZ.ISO8859-2.src: revision 1.2
share/locale/monetary/cs_CZ.UTF-8.src: revision 1.2
share/locale/monetary/da_DK.ISO8859-1.src: revision 1.2
share/locale/monetary/de_AT.ISO8859-1.src: revision 1.2
share/locale/monetary/de_CH.ISO8859-1.src: revision 1.2
share/locale/monetary/de_DE.ISO8859-1.src: revision 1.2
share/locale/monetary/el_GR.ISO8859-7.src: revision 1.2
share/locale/monetary/en_AU.ISO8859-1.src: revision 1.2
share/locale/monetary/en_CA.ISO8859-1.src: revision 1.2
share/locale/monetary/en_GB.ISO8859-1.src: revision 1.2
share/locale/monetary/en_GB.UTF-8.src: revision 1.2
share/locale/monetary/en_IE.UTF-8.src: revision 1.2
share/locale/monetary/en_NZ.ISO8859-1.src: revision 1.2
share/locale/monetary/en_US.ISO8859-1.src: revision 1.2
share/locale/monetary/en_US.US-ASCII.src: revision 1.1
share/locale/monetary/es_ES.ISO8859-1.src: revision 1.2
share/locale/monetary/et_EE.ISO8859-15.src: revision 1.2
share/locale/monetary/fi_FI.ISO8859-1.src: revision 1.2
share/locale/monetary/fr_BE.ISO8859-1.src: revision 1.2
share/locale/monetary/fr_CA.ISO8859-1.src: revision 1.2
share/locale/monetary/fr_FR.ISO8859-1.src: revision 1.2
share/locale/monetary/he_IL.UTF-8.src: revision 1.2
share/locale/monetary/hi_IN.ISCII-DEV.src: revision 1.2
share/locale/monetary/hr_HR.ISO8859-2.src: revision 1.2
share/locale/monetary/hu_HU.ISO8859-2.src: revision 1.2
share/locale/monetary/hy_AM.ARMSCII-8.src: revision 1.2
share/locale/monetary/hy_AM.UTF-8.src: revision 1.2
share/locale/monetary/is_IS.ISO8859-1.src: revision 1.2
share/locale/monetary/it_IT.ISO8859-1.src: revision 1.2
share/locale/monetary/ja_JP.ISO-2022-JP.src: revision 1.1
share/locale/monetary/ja_JP.UTF-8.src: revision 1.2
share/locale/monetary/ja_JP.eucJP.src: revision 1.2
share/locale/monetary/kk_KZ.PT154.src: revision 1.2
share/locale/monetary/kk_KZ.UTF-8.src: revision 1.2
share/locale/monetary/ko_KR.UTF-8.src: revision 1.2
share/locale/monetary/ko_KR.eucKR.src: revision 1.2
share/locale/monetary/lt_LT.ISO8859-13.src: revision 1.2
share/locale/monetary/mn_MN.UTF-8.src: revision 1.2
share/locale/monetary/nl_BE.ISO8859-1.src: revision 1.2
share/locale/monetary/nl_NL.ISO8859-1.src: revision 1.2
share/locale/monetary/no_NO.ISO8859-1.src: revision 1.2
share/locale/monetary/pl_PL.ISO8859-2.src: revision 1.2
share/locale/monetary/pl_PL.UTF-8.src: revision 1.2
share/locale/monetary/pt_BR.ISO8859-1.src: revision 1.2
share/locale/monetary/pt_PT.ISO8859-1.src: revision 1.2
share/locale/monetary/ro_RO.ISO8859-2.src: revision 1.2
share/locale/monetary/ru_RU.CP1251.src: revision 1.2
share/locale/monetary/ru_RU.CP866.src: revision 1.2
share/locale/monetary/ru_RU.ISO8859-5.src: revision 1.2
share/locale/monetary/ru_RU.KOI8-R.src: revision 1.2
share/locale/monetary/ru_RU.UTF-8.src: revision 1.2
share/locale/monetary/sk_SK.ISO8859-2.src: revision 1.2
share/locale/monetary/sl_SI.ISO8859-2.src: revision 1.2
share/locale/monetary/sr_ME.ISO8859-2.src: revision 1.1
share/locale/monetary/sr_RS.ISO8859-2.src: revision 1.1
share/locale/monetary/sr_YU.ISO8859-2.src: revision 1.2
share/locale/monetary/sr_YU.ISO8859-5.src: revision 1.2
share/locale/monetary/sr_YU.UTF-8.src: revision 1.2
share/locale/monetary/sv_SE.ISO8859-1.src: revision 1.2
share/locale/monetary/tr_TR.ISO8859-9.src: revision 1.2
share/locale/monetary/uk_UA.CP1251.src: revision 1.2
share/locale/monetary/uk_UA.ISO8859-5.src: revision 1.2
share/locale/monetary/uk_UA.KOI8-U.src: revision 1.2
share/locale/monetary/uk_UA.UTF-8.src: revision 1.2
share/locale/monetary/zh_CN.GB18030.src: revision 1.2
share/locale/monetary/zh_CN.UTF-8.src: revision 1.2
share/locale/monetary/zh_CN.eucCN.src: revision 1.2
share/locale/monetary/zh_HK.Big5hkscs.src: revision 1.1
share/locale/monetary/zh_HK.UTF-8.src: revision 1.2
share/locale/monetary/zh_TW.Big5.src: revision 1.2
share/locale/numeric/Makefile: revision 1.4
share/locale/numeric/am_ET.UTF-8.src: file removal
share/locale/numeric/en_US.ISO8859-1.src: file removal
share/locale/numeric/en_US.US-ASCII.src: revision 1.1
share/locale/numeric/ja_JP.eucJP.src: file removal
share/locale/numeric/ko_KR.eucKR.src: file removal
share/locale/numeric/mn_MN.UTF-8.src: file removal
share/locale/numeric/sr_ME.ISO8859-2.src: revision 1.1
share/locale/numeric/sr_YU.ISO8859-2.src: file removal
share/locale/numeric/sr_YU.ISO8859-5.src: file removal
share/locale/numeric/zh_CN.eucCN.src: file removal
share/locale/time/Makefile: revision 1.5
share/locale/time/en_US.ISO8859-1.src: file removal
share/locale/time/en_US.US-ASCII.src: revision 1.1
share/locale/time/ja_JP.ISO-2022-JP.src: revision 1.1
share/locale/time/ja_JP.ct.src: revision 1.1
share/locale/time/sr_ME.ISO8859-2.src: revision 1.1
share/locale/time/sr_ME.ISO8859-5.src: revision 1.1
share/locale/time/sr_ME.UTF-8.src: revision 1.1
share/locale/time/sr_YU.ISO8859-2.src: file removal
share/locale/time/sr_YU.ISO8859-5.src: file removal
share/locale/time/sr_YU.UTF-8.src: file removal
share/locale/time/zh_CN.GB18030.src: file removal
share/locale/time/zh_TW.eucTW.src: revision 1.1
usr.bin/locale/locale.c: revision 1.6
usr.bin/mklocale/Makefile: revision 1.12
usr.bin/mklocale/lex.l: revision 1.14
usr.bin/mklocale/mklocaledb.c: revision 1.1
usr.bin/mklocale/yacc.y: revision 1.25
usr.sbin/chrtbl/Makefile: revision 1.8
usr.sbin/chrtbl/ctypeio.c: revision 1.1
usr.sbin/chrtbl/ctypeio.h: revision 1.1
Fixes PR lib/39662, shortcomings in LC_{MONETARY,NUMERIC,TIME,MESSAGES}
db format.
ok'ed by core and releng.
(thanks for agc@, snj@ and i'm sorry for long time patience).
[libc]
- localeio.[ch] and lc*.[ch] in src/lib/libc/locale was replaced by
new locale-db implementation using citrus_db backend,
see src/lib/libc/citrus/citrus_lc_*.[ch].
- add citrus_bcs_strtou?l.c. don't use strtou?l locale implementation
internally, because they're locale-aware function.
- add some stubs for multi-locale issue, see {current,global}_locale.c.
- remove some obsolete file, setrunelocale.c, ___runetype_mb.c.
- remove __savectype() from ctypeio.[ch].
[tools]
- mklocale(1): add new option ``-t'' that generates new style
LC_{MONETARY,NUMERIC,TIME,MESSAGES} locale-db format.
- chrtbl(1): added ctypeio.[ch] for __savectype().
[locale-db]
- added en_US.US-ASCII locale.
- removed some shareable locale definition file:
en_US.US-ASCII -> en_US.ISO8859-1, en_US.UTF-8
zh_CN.eucCN -> zh_CN.GB18030
and more...see src/share/locale/*/Makefile.
- remove obsoleted locale sr_YU, added new locale sr_ME, sr_RS.
- change locale name ja_JP.ISO2022-JP* -> ja_JP.ISO-2022-JP*
for X11's locale.alias file alignments.
- fix regression test, wrong wcs?width(3), NAN/INF usage.

i tested release-build following arch:
i386, amd64, hpc{mips,arm,sh}, sparc64, vax.

citrus_lc_*.[ch] also can read old-plain-text style locale-db.
so that backward compatibility is keeped, but lc*.[ch] can't read
new citrus_db'ed locale-db and localeio.c never check sanity,
so forward compatibility is broken ;-<
old mklocale(1) doesn't know -t option, so you have to rebuild toolchain.
 1.14.14.1 23-Jun-2013  tls resync from head
 1.14.8.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.17.20.5 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.17.20.4 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.17.20.3 31-Jul-2017  perseant Make the compilation conditional on __STDC_ISO_10646__, so that applications
that need one of the East Asian locales but don't want __STDC_ISO_10646__
don't have to pay the penalty for it.
 1.17.20.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.17.20.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.2 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_euc.h was added on branch nathanw_sa on 2002-03-22 20:42:02 +0000
 1.1 21-Jul-2017  perseant branches: 1.1.2;
file citrus_euc_data.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.1 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.1 20-Jan-2018  perseant branches: 1.1.2;
file citrus_euc_k2u.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.2 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.1.2.1 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.1 20-Jan-2018  perseant branches: 1.1.2;
file citrus_euc_u2k.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.2 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.1.2.1 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.12 28-May-2013  joerg branches: 1.12.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.11 14-Jun-2008  tnozaki branches: 1.11.2; 1.11.6; 1.11.26;
remove unused include, locale.h.
 1.10 29-Oct-2005  tshiozak branches: 1.10.18; 1.10.20;
fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().
 1.9 27-Mar-2005  tnozaki add csmapper:CNS11643-1,2 and esdb:ISO-2022-CN,
integrate esdb:EUC-TW, locale:zh_TW.eucTW.
 1.8 14-Oct-2003  yamt don't use uninitialized variable ('chlenbak').
the problem reported by Markus Illenseer on tech-userlevel@
and analyzed by enami tsugutomo.
 1.7 26-Jun-2003  tshiozak correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.
 1.6 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.5 28-Mar-2002  yamt - move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.
 1.4 28-Mar-2002  yamt introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.
 1.3 27-Mar-2002  yamt fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.
 1.2 27-Mar-2002  yamt - change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.3 25-Apr-2002  nathanw Catch up to -current.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_euctw.c was added on branch nathanw_sa on 2002-03-22 20:42:02 +0000
 1.10.20.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.10.18.1 17-Jun-2008  yamt sync with head.
 1.11.26.1 23-Jun-2013  tls resync from head
 1.11.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.11.6.1 14-Jun-2008  tnozaki file citrus_euctw.c was added on branch christos-time_t on 2008-06-14 16:01:08 +0000
 1.11.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.12.22.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.12.22.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.2 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_euctw.h was added on branch nathanw_sa on 2002-03-22 20:42:02 +0000
 1.9 19-Apr-2022  rillig lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.
 1.8 28-May-2013  joerg branches: 1.8.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.7 14-Jun-2008  tnozaki branches: 1.7.2; 1.7.6; 1.7.26;
remove unused include, locale.h.
 1.6 15-Feb-2006  tnozaki branches: 1.6.18; 1.6.20;
MB_CUR_MAX should be 2 when MODE_2BYTE flag set.
 1.5 29-Oct-2005  tshiozak fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().
 1.4 26-Jun-2003  tshiozak correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.
 1.3 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.2 08-May-2003  petrov No need in default label if default case is not processed.
 1.1 25-Mar-2003  tshiozak add support for zh_CN.GB18030 locale.
(lib/20835 by sigsegv at s25 dot xrea dot com)
XXX: charclass for GBK compatible area is not supported.
 1.6.20.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.6.18.1 17-Jun-2008  yamt sync with head.
 1.7.26.1 23-Jun-2013  tls resync from head
 1.7.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.7.6.1 14-Jun-2008  tnozaki file citrus_gbk2k.c was added on branch christos-time_t on 2008-06-14 16:01:08 +0000
 1.7.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.8.22.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.8.22.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.2 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 25-Mar-2003  tshiozak add support for zh_CN.GB18030 locale.
(lib/20835 by sigsegv at s25 dot xrea dot com)
XXX: charclass for GBK compatible area is not supported.
 1.4 24-Jun-2014  spz branches: 1.4.18;
Fix our iconv version for the issues that apply to us from CVE-2014-3951
(which are the:
- Consistently pass around context information using a simple pointer.
This fixes some dereferencing bugs in Chinese character set conversions.
- Fix Simplified Chinese character set conversions by switching around the
fields of an internal struct so it corresponds with the way variables of
this type are initialised.
part)
Patch taken from FreeBSD and mutilated to fit.
FreeBSD credits: Manuel Mausz (reporter), Tijl Coosemans (report handler)
 1.3 28-May-2013  joerg branches: 1.3.4;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.2 14-Jun-2008  tnozaki branches: 1.2.2; 1.2.6; 1.2.26;
remove unused include, locale.h.
 1.1 22-Nov-2006  tnozaki branches: 1.1.16; 1.1.18;
1. add iconv support for following encodings:
Chinese Simplefied
HZ, HZ8 - 7/8bit stateful encoding, see RFC1842,1843. (libHZ)
zW - 7bit stateful encoding, see RFC1842. (libZW)
2. add citrus_prop.[ch] - parser for encoding module's init parameter strings.
 1.1.18.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.1.16.1 17-Jun-2008  yamt sync with head.
 1.2.26.2 20-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.2.26.1 23-Jun-2013  tls resync from head
 1.2.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.2.6.1 14-Jun-2008  tnozaki file citrus_hz.c was added on branch christos-time_t on 2008-06-14 16:01:08 +0000
 1.2.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.3.4.1 10-Aug-2014  tls Rebase.
 1.4.18.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.4.18.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.1 22-Nov-2006  tnozaki 1. add iconv support for following encodings:
Chinese Simplefied
HZ, HZ8 - 7/8bit stateful encoding, see RFC1842,1843. (libHZ)
zW - 7bit stateful encoding, see RFC1842. (libZW)
2. add citrus_prop.[ch] - parser for encoding module's init parameter strings.
 1.3 23-May-2011  joerg branches: 1.3.38;
Actually return something deterministic
 1.2 01-Jul-2003  tshiozak add iconv_t cache.
- separate shared part and context part from iconv_t.
- add cache of shared part.
and, bump shlib_version minor of i18n modules because iconv i/f is changed.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.3.38.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.3.38.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.16 12-Feb-2012  wiz branches: 1.16.34;
Remove unused variables. From cppcheck via Henning Petersen in PR 45997.
 1.15 13-Nov-2006  tnozaki branches: 1.15.42;
avoid infinity loop, iso2022 module's stdenc_get_state_desc_generic()
never return _STDENC_SDGEN_INITIAL.
 1.14 13-Nov-2006  tnozaki 1. add iconv support for followint encoding:
Vietnamese
TCVN 8bit Viet Nam National Standard
VISCII 8bit RFC1456
VIQR 7bit RFC1456(libVIQR)
Unicode Escape (GNU libiconv compatibility)
C99, JAVA (libUES)
2. fix iconv_std module:
add special treatment for POSIX Defect Report #288 case.
 1.13 19-Mar-2006  christos Coverity CID 2473: Fix uninitialized variable reference.
 1.12 09-Feb-2006  dogcow change #include <sys/endian.h> => #include <machine/endian.h> so that
it's (more) consistent in the tree; this, along with changing tools/compat's
autoconf detection from AC_CHECK_FUNCS to AC_CHECK_DECLS makes the vast
majority of htobe16 and friends' redefinition errors bite the dust.
Tested with -current and FreeBSD.
 1.11 29-Oct-2005  tshiozak fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().
 1.10 11-Feb-2005  simonb White space nit- don't put a space before/after increment/decrement
operators.
 1.9 21-Dec-2004  yamt constify.
 1.8 21-Dec-2004  yamt constify.
 1.7 29-Dec-2003  yamt _citrus_iconv_std_iconv_convert:
- allow 0 byte input. PR/23889 (Valeriy E. Ushakov)
- remove redundant E2BIG check, which should be done by
each encoding modules anyway.
 1.6 01-Sep-2003  tshiozak fix the problem about resetting the state of iconv().
iconv() was not conforming to the SUSv3 at this point.
reported by Bruno Haible.
 1.5 12-Jul-2003  tshiozak separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."
 1.4 01-Jul-2003  tshiozak add iconv_t cache.
- separate shared part and context part from iconv_t.
- add cache of shared part.
and, bump shlib_version minor of i18n modules because iconv i/f is changed.
 1.3 01-Jul-2003  tshiozak cosmetic change.
 1.2 26-Jun-2003  tshiozak correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.15.42.1 17-Apr-2012  yamt sync with head
 1.16.34.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.16.34.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.2 01-Jul-2003  tshiozak add iconv_t cache.
- separate shared part and context part from iconv_t.
- add cache of shared part.
and, bump shlib_version minor of i18n modules because iconv i/f is changed.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.23 28-May-2013  joerg branches: 1.23.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.22 10-Oct-2011  tnozaki branches: 1.22.2; 1.22.8;
revert r1.21, still problem exists for posix2008 mbsnrtowcs(not yet commited),
but i have no time to investigate t_mbrtowc failure.
 1.21 07-Oct-2011  tnozaki update string pointer when input is partial escape sequence or multibyte.
 1.20 07-Dec-2010  joerg Remove tautology.
 1.19 14-Jun-2008  tnozaki branches: 1.19.6;
remove unused include, locale.h.
 1.18 21-Nov-2007  tnozaki branches: 1.18.6; 1.18.8;
fix lib/37290
- don't call abort(3) when there's no suitable charset found.
- use iso-8859-1(or INIT1 if specified) for C1 control char.
 1.17 18-Sep-2007  tnozaki lib/36938 mbtowc misbehaving after invalid char sequence
- make sure to initialize mbtowc's internal state.
- add regression test.
 1.16 07-Jun-2006  tnozaki branches: 1.16.10;
if INIT0 specified, use it instead of ASCII.
 1.15 19-Mar-2006  christos Coverity CID 2464: Don't use uninitialized variables; exit with error quickly.
 1.14 29-Oct-2005  tshiozak fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().
 1.13 10-Feb-2005  tnozaki fix typo in _DIAGASSERT
 1.12 21-Dec-2004  yamt constify.
 1.11 02-Jan-2004  itojun comment on values from 0x80000000 to 0xffffffff
 1.10 02-Jan-2004  itojun safer realloc
 1.9 16-Jul-2003  itojun use bounded string op
 1.8 26-Jun-2003  tshiozak correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.
 1.7 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.6 28-Mar-2002  yamt - move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.
 1.5 28-Mar-2002  yamt introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.
 1.4 28-Mar-2002  yamt drop __packed__ attribute from _ISO2022State.
 1.3 27-Mar-2002  yamt fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.
 1.2 27-Mar-2002  yamt - change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.3 25-Apr-2002  nathanw Catch up to -current.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_iso2022.c was added on branch nathanw_sa on 2002-03-22 20:42:03 +0000
 1.16.10.2 09-Jan-2008  matt sync with HEAD
 1.16.10.1 06-Nov-2007  matt sync with HEAD
 1.18.8.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.18.6.1 17-Jun-2008  yamt sync with head.
 1.19.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.19.6.1 14-Jun-2008  tnozaki file citrus_iso2022.c was added on branch christos-time_t on 2008-06-14 16:01:08 +0000
 1.22.8.1 23-Jun-2013  tls resync from head
 1.22.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.23.22.5 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.23.22.4 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.23.22.3 31-Jul-2017  perseant Make the compilation conditional on __STDC_ISO_10646__, so that applications
that need one of the East Asian locales but don't want __STDC_ISO_10646__
don't have to pay the penalty for it.
 1.23.22.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.23.22.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.2 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_iso2022.h was added on branch nathanw_sa on 2002-03-22 20:42:03 +0000
 1.1 21-Jul-2017  perseant branches: 1.1.2;
file citrus_iso2022_data.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.1 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.1 20-Jan-2018  perseant branches: 1.1.2;
file citrus_iso2022_k2u.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.2 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.1.2.1 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.1 20-Jan-2018  perseant branches: 1.1.2;
file citrus_iso2022_u2k.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.2 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.1.2.1 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.7 08-Jul-2019  mlelstv The correct fix for lib/51000 is not to remove the apparently dead assignment
but to add the missing usage.

From https://bitbucket.org/tnozaki/netbsd/commits/24b35fdce250
Thanks to Takehiko Nozaki
 1.6 31-May-2016  dholland branches: 1.6.10; 1.6.16;
PR 51000 David Binderman: comment out unused assignment
 1.5 28-May-2013  joerg Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.4 14-Jun-2008  tnozaki branches: 1.4.2; 1.4.6; 1.4.26;
remove unused include, locale.h.
 1.3 23-Oct-2007  tnozaki branches: 1.3.6; 1.3.8;
remove invalid range check.
 1.2 13-Dec-2006  tnozaki branches: 1.2.4;
make del(\x7f) pass through.
 1.1 18-Oct-2006  tnozaki add new iconv module libJOHAB,
this supports S.Korean character encoding scheme ``JOHAB''.
 1.2.4.1 06-Nov-2007  matt sync with HEAD
 1.3.8.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.3.6.1 17-Jun-2008  yamt sync with head.
 1.4.26.1 23-Jun-2013  tls resync from head
 1.4.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.4.6.1 14-Jun-2008  tnozaki file citrus_johab.c was added on branch christos-time_t on 2008-06-14 16:01:08 +0000
 1.4.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.16.1 13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.6.10.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.6.10.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.1 18-Oct-2006  tnozaki add new iconv module libJOHAB,
this supports S.Korean character encoding scheme ``JOHAB''.
 1.4 14-Jul-2003  tshiozak use NONIDENTICAL instead of ILSEQ if the reverse conversion of ISO646.
(fix for lib/22135)
 1.3 12-Jul-2003  tshiozak separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."
 1.2 27-Jun-2003  tshiozak sync with the last change for citrus_hash.h.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.2 27-Jun-2003  tshiozak sync with the last change for citrus_hash.h.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.2 12-Jul-2003  tshiozak separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.13 26-Feb-2025  andvar Fix typos in comments, mainly s/calcurate/calculate/.
 1.12 11-Jun-2018  maya branches: 1.12.14;
Avoid left-shifting a negative number in the non-compat case, too.
 1.11 11-Jun-2018  kamil Correct Undefined Behavior in libc/citrus

Unportable left shift reported with MKSANITIZER=yes USE_SANITIZER=undefined:

# nm /usr/lib/libc.so|grep sanit
/public/src.git/lib/libc/citrus/modules/citrus_mapper_std.c:173:8: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'

Sponsored by <The NetBSD Foundation>
 1.10 19-Nov-2011  tnozaki branches: 1.10.38;
fix memory leak, pointed by nonaka-san(again^3).
 1.9 30-Oct-2011  wiz branches: 1.9.2;
Use boolean AND instead of bitwise one in _DIAGASSERT.
From Henning Petersen in PR 45518.
 1.8 11-Sep-2006  tnozaki correct typo in _DIAGASSERT() and some KNF.
pointed by uebayashi-san, thanks!
 1.7 09-Sep-2006  tnozaki mapper_std iconv module and mkcsmapper(1) now can treat
plain-row-col charset and 4byte code(like GB18030) as SRC_ZONE.
 1.6 19-Mar-2006  christos Coveriry CID 2472: If the number of bits is invalid, return immediately
instead of accessing uninitialized variables.
 1.5 09-Feb-2006  dogcow change #include <sys/endian.h> => #include <machine/endian.h> so that
it's (more) consistent in the tree; this, along with changing tools/compat's
autoconf detection from AC_CHECK_FUNCS to AC_CHECK_DECLS makes the vast
majority of htobe16 and friends' redefinition errors bite the dust.
Tested with -current and FreeBSD.
 1.4 21-Dec-2004  yamt constify.
 1.3 12-Jul-2003  tshiozak separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."
 1.2 27-Jun-2003  tshiozak sync with the last change for citrus_hash.h.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.9.2.1 17-Apr-2012  yamt sync with head
 1.10.38.1 25-Jun-2018  pgoyette Sync with HEAD
 1.12.14.1 02-Aug-2025  perseant Sync with HEAD
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.3 09-Sep-2006  tnozaki mapper_std iconv module and mkcsmapper(1) now can treat
plain-row-col charset and 4byte code(like GB18030) as SRC_ZONE.
 1.2 12-Jul-2003  tshiozak separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.3 09-Sep-2006  tnozaki mapper_std iconv module and mkcsmapper(1) now can treat
plain-row-col charset and 4byte code(like GB18030) as SRC_ZONE.
 1.2 12-Jul-2003  tshiozak separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.4 12-Jul-2003  tshiozak separate "nonidentical conversion" and "ilseq" under csmapper.
"nonidentical" means "the input character index is valid, but not
corresponding to the output codeset."
"ilseq" means "the input character index is invalid for the input codeset."
 1.3 01-Jul-2003  tshiozak fix lib/22023 - there were references to uninitialized variables.
 1.2 27-Jun-2003  tshiozak sync with the last change for citrus_hash.h.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.15 19-Apr-2022  rillig lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.
 1.14 28-May-2013  joerg branches: 1.14.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.13 14-Jun-2008  tnozaki branches: 1.13.2; 1.13.6; 1.13.26;
remove unused include, locale.h.
 1.12 05-Mar-2007  tnozaki branches: 1.12.10; 1.12.12;
iconv: add following CCS/CES support.
- CNS11643-[3-7] <-> UCS:BMP/SIP (EUC-TW, ISO-2022-CN-EXT)
- HKSCS <-> UCS:BMP/SIP (Big5-HKSCS)
- JISX0213-[1-2] <-> UCS:BMP/SIP (EUC-JIS-2004,Shift_JIS-2004,ISO-2022-JP-2004)
 1.11 07-Dec-2005  tshiozak add missing _citrus_MSKanji_stdenc_get_state_desc_generic() function.
pointed out by Patrick Welche <prlw1 _at_ newn _dot_ cam _dot_ ac _dot_ uk>
 1.10 18-Oct-2005  tshiozak make sure that this module can handle all private/vendor-defined
character area.
This is reported by MORIYAMA Masayuki <msyk _at_ mtg.biglobe.ne.jp> and
"NARUSE, Yui" <naruse _at_ airemix.com>, and fixed by MORIYAMA-san.
 1.9 10-Feb-2005  tnozaki fix typo in _DIAGASSERT
 1.8 29-Dec-2003  yamt _citrus_MSKanji_wcrtomb_priv: return E2BIG correctly.
 1.7 26-Jun-2003  tshiozak correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.
 1.6 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.5 28-Mar-2002  yamt - move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.
 1.4 28-Mar-2002  yamt introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.
 1.3 27-Mar-2002  yamt fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.
 1.2 27-Mar-2002  yamt - change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.3 25-Apr-2002  nathanw Catch up to -current.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_mskanji.c was added on branch nathanw_sa on 2002-03-22 20:42:03 +0000
 1.12.12.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.12.10.1 17-Jun-2008  yamt sync with head.
 1.13.26.1 23-Jun-2013  tls resync from head
 1.13.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.13.6.1 14-Jun-2008  tnozaki file citrus_mskanji.c was added on branch christos-time_t on 2008-06-14 16:01:09 +0000
 1.13.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.14.22.5 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.14.22.4 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.14.22.3 31-Jul-2017  perseant Make the compilation conditional on __STDC_ISO_10646__, so that applications
that need one of the East Asian locales but don't want __STDC_ISO_10646__
don't have to pay the penalty for it.
 1.14.22.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.14.22.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.2 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_mskanji.h was added on branch nathanw_sa on 2002-03-22 20:42:03 +0000
 1.1 21-Jul-2017  perseant branches: 1.1.2;
file citrus_mskanji_data.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.1 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.1 20-Jan-2018  perseant branches: 1.1.2;
file citrus_mskanji_k2u.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.2 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.1.2.1 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.1 20-Jan-2018  perseant branches: 1.1.2;
file citrus_mskanji_u2k.h was initially added on branch perseant-stdc-iso10646.
 1.1.2.2 29-Jan-2018  perseant Fix issues with trie implementation. Now passes the locale tests when __STDC_ISO_10646__ is defined, with the exception of tests requiring LC_COLLATE support.
 1.1.2.1 20-Jan-2018  perseant Use multilevel tables (tries) to convert kuten <-> unicode.
Get rid of dchains, replace with chain/rchain conversions through keys
allocated from the PUA.
 1.5 19-Apr-2022  rillig lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.
 1.4 28-May-2013  joerg branches: 1.4.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.3 12-Feb-2012  wiz branches: 1.3.6;
Remove unused variables. From cppcheck via Henning Petersen in PR 45997.
 1.2 07-Dec-2010  joerg branches: 1.2.6;
Mark function as static and give it an explicit return type.
 1.1 13-Nov-2006  tnozaki 1. add iconv support for followint encoding:
Vietnamese
TCVN 8bit Viet Nam National Standard
VISCII 8bit RFC1456
VIQR 7bit RFC1456(libVIQR)
Unicode Escape (GNU libiconv compatibility)
C99, JAVA (libUES)
2. fix iconv_std module:
add special treatment for POSIX Defect Report #288 case.
 1.2.6.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.2.6.1 17-Apr-2012  yamt sync with head
 1.3.6.1 23-Jun-2013  tls resync from head
 1.4.22.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.4.22.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.1 13-Nov-2006  tnozaki 1. add iconv support for followint encoding:
Vietnamese
TCVN 8bit Viet Nam National Standard
VISCII 8bit RFC1456
VIQR 7bit RFC1456(libVIQR)
Unicode Escape (GNU libiconv compatibility)
C99, JAVA (libUES)
2. fix iconv_std module:
add special treatment for POSIX Defect Report #288 case.
 1.13 19-Apr-2022  rillig lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.
 1.12 12-Feb-2012  wiz branches: 1.12.34;
Remove unused variables. From cppcheck via Henning Petersen in PR 45997.
 1.11 20-Mar-2010  tnozaki branches: 1.11.6;
fix byte order mark related bug introduced by previous commit,
reported by Sverre Froyen via current-user, thanks!
 1.10 15-Mar-2010  tnozaki 1. fix wrong byte order mark of utf-16, reported by NARUSE Yui -san.
patch provided by tshiozak@ -san.

2. don't eat 0xfeff/0xfffe if they don't appear at the first of bytestream.
noticed y tshiozak@ -san, patch provied by me.

thanks a lot.
 1.9 14-Jun-2008  tnozaki branches: 1.9.4; 1.9.6;
remove unused include, locale.h.
 1.8 20-Mar-2008  tnozaki branches: 1.8.2; 1.8.4;
add BOM to utf-16/32 stream in case that endian is not specified,
like other iconv implementation, GNU libiconv, glibc2 iconv, perl iconv.
 1.7 27-Oct-2006  tnozaki branches: 1.7.8; 1.7.14;
don't pass through surrogate character(0xD800 - 0xDFFF).
 1.6 19-Mar-2006  christos Coverity 2462: Bail out quickly instead of accessing uninitialized variables.
 1.5 09-Feb-2006  dogcow change #include <sys/endian.h> => #include <machine/endian.h> so that
it's (more) consistent in the tree; this, along with changing tools/compat's
autoconf detection from AC_CHECK_FUNCS to AC_CHECK_DECLS makes the vast
majority of htobe16 and friends' redefinition errors bite the dust.
Tested with -current and FreeBSD.
 1.4 29-Oct-2005  tshiozak fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().
 1.3 27-Jun-2003  yamt $NetBSD $ -> $NetBSD$
 1.2 26-Jun-2003  tshiozak correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.7.14.1 24-Mar-2008  keiichi sync with head.
 1.7.8.1 23-Mar-2008  matt sync with HEAD
 1.8.4.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.8.2.1 17-Jun-2008  yamt sync with head.
 1.9.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.9.6.1 14-Jun-2008  tnozaki file citrus_utf1632.c was added on branch christos-time_t on 2008-06-14 16:01:09 +0000
 1.9.4.1 19-Nov-2010  riz Pull up following revision(s) (requested by tnozaki in ticket #1356):
lib/libc/citrus/modules/citrus_utf1632.c: revision 1.10
lib/libc/citrus/modules/citrus_utf1632.c: revision 1.11
fix byte order mark related bug introduced by previous commit,
reported by Sverre Froyen via current-user, thanks!
1. fix wrong byte order mark of utf-16, reported by NARUSE Yui -san.
patch provided by tshiozak@ -san.
2. don't eat 0xfeff/0xfffe if they don't appear at the first of bytestream.
noticed y tshiozak@ -san, patch provied by me.
thanks a lot.
 1.11.6.1 17-Apr-2012  yamt sync with head
 1.12.34.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.12.34.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.1 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.7 19-Apr-2022  rillig lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.
 1.6 28-May-2013  joerg branches: 1.6.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.5 23-Aug-2006  tnozaki branches: 1.5.42; 1.5.48;
cleanup code
 1.4 19-Mar-2006  christos Coverity CID 1439: Prevent array index out of bounds access.
 1.3 29-Oct-2005  tshiozak fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().
 1.2 14-Mar-2005  tnozaki anonymous union between chlen and _UTF7StatePrive has
compilation problem with gcc295.
this union attempt to make mbsinit(3) handle multibyte state correctly,
but it's useless as far as we use utf-7 only iconv interface.
so i eliminate ctype feature.

patch contributed by Joerg Sonnenberg(who porting Citrus to DragonFlyBSD).
and yamt-san gave me advice, thanks a lot.
 1.1 05-Mar-2005  tnozaki add new citrus iconv module UTF-7.
thanks advice, yamt-san.
 1.5.48.1 23-Jun-2013  tls resync from head
 1.5.42.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.6.22.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.6.22.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.1 05-Mar-2005  tnozaki add new citrus iconv module UTF-7.
thanks advice, yamt-san.
 1.18 28-May-2013  joerg branches: 1.18.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.17 14-Jun-2008  tnozaki branches: 1.17.2; 1.17.6; 1.17.26;
remove unused include, locale.h.
 1.16 06-Mar-2007  tnozaki branches: 1.16.10; 1.16.12;
disallow illegal utf-8 byte sequence and surrogate chars (RFC3629).
5-6 byte sequence(0x110000 - 0x7FFFFFFF) are still available
for backward compatibility.
 1.15 22-Mar-2006  christos Coverity CID 1440: off by one in array count.
 1.14 19-Mar-2006  christos Coverity CID 2463: Bail out instead of accessing uninitialized variables.
 1.13 29-Oct-2005  tshiozak fix lib/31874.
- add _citrus_stdenc_get_state_desc() interface to get
encoding-scheme-independent encoder/decoder state descriptions.
- make sure that iconv_std module uses it to judge whether the last
sequences forms complete shift sequences.
- bump minor of i18n_module because of get_state_desc().
 1.12 11-Mar-2005  christos UTF8EncodingInfo is an empty struct; remove noop code and DIAGASSERT.
From Joerg Sonnenberger
 1.11 21-Dec-2004  yamt constify.
 1.10 07-Aug-2003  agc Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.
 1.9 26-Jun-2003  tshiozak correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.
 1.8 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.7 25-Mar-2003  tshiozak fix the problem if EILSEQ is caused in mbrtowc(3) under *.UTF-8 locales.
(lib/20873 by khym at azeotrope dot org)
 1.6 28-Mar-2002  yamt - move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.
 1.5 28-Mar-2002  yamt introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.
 1.4 27-Mar-2002  yamt fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.
 1.3 27-Mar-2002  yamt - change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
use _ENCODING_MB_CUR_MAX instead.
 1.2 18-Mar-2002  yamt branches: 1.2.2;
don't check variable s0 before assign.
 1.1 17-Mar-2002  tshiozak refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.2.2.3 25-Apr-2002  nathanw Catch up to -current.
 1.2.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.2.2.1 18-Mar-2002  nathanw file citrus_utf8.c was added on branch nathanw_sa on 2002-03-22 20:42:03 +0000
 1.16.12.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.16.10.1 17-Jun-2008  yamt sync with head.
 1.17.26.1 23-Jun-2013  tls resync from head
 1.17.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.17.6.1 14-Jun-2008  tnozaki file citrus_utf8.c was added on branch christos-time_t on 2008-06-14 16:01:09 +0000
 1.17.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.18.22.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.18.22.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.2 25-Jun-2003  tshiozak Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
- iconv core (citrus_iconv)
- coded character set mapper (csmapper)
- encoding scheme database (esdb)
- standard encoding scheme handler (stdenc)
- basic character set manipulation utilities (bcs)
- mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
- some source codes of new plugins (not built yet)
 1.1 17-Mar-2002  tshiozak branches: 1.1.2;
refine i18n stuffs.
- add libc/citrus directory.
this directory contains the common stuffs not only for locale
but also other facilities relating to i18n (e.g. iconv).
- To keep the binary compatibility for the ctype modules easily,
the interface between libc and the modules is simplified.
- For the future integrated extension, module pool is renamed
from "/usr/lib/runemodule" to "/usr/lib/i18n".
In the future, this directory will contain the modules for "iconv",
"collation", etc.
- some cosmetic changes.
- Bug fix for runetype.h; __attribute__((__packed__)) is placed at
the wrong position and it is invalid unintentionally.
But, the all members of the structures seem well-aligned. Thus,
this bug causes no problem, hopefully.

HEADS UP:
- /usr/lib/rumemodule is obsoleted. If you use the multibyte locales,
you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
- The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
Perhaps, the bug fix about __packed__ mentioned above breaks the
compatibility... Be careful especially on 64bit platforms.
 1.1.2.2 22-Mar-2002  nathanw Catch up to -current.
 1.1.2.1 17-Mar-2002  nathanw file citrus_utf8.h was added on branch nathanw_sa on 2002-03-22 20:42:03 +0000
 1.6 28-May-2013  joerg branches: 1.6.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.5 19-Nov-2011  tnozaki branches: 1.5.6;
remove unused variable, pointed by nonaka-san, thanks.
 1.4 14-Jun-2008  tnozaki branches: 1.4.2; 1.4.6;
remove unused include, locale.h.
 1.3 22-Nov-2006  tnozaki branches: 1.3.16; 1.3.18;
fix memory leak.
 1.2 14-Nov-2006  dogcow change uint32_t to size_t; fixes build issue on 64-bit platforms.
 1.1 13-Nov-2006  tnozaki 1. add iconv support for followint encoding:
Vietnamese
TCVN 8bit Viet Nam National Standard
VISCII 8bit RFC1456
VIQR 7bit RFC1456(libVIQR)
Unicode Escape (GNU libiconv compatibility)
C99, JAVA (libUES)
2. fix iconv_std module:
add special treatment for POSIX Defect Report #288 case.
 1.3.18.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.3.16.1 17-Jun-2008  yamt sync with head.
 1.4.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.4.6.1 14-Jun-2008  tnozaki file citrus_viqr.c was added on branch christos-time_t on 2008-06-14 16:01:09 +0000
 1.4.2.2 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.4.2.1 17-Apr-2012  yamt sync with head
 1.5.6.1 23-Jun-2013  tls resync from head
 1.6.22.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.6.22.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.1 13-Nov-2006  tnozaki 1. add iconv support for followint encoding:
Vietnamese
TCVN 8bit Viet Nam National Standard
VISCII 8bit RFC1456
VIQR 7bit RFC1456(libVIQR)
Unicode Escape (GNU libiconv compatibility)
C99, JAVA (libUES)
2. fix iconv_std module:
add special treatment for POSIX Defect Report #288 case.
 1.6 19-Apr-2022  rillig lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.
 1.5 28-May-2013  joerg branches: 1.5.22;
Add mbsnrtowcs and wcsnrtombs. Approved by core.
 1.4 14-Jun-2008  tnozaki branches: 1.4.2; 1.4.6; 1.4.26;
remove unused include, locale.h.
 1.3 24-Nov-2006  tnozaki branches: 1.3.16; 1.3.18;
don't read input string more than MB_LEN_MAX(maybe redundant escape sequence).
 1.2 24-Nov-2006  tnozaki don't throw EILSEQ when byte sequence is "zW ".
 1.1 22-Nov-2006  tnozaki 1. add iconv support for following encodings:
Chinese Simplefied
HZ, HZ8 - 7/8bit stateful encoding, see RFC1842,1843. (libHZ)
zW - 7bit stateful encoding, see RFC1842. (libZW)
2. add citrus_prop.[ch] - parser for encoding module's init parameter strings.
 1.3.18.1 23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.3.16.1 17-Jun-2008  yamt sync with head.
 1.4.26.1 23-Jun-2013  tls resync from head
 1.4.6.2 14-Jun-2008  tnozaki remove unused include, locale.h.
 1.4.6.1 14-Jun-2008  tnozaki file citrus_zw.c was added on branch christos-time_t on 2008-06-14 16:01:09 +0000
 1.4.2.1 22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.5.22.2 21-Jul-2017  perseant Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.
 1.5.22.1 14-Jul-2017  perseant Initial commit of a mostly-working implementation of __STDC_ISO_10646__,
with collation support using the Unicode Collation Algorithm.

The conversion from men/ku/ten form to Unicode is a gross hack at present.
Fixing this, and fleshing out the LC_COLLATE locale component, are next
on the agenda.
 1.1 22-Nov-2006  tnozaki 1. add iconv support for following encodings:
Chinese Simplefied
HZ, HZ8 - 7/8bit stateful encoding, see RFC1842,1843. (libHZ)
zW - 7bit stateful encoding, see RFC1842. (libZW)
2. add citrus_prop.[ch] - parser for encoding module's init parameter strings.

RSS XML Feed