README revision 1.1.1.3
1 1.1 mrg This directory contains a mechanism for GCC to have its own internal 2 1.1.1.3 mrg implementation of wcwidth functionality (cpp_wcwidth () in libcpp/charset.c), 3 1.1.1.3 mrg as well as a mechanism to update the information about codepoints permitted in 4 1.1.1.3 mrg identifiers, which is encoded in libcpp/ucnid.h, and mapping between Unicode 5 1.1.1.3 mrg names and codepoints, which is encoded in libcpp/uname2c.h. 6 1.1.1.3 mrg 7 1.1.1.3 mrg The idea is to produce the necessary lookup tables 8 1.1.1.3 mrg (../../libcpp/{ucnid.h,uname2c.h,generated_cpp_wcwidth.h}) in a reproducible 9 1.1.1.3 mrg way, starting from the following files that are distributed by the Unicode 10 1.1.1.3 mrg Consortium: 11 1.1 mrg 12 1.1 mrg ftp://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt 13 1.1 mrg ftp://ftp.unicode.org/Public/UNIDATA/EastAsianWidth.txt 14 1.1 mrg ftp://ftp.unicode.org/Public/UNIDATA/PropList.txt 15 1.1.1.3 mrg ftp://ftp.unicode.org/Public/UNIDATA/DerivedNormalizationProps.txt 16 1.1.1.3 mrg ftp://ftp.unicode.org/Public/UNIDATA/DerivedCoreProperties.txt 17 1.1.1.3 mrg ftp://ftp.unicode.org/Public/UNIDATA/NameAliases.txt 18 1.1.1.3 mrg 19 1.1.1.3 mrg Two additional files are needed for lookup tables in libstdc++: 20 1.1.1.3 mrg 21 1.1.1.3 mrg ftp://ftp.unicode.org/Public/UNIDATA/auxiliary/GraphemeBreakProperty.txt 22 1.1.1.3 mrg ftp://ftp.unicode.org/Public/UNIDATA/emoji/emoji-data.txt 23 1.1 mrg 24 1.1.1.3 mrg All these files have been added to source control in this directory; 25 1.1 mrg please see unicode-license.txt for the relevant copyright information. 26 1.1 mrg 27 1.1 mrg In order to keep in sync with glibc's wcwidth as much as possible, it is 28 1.1 mrg desirable for the logic that processes the Unicode data to be the same as 29 1.1 mrg glibc's. To that end, we also put in this directory, in the from_glibc/ 30 1.1 mrg directory, the glibc python code that implements their logic. This code was 31 1.1 mrg copied verbatim from glibc, and it can be updated at any time from the glibc 32 1.1.1.3 mrg source code repository. The files copied from that repository are: 33 1.1 mrg 34 1.1 mrg localedata/unicode-gen/unicode_utils.py 35 1.1 mrg localedata/unicode-gen/utf8_gen.py 36 1.1 mrg 37 1.1 mrg And the most recent versions added to GCC are from glibc git commit: 38 1.1.1.3 mrg 71de3aead9fffe89556e80ebc94aa918d8ee7bca 39 1.1 mrg 40 1.1.1.3 mrg The script gen_wcwidth.py found here contains the GCC-specific code to 41 1.1 mrg map glibc's output to the lookup tables we require. This script should not need 42 1.1 mrg to change, unless there are structural changes to the Unicode data files or to 43 1.1.1.3 mrg the glibc code. Similarly, makeucnid.cc in ../../libcpp contains the logic to 44 1.1.1.3 mrg produce ucnid.h. 45 1.1 mrg 46 1.1.1.3 mrg The procedure to update GCC's Unicode support is the following: 47 1.1 mrg 48 1.1.1.3 mrg 1. Update the six Unicode data files from the above URLs. 49 1.1 mrg 50 1.1 mrg 2. Update the two glibc files in from_glibc/ from glibc's git. Update 51 1.1 mrg the commit number above in this README. 52 1.1 mrg 53 1.1 mrg 3. Run ./gen_wcwidth.py X.Y > ../../libcpp/generated_cpp_wcwidth.h 54 1.1 mrg (where X.Y is the version of the Unicode standard corresponding to the 55 1.1.1.3 mrg Unicode data files being used, most recently, 15.1.0). 56 1.1.1.3 mrg 57 1.1.1.3 mrg 4. Update Unicode Copyright years in libcpp/makeucnid.cc and in 58 1.1.1.3 mrg libcpp/makeuname2c.cc up to the year in which the Unicode 59 1.1.1.3 mrg standard has been released. 60 1.1.1.3 mrg 61 1.1.1.3 mrg 5. Compile makeucnid, e.g. with: 62 1.1.1.3 mrg g++ -O2 ../../libcpp/makeucnid.cc -o ../../libcpp/makeucnid 63 1.1.1.3 mrg 64 1.1.1.3 mrg 6. Generate ucnid.h as follows: 65 1.1.1.3 mrg ../../libcpp/makeucnid ../../libcpp/ucnid.tab UnicodeData.txt \ 66 1.1.1.3 mrg DerivedNormalizationProps.txt DerivedCoreProperties.txt \ 67 1.1.1.3 mrg > ../../libcpp/ucnid.h 68 1.1.1.3 mrg 69 1.1.1.3 mrg 7. Read the corresponding Unicode's standard and update correspondingly 70 1.1.1.3 mrg generated_ranges table in libcpp/makeuname2c.cc (in Unicode 15 all 71 1.1.1.3 mrg the needed information was in Table 4-8). 72 1.1.1.3 mrg 73 1.1.1.3 mrg 8. Compile makeuname2c, e.g. with: 74 1.1.1.3 mrg g++ -O2 ../../libcpp/makeuname2c.cc -o ../../libcpp/makeuname2c 75 1.1.1.3 mrg 76 1.1.1.3 mrg 9: Generate uname2c.h as follows: 77 1.1.1.3 mrg ../../libcpp/makeuname2c UnicodeData.txt NameAliases.txt \ 78 1.1.1.3 mrg > ../../libcpp/uname2c.h 79 1.1 mrg 80 1.1.1.3 mrg See gen_libstdcxx_unicode_data.py for instructions on updating the lookup 81 1.1.1.3 mrg tables in libstdc++. 82
Indexes created Fri Aug 14 00:25:39 UTC 2026