Home | History | Annotate | Line # | Download | only in dist
      1 
      2 XZ Utils
      3 ========
      4 
      5     0. Overview
      6     1. Documentation
      7        1.1. Overall documentation
      8        1.2. Documentation for command-line tools
      9        1.3. Documentation for liblzma
     10     2. Version numbering
     11     3. Reporting bugs
     12     4. Translating the xz tool
     13     5. Other implementations of the .xz format
     14     6. Contact information
     15 
     16 
     17 0. Overview
     18 -----------
     19 
     20     XZ Utils provide a general-purpose data-compression library plus
     21     command-line tools. The native file format is the .xz format, but
     22     also the legacy .lzma format is supported. The .xz format supports
     23     multiple compression algorithms, which are called "filters" in the
     24     context of XZ Utils. The primary filter is currently LZMA2. With
     25     typical files, XZ Utils create about 30 % smaller files than gzip.
     26 
     27     To ease adapting support for the .xz format into existing applications
     28     and scripts, the API of liblzma is somewhat similar to the API of the
     29     popular zlib library. For the same reason, the command-line tool xz
     30     has a command-line syntax similar to that of gzip.
     31 
     32     When aiming for the highest compression ratio, the LZMA2 encoder uses
     33     a lot of CPU time and may use, depending on the settings, even
     34     hundreds of megabytes of RAM. However, in fast modes, the LZMA2 encoder
     35     competes with bzip2 in compression speed, RAM usage, and compression
     36     ratio.
     37 
     38     LZMA2 is reasonably fast to decompress. It is a little slower than
     39     gzip, but a lot faster than bzip2. Being fast to decompress means
     40     that the .xz format is especially nice when the same file will be
     41     decompressed very many times (usually on different computers), which
     42     is the case e.g. when distributing software packages. In such
     43     situations, it's not too bad if the compression takes some time,
     44     since that needs to be done only once to benefit many people.
     45 
     46     With some file types, combining (or "chaining") LZMA2 with an
     47     additional filter can improve the compression ratio. A filter chain may
     48     contain up to four filters, although usually only one or two are used.
     49     For example, putting a BCJ (Branch/Call/Jump) filter before LZMA2
     50     in the filter chain can improve compression ratio of executable files.
     51 
     52     Since the .xz format allows adding new filter IDs, it is possible that
     53     some day there will be a filter that is, for example, much faster to
     54     compress than LZMA2 (but probably with worse compression ratio).
     55     Similarly, it is possible that some day there is a filter that will
     56     compress better than LZMA2.
     57 
     58     XZ Utils doesn't support multithreaded compression or decompression
     59     yet. It has been planned though and taken into account when designing
     60     the .xz file format.
     61 
     62 
     63 1. Documentation
     64 ----------------
     65 
     66 1.1. Overall documentation
     67 
     68     README              This file
     69 
     70     INSTALL.generic     Generic install instructions for those not familiar
     71                         with packages using GNU Autotools
     72     INSTALL             Installation instructions specific to XZ Utils
     73     PACKAGERS           Information to packagers of XZ Utils
     74 
     75     COPYING             XZ Utils copyright and license information
     76     COPYING.GPLv2       GNU General Public License version 2
     77     COPYING.GPLv3       GNU General Public License version 3
     78     COPYING.LGPLv2.1    GNU Lesser General Public License version 2.1
     79 
     80     AUTHORS             The main authors of XZ Utils
     81     THANKS              Incomplete list of people who have helped making
     82                         this software
     83     NEWS                User-visible changes between XZ Utils releases
     84     ChangeLog           Detailed list of changes (commit log)
     85     TODO                Known bugs and some sort of to-do list
     86 
     87     Note that only some of the above files are included in binary
     88     packages.
     89 
     90 
     91 1.2. Documentation for command-line tools
     92 
     93     The command-line tools are documented as man pages. In source code
     94     releases (and possibly also in some binary packages), the man pages
     95     are also provided in plain text (ASCII only) and PDF formats in the
     96     directory "doc/man" to make the man pages more accessible to those
     97     whose operating system doesn't provide an easy way to view man pages.
     98 
     99 
    100 1.3. Documentation for liblzma
    101 
    102     The liblzma API headers include short docs about each function
    103     and data type as Doxygen tags. These docs should be quite OK as
    104     a quick reference.
    105 
    106     I have planned to write a bunch of very well documented example
    107     programs, which (due to comments) should work as a tutorial to
    108     various features of liblzma. No such example programs have been
    109     written yet.
    110 
    111     For now, if you have never used liblzma, libbzip2, or zlib, I
    112     recommend learning the *basics* of the zlib API. Once you know that,
    113     it should be easier to learn liblzma.
    114 
    115         http://zlib.net/manual.html
    116         http://zlib.net/zlib_how.html
    117 
    118 
    119 2. Version numbering
    120 --------------------
    121 
    122     The version number format of XZ Utils is X.Y.ZS:
    123 
    124       - X is the major version. When this is incremented, the library
    125         API and ABI break.
    126 
    127       - Y is the minor version. It is incremented when new features
    128         are added without breaking the existing API or ABI. An even Y
    129         indicates a stable release and an odd Y indicates unstable
    130         (alpha or beta version).
    131 
    132       - Z is the revision. This has a different meaning for stable and
    133         unstable releases:
    134 
    135           * Stable: Z is incremented when bugs get fixed without adding
    136             any new features. This is intended to be convenient for
    137             downstream distributors that want bug fixes but don't want
    138             any new features to minimize the risk of introducing new bugs.
    139 
    140           * Unstable: Z is just a counter. API or ABI of features added
    141             in earlier unstable releases having the same X.Y may break.
    142 
    143       - S indicates stability of the release. It is missing from the
    144         stable releases, where Y is an even number. When Y is odd, S
    145         is either "alpha" or "beta" to make it very clear that such
    146         versions are not stable releases. The same X.Y.Z combination is
    147         not used for more than one stability level, i.e. after X.Y.Zalpha,
    148         the next version can be X.Y.(Z+1)beta but not X.Y.Zbeta.
    149 
    150 
    151 3. Reporting bugs
    152 -----------------
    153 
    154     Naturally it is easiest for me if you already know what causes the
    155     unexpected behavior. Even better if you have a patch to propose.
    156     However, quite often the reason for unexpected behavior is unknown,
    157     so here are a few things to do before sending a bug report:
    158 
    159       1. Try to create a small example how to reproduce the issue.
    160 
    161       2. Compile XZ Utils with debugging code using configure switches
    162          --enable-debug and, if possible, --disable-shared. If you are
    163          using GCC, use CFLAGS='-O0 -ggdb3'. Don't strip the resulting
    164          binaries.
    165 
    166       3. Turn on core dumps. The exact command depends on your shell;
    167          for example in GNU bash it is done with "ulimit -c unlimited",
    168          and in tcsh with "limit coredumpsize unlimited".
    169 
    170       4. Try to reproduce the suspected bug. If you get "assertion failed"
    171          message, be sure to include the complete message in your bug
    172          report. If the application leaves a coredump, get a backtrace
    173          using gdb:
    174            $ gdb /path/to/app-binary   # Load the app to the debugger.
    175            (gdb) core core   # Open the coredump.
    176            (gdb) bt   # Print the backtrace. Copy & paste to bug report.
    177            (gdb) quit   # Quit gdb.
    178 
    179     Report your bug via email or IRC (see Contact information below).
    180     Don't send core dump files or any executables. If you have a small
    181     example file(s) (total size less than 256 KiB), please include
    182     it/them as an attachment. If you have bigger test files, put them
    183     online somewhere and include a URL to the file(s) in the bug report.
    184 
    185     Always include the exact version number of XZ Utils in the bug report.
    186     If you are using a snapshot from the git repository, use "git describe"
    187     to get the exact snapshot version. If you are using XZ Utils shipped
    188     in an operating system distribution, mention the distribution name,
    189     distribution version, and exact xz package version; if you cannot
    190     repeat the bug with the code compiled from unpatched source code,
    191     you probably need to report a bug to your distribution's bug tracking
    192     system.
    193 
    194 
    195 4. Translating the xz tool
    196 --------------------------
    197 
    198     The messages from the xz tool have been translated into a few
    199     languages. Before starting to translate into a new language, ask
    200     the author whether someone else hasn't already started working on it.
    201 
    202     Test your translation. Testing includes comparing the translated
    203     output to the original English version by running the same commands
    204     in both your target locale and with LC_ALL=C. Ask someone to
    205     proof-read and test the translation.
    206 
    207     Testing can be done e.g. by installing xz into a temporary directory:
    208 
    209         ./configure --disable-shared --prefix=/tmp/xz-test
    210         # <Edit the .po file in the po directory.>
    211         make -C po update-po
    212         make install
    213         bash debug/translation.bash | less
    214         bash debug/translation.bash | less -S  # For --list outputs
    215 
    216     Repeat the above as needed (no need to re-run configure though).
    217 
    218     Note especially the following:
    219 
    220       - The output of --help and --long-help must look nice on
    221         an 80-column terminal. It's OK to add extra lines if needed.
    222 
    223       - In contrast, don't add extra lines to error messages and such.
    224         They are often preceded with e.g. a filename on the same line,
    225         so you have no way to predict where to put a \n. Let the terminal
    226         do the wrapping even if it looks ugly. Adding new lines will be
    227         even uglier in the generic case even if it looks nice in a few
    228         limited examples.
    229 
    230       - Be careful with column alignment in tables and table-like output
    231         (--list, --list --verbose --verbose, --info-memory, --help, and
    232         --long-help):
    233 
    234           * All descriptions of options in --help should start in the
    235             same column (but it doesn't need to be the same column as
    236             in the English messages; just be consistent if you change it).
    237             Check that both --help and --long-help look OK, since they
    238             share several strings.
    239 
    240           * --list --verbose and --info-memory print lines that have
    241             the format "Description:   %s". If you need a longer
    242             description, you can put extra space between the colon
    243             and %s. Then you may need to add extra space to other
    244             strings too so that the result as a whole looks good (all
    245             values start at the same column).
    246 
    247           * The columns of the actual tables in --list --verbose --verbose
    248             should be aligned properly. Abbreviate if necessary. It might
    249             be good to keep at least 2 or 3 spaces between column headings
    250             and avoid spaces in the headings so that the columns stand out
    251             better, but this is a matter of opinion. Do what you think
    252             looks best.
    253 
    254       - Be careful to put a period at the end of a sentence when the
    255         original version has it, and don't put it when the original
    256         doesn't have it. Similarly, be careful with \n characters
    257         at the beginning and end of the strings.
    258 
    259       - Read the TRANSLATORS comments that have been extracted from the
    260         source code and included in xz.pot. If they suggest testing the
    261         translation with some type of command, do it. If testing needs
    262         input files, use e.g. tests/files/good-*.xz.
    263 
    264       - When updating the translation, read the fuzzy (modified) strings
    265         carefully, and don't mark them as updated before you actually
    266         have updated them. Reading through the unchanged messages can be
    267         good too; sometimes you may find a better wording for them.
    268 
    269       - If you find language problems in the original English strings,
    270         feel free to suggest improvements. Ask if something is unclear.
    271 
    272       - The translated messages should be understandable (sometimes this
    273         may be a problem with the original English messages too). Don't
    274         make a direct word-by-word translation from English especially if
    275         the result doesn't sound good in your language.
    276 
    277     In short, take your time and pay attention to the details. Making
    278     a good translation is not a quick and trivial thing to do. The
    279     translated xz should look as polished as the English version.
    280 
    281 
    282 5. Other implementations of the .xz format
    283 ------------------------------------------
    284 
    285     7-Zip and the p7zip port of 7-Zip support the .xz format starting
    286     from the version 9.00alpha.
    287 
    288         http://7-zip.org/
    289         http://p7zip.sourceforge.net/
    290 
    291     XZ Embedded is a limited implementation written for use in the Linux
    292     kernel, but it is also suitable for other embedded use.
    293 
    294         https://tukaani.org/xz/embedded.html
    295 
    296 
    297 6. Contact information
    298 ----------------------
    299 
    300     If you have questions, bug reports, patches etc. related to XZ Utils,
    301     contact Lasse Collin <lasse.collin (a] tukaani.org> (in Finnish or English).
    302     I'm sometimes slow at replying. If you haven't got a reply within two
    303     weeks, assume that your email has got lost and resend it or use IRC.
    304 
    305     You can find me also from #tukaani on Freenode; my nick is Larhzu.
    306     The channel tends to be pretty quiet, so just ask your question and
    307     someone may wake up.
    308 
    309