1 Command Line Interface for Zstandard library 2 ============================================ 3 4 Command Line Interface (CLI) can be created using the `make` command without any additional parameters. 5 There are however other Makefile targets that create different variations of CLI: 6 - `zstd` : default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and supports decompression of legacy zstd formats 7 - `zstd_nolegacy` : Same as `zstd` but without support for legacy zstd formats 8 - `zstd-small` : CLI optimized for minimal size; no dictionary builder, no benchmark, and no support for legacy zstd formats 9 - `zstd-compress` : version of CLI which can only compress into zstd format 10 - `zstd-decompress` : version of CLI which can only decompress zstd format 11 12 13 ### Compilation variables 14 `zstd` scope can be altered by modifying the following `make` variables : 15 16 - __HAVE_THREAD__ : multithreading is automatically enabled when `pthread` is detected. 17 It's possible to disable multithread support, by setting `HAVE_THREAD=0`. 18 Example : `make zstd HAVE_THREAD=0` 19 It's also possible to force multithread support, using `HAVE_THREAD=1`. 20 In which case, linking stage will fail if neither `pthread` nor `windows.h` library can be found. 21 This is useful to ensure this feature is not silently disabled. 22 23 - __ZSTD_LEGACY_SUPPORT__ : `zstd` can decompress files compressed by older versions of `zstd`. 24 Starting v0.8.0, all versions of `zstd` produce frames compliant with the [specification](../doc/zstd_compression_format.md), and are therefore compatible. 25 But older versions (< v0.8.0) produced different, incompatible, frames. 26 By default, `zstd` supports decoding legacy formats >= v0.4.0 (`ZSTD_LEGACY_SUPPORT=4`). 27 This can be altered by modifying this compilation variable. 28 `ZSTD_LEGACY_SUPPORT=1` means "support all formats >= v0.1.0". 29 `ZSTD_LEGACY_SUPPORT=2` means "support all formats >= v0.2.0", and so on. 30 `ZSTD_LEGACY_SUPPORT=0` means _DO NOT_ support any legacy format. 31 if `ZSTD_LEGACY_SUPPORT >= 8`, it's the same as `0`, since there is no legacy format after `7`. 32 Note : `zstd` only supports decoding older formats, and cannot generate any legacy format. 33 34 - __HAVE_ZLIB__ : `zstd` can compress and decompress files in `.gz` format. 35 This is ordered through command `--format=gzip`. 36 Alternatively, symlinks named `gzip` or `gunzip` will mimic intended behavior. 37 `.gz` support is automatically enabled when `zlib` library is detected at build time. 38 It's possible to disable `.gz` support, by setting `HAVE_ZLIB=0`. 39 Example : `make zstd HAVE_ZLIB=0` 40 It's also possible to force compilation with zlib support, using `HAVE_ZLIB=1`. 41 In which case, linking stage will fail if `zlib` library cannot be found. 42 This is useful to prevent silent feature disabling. 43 44 - __HAVE_LZMA__ : `zstd` can compress and decompress files in `.xz` and `.lzma` formats. 45 This is ordered through commands `--format=xz` and `--format=lzma` respectively. 46 Alternatively, symlinks named `xz`, `unxz`, `lzma`, or `unlzma` will mimic intended behavior. 47 `.xz` and `.lzma` support is automatically enabled when `lzma` library is detected at build time. 48 It's possible to disable `.xz` and `.lzma` support, by setting `HAVE_LZMA=0`. 49 Example : `make zstd HAVE_LZMA=0` 50 It's also possible to force compilation with lzma support, using `HAVE_LZMA=1`. 51 In which case, linking stage will fail if `lzma` library cannot be found. 52 This is useful to prevent silent feature disabling. 53 54 - __HAVE_LZ4__ : `zstd` can compress and decompress files in `.lz4` formats. 55 This is ordered through commands `--format=lz4`. 56 Alternatively, symlinks named `lz4`, or `unlz4` will mimic intended behavior. 57 `.lz4` support is automatically enabled when `lz4` library is detected at build time. 58 It's possible to disable `.lz4` support, by setting `HAVE_LZ4=0` . 59 Example : `make zstd HAVE_LZ4=0` 60 It's also possible to force compilation with lz4 support, using `HAVE_LZ4=1`. 61 In which case, linking stage will fail if `lz4` library cannot be found. 62 This is useful to prevent silent feature disabling. 63 64 - __ZSTD_NOBENCH__ : `zstd` cli will be compiled without its integrated benchmark module. 65 This can be useful to produce smaller binaries. 66 In this case, the corresponding unit can also be excluded from compilation target. 67 68 - __ZSTD_NODICT__ : `zstd` cli will be compiled without support for the integrated dictionary builder. 69 This can be useful to produce smaller binaries. 70 In this case, the corresponding unit can also be excluded from compilation target. 71 72 - __ZSTD_NOCOMPRESS__ : `zstd` cli will be compiled without support for compression. 73 The resulting binary will only be able to decompress files. 74 This can be useful to produce smaller binaries. 75 A corresponding `Makefile` target using this ability is `zstd-decompress`. 76 77 - __ZSTD_NODECOMPRESS__ : `zstd` cli will be compiled without support for decompression. 78 The resulting binary will only be able to compress files. 79 This can be useful to produce smaller binaries. 80 A corresponding `Makefile` target using this ability is `zstd-compress`. 81 82 - __BACKTRACE__ : `zstd` can display a stack backtrace when execution 83 generates a runtime exception. By default, this feature may be 84 degraded/disabled on some platforms unless additional compiler directives are 85 applied. When triaging a runtime issue, enabling this feature can provide 86 more context to determine the location of the fault. 87 Example : `make zstd BACKTRACE=1` 88 89 90 ### Aggregation of parameters 91 CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`. 92 93 94 ### Symlink shortcuts 95 It's possible to invoke `zstd` through a symlink. 96 When the name of the symlink has a specific value, it triggers an associated behavior. 97 - `zstdmt` : compress using all cores available on local system. 98 - `zcat` : will decompress and output target file using any of the supported formats. `gzcat` and `zstdcat` are also equivalent. 99 - `gzip` : if zlib support is enabled, will mimic `gzip` by compressing file using `.gz` format, removing source file by default (use `--keep` to preserve). If zlib is not supported, triggers an error. 100 - `xz` : if lzma support is enabled, will mimic `xz` by compressing file using `.xz` format, removing source file by default (use `--keep` to preserve). If xz is not supported, triggers an error. 101 - `lzma` : if lzma support is enabled, will mimic `lzma` by compressing file using `.lzma` format, removing source file by default (use `--keep` to preserve). If lzma is not supported, triggers an error. 102 - `lz4` : if lz4 support is enabled, will mimic `lz4` by compressing file using `.lz4` format. If lz4 is not supported, triggers an error. 103 - `unzstd` and `unlz4` will decompress any of the supported format. 104 - `ungz`, `unxz` and `unlzma` will do the same, and will also remove source file by default (use `--keep` to preserve). 105 106 107 ### Dictionary builder in Command Line Interface 108 Zstd offers a training mode, which can be used to tune the algorithm for a selected 109 type of data, by providing it with a few samples. The result of the training is stored 110 in a file selected with the `-o` option (default name is `dictionary`), 111 which can be loaded before compression and decompression. 112 113 Using a dictionary, the compression ratio achievable on small data improves dramatically. 114 These compression gains are achieved while simultaneously providing faster compression and decompression speeds. 115 Dictionary work if there is some correlation in a family of small data (there is no universal dictionary). 116 Hence, deploying one dictionary per type of data will provide the greater benefits. 117 Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm 118 will rely more and more on previously decoded content to compress the rest of the file. 119 120 Usage of the dictionary builder and created dictionaries with CLI: 121 122 1. Create the dictionary : `zstd --train PathToTrainingSet/* -o dictionaryName` 123 2. Compress with the dictionary: `zstd FILE -D dictionaryName` 124 3. Decompress with the dictionary: `zstd --decompress FILE.zst -D dictionaryName` 125 126 127 ### Benchmark in Command Line Interface 128 CLI includes in-memory compression benchmark module for zstd. 129 The benchmark is conducted using given filenames. The files are read into memory and joined together. 130 It makes benchmark more precise as it eliminates I/O overhead. 131 Multiple filenames can be supplied, as multiple parameters, with wildcards, 132 or directory names can be used with `-r` option. 133 If no file is provided, the benchmark will use a procedurally generated "lorem ipsum" content. 134 135 The benchmark measures ratio, compressed size, compression and decompression speed. 136 One can select compression levels starting from `-b` and ending with `-e`. 137 The `-i` parameter selects minimal time used for each of tested levels. 138 139 The benchmark can also be used to test specific parameters, 140 such as number of threads (`-T#`), or advanced parameters (`--zstd=#`), or dictionary compression (`-D DICTIONARY`), 141 and many others available on command for regular compression and decompression. 142 143 144 ### Usage of Command Line Interface 145 The full list of options can be obtained with `-h` or `-H` parameter: 146 ``` 147 *** Zstandard CLI (64-bit) v1.5.6, by Yann Collet *** 148 149 Compress or decompress the INPUT file(s); reads from STDIN if INPUT is `-` or not provided. 150 151 Usage: zstd [OPTIONS...] [INPUT... | -] [-o OUTPUT] 152 153 Options: 154 -o OUTPUT Write output to a single file, OUTPUT. 155 -k, --keep Preserve INPUT file(s). [Default] 156 --rm Remove INPUT file(s) after successful (de)compression. 157 158 -# Desired compression level, where `#` is a number between 1 and 19; 159 lower numbers provide faster compression, higher numbers yield 160 better compression ratios. [Default: 3] 161 162 -d, --decompress Perform decompression. 163 -D DICT Use DICT as the dictionary for compression or decompression. 164 165 -f, --force Disable input and output checks. Allows overwriting existing files, 166 receiving input from the console, printing output to STDOUT, and 167 operating on links, block devices, etc. Unrecognized formats will be 168 passed-through through as-is. 169 170 -h Display short usage and exit. 171 -H, --help Display full help and exit. 172 -V, --version Display the program version and exit. 173 174 Advanced options: 175 -c, --stdout Write to STDOUT (even if it is a console) and keep the INPUT file(s). 176 177 -v, --verbose Enable verbose output; pass multiple times to increase verbosity. 178 -q, --quiet Suppress warnings; pass twice to suppress errors. 179 --trace LOG Log tracing information to LOG. 180 181 --[no-]progress Forcibly show/hide the progress counter. NOTE: Any (de)compressed 182 output to terminal will mix with progress counter text. 183 184 -r Operate recursively on directories. 185 --filelist LIST Read a list of files to operate on from LIST. 186 --output-dir-flat DIR Store processed files in DIR. 187 --output-dir-mirror DIR Store processed files in DIR, respecting original directory structure. 188 --[no-]asyncio Use asynchronous IO. [Default: Enabled] 189 190 --[no-]check Add XXH64 integrity checksums during compression. [Default: Add, Validate] 191 If `-d` is present, ignore/validate checksums during decompression. 192 193 -- Treat remaining arguments after `--` as files. 194 195 Advanced compression options: 196 --ultra Enable levels beyond 19, up to 22; requires more memory. 197 --fast[=#] Use to very fast compression levels. [Default: 1] 198 --adapt Dynamically adapt compression level to I/O conditions. 199 --long[=#] Enable long distance matching with window log #. [Default: 27] 200 --patch-from=REF Use REF as the reference point for Zstandard's diff engine. 201 202 -T# Spawn # compression threads. [Default: 1; pass 0 for core count.] 203 --single-thread Share a single thread for I/O and compression (slightly different than `-T1`). 204 --auto-threads={physical|logical} 205 Use physical/logical cores when using `-T0`. [Default: Physical] 206 207 -B# Set job size to #. [Default: 0 (automatic)] 208 --rsyncable Compress using a rsync-friendly method (`-B` sets block size). 209 210 --exclude-compressed Only compress files that are not already compressed. 211 212 --stream-size=# Specify size of streaming input from STDIN. 213 --size-hint=# Optimize compression parameters for streaming input of approximately size #. 214 --target-compressed-block-size=# 215 Generate compressed blocks of approximately # size. 216 217 --no-dictID Don't write `dictID` into the header (dictionary compression only). 218 --[no-]compress-literals Force (un)compressed literals. 219 --[no-]row-match-finder Explicitly enable/disable the fast, row-based matchfinder for 220 the 'greedy', 'lazy', and 'lazy2' strategies. 221 222 --format=zstd Compress files to the `.zst` format. [Default] 223 --[no-]mmap-dict Memory-map dictionary file rather than mallocing and loading all at once 224 --format=gzip Compress files to the `.gz` format. 225 --format=xz Compress files to the `.xz` format. 226 --format=lzma Compress files to the `.lzma` format. 227 --format=lz4 Compress files to the `.lz4` format. 228 229 Advanced decompression options: 230 -l Print information about Zstandard-compressed files. 231 --test Test compressed file integrity. 232 -M# Set the memory usage limit to # megabytes. 233 --[no-]sparse Enable sparse mode. [Default: Enabled for files, disabled for STDOUT.] 234 --[no-]pass-through Pass through uncompressed files as-is. [Default: Disabled] 235 236 Dictionary builder: 237 --train Create a dictionary from a training set of files. 238 239 --train-cover[=k=#,d=#,steps=#,split=#,shrink[=#]] 240 Use the cover algorithm (with optional arguments). 241 --train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#,shrink[=#]] 242 Use the fast cover algorithm (with optional arguments). 243 244 --train-legacy[=s=#] Use the legacy algorithm with selectivity #. [Default: 9] 245 -o NAME Use NAME as dictionary name. [Default: dictionary] 246 --maxdict=# Limit dictionary to specified size #. [Default: 112640] 247 --dictID=# Force dictionary ID to #. [Default: Random] 248 249 Benchmark options: 250 -b# Perform benchmarking with compression level #. [Default: 3] 251 -e# Test all compression levels up to #; starting level is `-b#`. [Default: 1] 252 -i# Set the minimum evaluation to time # seconds. [Default: 3] 253 -B# Cut file into independent chunks of size #. [Default: No chunking] 254 -S Output one benchmark result per input file. [Default: Consolidated result] 255 -D dictionary Benchmark using dictionary 256 --priority=rt Set process priority to real-time. 257 ``` 258 259 ### Passing parameters through Environment Variables 260 There is no "generic" way to pass "any kind of parameter" to `zstd` in a pass-through manner. 261 Using environment variables for this purpose has security implications. 262 Therefore, this avenue is intentionally restricted and only supports `ZSTD_CLEVEL` and `ZSTD_NBTHREADS`. 263 264 `ZSTD_CLEVEL` can be used to modify the default compression level of `zstd` 265 (usually set to `3`) to another value between 1 and 19 (the "normal" range). 266 267 `ZSTD_NBTHREADS` can be used to specify a number of threads 268 that `zstd` will use for compression, which by default is `1`. 269 This functionality only exists when `zstd` is compiled with multithread support. 270 `0` means "use as many threads as detected cpu cores on local system". 271 The max # of threads is capped at `ZSTDMT_NBWORKERS_MAX`, 272 which is either 64 in 32-bit mode, or 256 for 64-bit environments. 273 274 This functionality can be useful when `zstd` CLI is invoked in a way that doesn't allow passing arguments. 275 One such scenario is `tar --zstd`. 276 As `ZSTD_CLEVEL` and `ZSTD_NBTHREADS` only replace the default compression level 277 and number of threads respectively, they can both be overridden by corresponding command line arguments: 278 `-#` for compression level and `-T#` for number of threads. 279 280 281 ### Long distance matching mode 282 The long distance matching mode, enabled with `--long`, is designed to improve 283 the compression ratio for files with long matches at a large distance (up to the 284 maximum window size, `128 MiB`) while still maintaining compression speed. 285 286 Enabling this mode sets the window size to `128 MiB` and thus increases the memory 287 usage for both the compressor and decompressor. Performance in terms of speed is 288 dependent on long matches being found. Compression speed may degrade if few long 289 matches are found. Decompression speed usually improves when there are many long 290 distance matches. 291 292 Below are graphs comparing the compression speed, compression ratio, and 293 decompression speed with and without long distance matching on an ideal use 294 case: a tar of four versions of clang (versions `3.4.1`, `3.4.2`, `3.5.0`, 295 `3.5.1`) with a total size of `244889600 B`. This is an ideal use case as there 296 are many long distance matches within the maximum window size of `128 MiB` (each 297 version is less than `128 MiB`). 298 299 Compression Speed vs Ratio | Decompression Speed 300 ---------------------------|--------------------- 301  |  302 303 | Method | Compression ratio | Compression speed | Decompression speed | 304 |:-------|------------------:|-------------------------:|---------------------------:| 305 | `zstd -1` | `5.065` | `284.8 MB/s` | `759.3 MB/s` | 306 | `zstd -5` | `5.826` | `124.9 MB/s` | `674.0 MB/s` | 307 | `zstd -10` | `6.504` | `29.5 MB/s` | `771.3 MB/s` | 308 | `zstd -1 --long` | `17.426` | `220.6 MB/s` | `1638.4 MB/s` | 309 | `zstd -5 --long` | `19.661` | `165.5 MB/s` | `1530.6 MB/s` | 310 | `zstd -10 --long`| `21.949` | `75.6 MB/s` | `1632.6 MB/s` | 311 312 On this file, the compression ratio improves significantly with minimal impact 313 on compression speed, and the decompression speed doubles. 314 315 On the other extreme, compressing a file with few long distance matches (such as 316 the [Silesia compression corpus]) will likely lead to a deterioration in 317 compression speed (for lower levels) with minimal change in compression ratio. 318 319 The below table illustrates this on the [Silesia compression corpus]. 320 321 [Silesia compression corpus]: https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia 322 323 | Method | Compression ratio | Compression speed | Decompression speed | 324 |:-------|------------------:|------------------:|---------------------:| 325 | `zstd -1` | `2.878` | `231.7 MB/s` | `594.4 MB/s` | 326 | `zstd -1 --long` | `2.929` | `106.5 MB/s` | `517.9 MB/s` | 327 | `zstd -5` | `3.274` | `77.1 MB/s` | `464.2 MB/s` | 328 | `zstd -5 --long` | `3.319` | `51.7 MB/s` | `371.9 MB/s` | 329 | `zstd -10` | `3.523` | `16.4 MB/s` | `489.2 MB/s` | 330 | `zstd -10 --long`| `3.566` | `16.2 MB/s` | `415.7 MB/s` | 331 332 333 ### zstdgrep 334 335 `zstdgrep` is a utility which makes it possible to `grep` directly a `.zst` compressed file. 336 It's used the same way as normal `grep`, for example : 337 `zstdgrep pattern file.zst` 338 339 `zstdgrep` is _not_ compatible with dictionary compression. 340 341 To search into a file compressed with a dictionary, 342 it's necessary to decompress it using `zstd` or `zstdcat`, 343 and then pipe the result to `grep`. For example : 344 `zstdcat -D dictionary -qc -- file.zst | grep pattern` 345
Indexes created Sun Aug 02 00:25:04 UTC 2026