1 <html> 2 <head> 3 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> 4 <title>bzip2 and libbzip2, version 1.0.8</title> 5 <meta name="generator" content="DocBook XSL Stylesheets V1.78.1"> 6 <link rel="stylesheet" type="text/css" href="bzip.css" /> 7 <style type="text/css" media="screen">/* Colours: 8 #74240f dark brown h1, h2, h3, h4 9 #336699 medium blue links 10 #339999 turquoise link hover colour 11 #202020 almost black general text 12 #761596 purple md5sum text 13 #626262 dark gray pre border 14 #eeeeee very light gray pre background 15 #f2f2f9 very light blue nav table background 16 #3366cc medium blue nav table border 17 */ 18 19 a, a:link, a:visited, a:active { color: #336699; } 20 a:hover { color: #339999; } 21 22 body { font: 80%/126% sans-serif; } 23 h1, h2, h3, h4 { color: #74240f; } 24 25 dt { color: #336699; font-weight: bold } 26 dd { 27 margin-left: 1.5em; 28 padding-bottom: 0.8em; 29 } 30 31 /* -- ruler -- */ 32 div.hr_blue { 33 height: 3px; 34 background:#ffffff url("../images/hr_blue.png") repeat-x; } 35 div.hr_blue hr { display:none; } 36 37 /* release styles */ 38 #release p { margin-top: 0.4em; } 39 #release .md5sum { color: #761596; } 40 41 42 /* ------ styles for docs|manuals|howto ------ */ 43 /* -- lists -- */ 44 ul { 45 margin: 0px 4px 16px 16px; 46 padding: 0px; 47 list-style: url("../images/li-blue.png"); 48 } 49 ul li { 50 margin-bottom: 10px; 51 } 52 ul ul { 53 list-style-type: none; 54 list-style-image: none; 55 margin-left: 0px; 56 } 57 58 /* header / footer nav tables */ 59 table.nav { 60 border: solid 1px #3366cc; 61 background: #f2f2f9; 62 background-color: #f2f2f9; 63 margin-bottom: 0.5em; 64 } 65 /* don't have underlined links in chunked nav menus */ 66 table.nav a { text-decoration: none; } 67 table.nav a:hover { text-decoration: underline; } 68 table.nav td { font-size: 85%; } 69 70 code, tt, pre { font-size: 120%; } 71 code, tt { color: #761596; } 72 73 div.literallayout, pre.programlisting, pre.screen { 74 color: #000000; 75 padding: 0.5em; 76 background: #eeeeee; 77 border: 1px solid #626262; 78 background-color: #eeeeee; 79 margin: 4px 0px 4px 0px; 80 } 81 </style> 82 </head> 83 <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div lang="en" class="book"> 84 <div class="titlepage"> 85 <div> 86 <div><h1 class="title"> 87 <a name="userman"></a>bzip2 and libbzip2, version 1.0.8</h1></div> 88 <div><h2 class="subtitle">A program and library for data compression</h2></div> 89 <div><div class="authorgroup"><div class="author"> 90 <h3 class="author"> 91 <span class="firstname">Julian</span> <span class="surname">Seward</span> 92 </h3> 93 <div class="affiliation"><span class="orgname">https://sourceware.org/bzip2/<br></span></div> 94 </div></div></div> 95 <div><p class="releaseinfo">Version 1.0.8 of 13 July 2019</p></div> 96 <div><p class="copyright">Copyright 1996-2019 Julian Seward</p></div> 97 <div><div class="legalnotice"> 98 <a name="legal"></a><p>This program, <code class="computeroutput">bzip2</code>, the 99 associated library <code class="computeroutput">libbzip2</code>, and 100 all documentation, are copyright 1996-2019 Julian Seward. 101 All rights reserved.</p> 102 <p>Redistribution and use in source and binary forms, with 103 or without modification, are permitted provided that the 104 following conditions are met:</p> 105 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: bullet; "> 106 <li class="listitem" style="list-style-type: disc"><p>Redistributions of source code must retain the 107 above copyright notice, this list of conditions and the 108 following disclaimer.</p></li> 109 <li class="listitem" style="list-style-type: disc"><p>The origin of this software must not be 110 misrepresented; you must not claim that you wrote the original 111 software. If you use this software in a product, an 112 acknowledgment in the product documentation would be 113 appreciated but is not required.</p></li> 114 <li class="listitem" style="list-style-type: disc"><p>Altered source versions must be plainly marked 115 as such, and must not be misrepresented as being the original 116 software.</p></li> 117 <li class="listitem" style="list-style-type: disc"><p>The name of the author may not be used to 118 endorse or promote products derived from this software without 119 specific prior written permission.</p></li> 120 </ul></div> 121 <p>THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY 122 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, 123 THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 124 PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 125 AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 126 EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED 127 TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 128 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 129 ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 130 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 131 IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 132 THE POSSIBILITY OF SUCH DAMAGE.</p> 133 <p>PATENTS: To the best of my knowledge, 134 <code class="computeroutput">bzip2</code> and 135 <code class="computeroutput">libbzip2</code> do not use any patented 136 algorithms. However, I do not have the resources to carry 137 out a patent search. Therefore I cannot give any guarantee of 138 the above statement. 139 </p> 140 </div></div> 141 </div> 142 <hr> 143 </div> 144 <div class="toc"> 145 <p><b>Table of Contents</b></p> 146 <dl class="toc"> 147 <dt><span class="chapter"><a href="#intro">1. Introduction</a></span></dt> 148 <dt><span class="chapter"><a href="#using">2. How to use bzip2</a></span></dt> 149 <dd><dl> 150 <dt><span class="sect1"><a href="#name">2.1. NAME</a></span></dt> 151 <dt><span class="sect1"><a href="#synopsis">2.2. SYNOPSIS</a></span></dt> 152 <dt><span class="sect1"><a href="#description">2.3. DESCRIPTION</a></span></dt> 153 <dt><span class="sect1"><a href="#options">2.4. OPTIONS</a></span></dt> 154 <dt><span class="sect1"><a href="#memory-management">2.5. MEMORY MANAGEMENT</a></span></dt> 155 <dt><span class="sect1"><a href="#recovering">2.6. RECOVERING DATA FROM DAMAGED FILES</a></span></dt> 156 <dt><span class="sect1"><a href="#performance">2.7. PERFORMANCE NOTES</a></span></dt> 157 <dt><span class="sect1"><a href="#caveats">2.8. CAVEATS</a></span></dt> 158 <dt><span class="sect1"><a href="#author">2.9. AUTHOR</a></span></dt> 159 </dl></dd> 160 <dt><span class="chapter"><a href="#libprog">3. 161 Programming with <code class="computeroutput">libbzip2</code> 162 </a></span></dt> 163 <dd><dl> 164 <dt><span class="sect1"><a href="#top-level">3.1. Top-level structure</a></span></dt> 165 <dd><dl> 166 <dt><span class="sect2"><a href="#ll-summary">3.1.1. Low-level summary</a></span></dt> 167 <dt><span class="sect2"><a href="#hl-summary">3.1.2. High-level summary</a></span></dt> 168 <dt><span class="sect2"><a href="#util-fns-summary">3.1.3. Utility functions summary</a></span></dt> 169 </dl></dd> 170 <dt><span class="sect1"><a href="#err-handling">3.2. Error handling</a></span></dt> 171 <dt><span class="sect1"><a href="#low-level">3.3. Low-level interface</a></span></dt> 172 <dd><dl> 173 <dt><span class="sect2"><a href="#bzcompress-init">3.3.1. BZ2_bzCompressInit</a></span></dt> 174 <dt><span class="sect2"><a href="#bzCompress">3.3.2. BZ2_bzCompress</a></span></dt> 175 <dt><span class="sect2"><a href="#bzCompress-end">3.3.3. BZ2_bzCompressEnd</a></span></dt> 176 <dt><span class="sect2"><a href="#bzDecompress-init">3.3.4. BZ2_bzDecompressInit</a></span></dt> 177 <dt><span class="sect2"><a href="#bzDecompress">3.3.5. BZ2_bzDecompress</a></span></dt> 178 <dt><span class="sect2"><a href="#bzDecompress-end">3.3.6. BZ2_bzDecompressEnd</a></span></dt> 179 </dl></dd> 180 <dt><span class="sect1"><a href="#hl-interface">3.4. High-level interface</a></span></dt> 181 <dd><dl> 182 <dt><span class="sect2"><a href="#bzreadopen">3.4.1. BZ2_bzReadOpen</a></span></dt> 183 <dt><span class="sect2"><a href="#bzread">3.4.2. BZ2_bzRead</a></span></dt> 184 <dt><span class="sect2"><a href="#bzreadgetunused">3.4.3. BZ2_bzReadGetUnused</a></span></dt> 185 <dt><span class="sect2"><a href="#bzreadclose">3.4.4. BZ2_bzReadClose</a></span></dt> 186 <dt><span class="sect2"><a href="#bzwriteopen">3.4.5. BZ2_bzWriteOpen</a></span></dt> 187 <dt><span class="sect2"><a href="#bzwrite">3.4.6. BZ2_bzWrite</a></span></dt> 188 <dt><span class="sect2"><a href="#bzwriteclose">3.4.7. BZ2_bzWriteClose</a></span></dt> 189 <dt><span class="sect2"><a href="#embed">3.4.8. Handling embedded compressed data streams</a></span></dt> 190 <dt><span class="sect2"><a href="#std-rdwr">3.4.9. Standard file-reading/writing code</a></span></dt> 191 </dl></dd> 192 <dt><span class="sect1"><a href="#util-fns">3.5. Utility functions</a></span></dt> 193 <dd><dl> 194 <dt><span class="sect2"><a href="#bzbufftobuffcompress">3.5.1. BZ2_bzBuffToBuffCompress</a></span></dt> 195 <dt><span class="sect2"><a href="#bzbufftobuffdecompress">3.5.2. BZ2_bzBuffToBuffDecompress</a></span></dt> 196 </dl></dd> 197 <dt><span class="sect1"><a href="#zlib-compat">3.6. zlib compatibility functions</a></span></dt> 198 <dt><span class="sect1"><a href="#stdio-free">3.7. Using the library in a stdio-free environment</a></span></dt> 199 <dd><dl> 200 <dt><span class="sect2"><a href="#stdio-bye">3.7.1. Getting rid of stdio</a></span></dt> 201 <dt><span class="sect2"><a href="#critical-error">3.7.2. Critical error handling</a></span></dt> 202 </dl></dd> 203 <dt><span class="sect1"><a href="#win-dll">3.8. Making a Windows DLL</a></span></dt> 204 </dl></dd> 205 <dt><span class="chapter"><a href="#misc">4. Miscellanea</a></span></dt> 206 <dd><dl> 207 <dt><span class="sect1"><a href="#limits">4.1. Limitations of the compressed file format</a></span></dt> 208 <dt><span class="sect1"><a href="#port-issues">4.2. Portability issues</a></span></dt> 209 <dt><span class="sect1"><a href="#bugs">4.3. Reporting bugs</a></span></dt> 210 <dt><span class="sect1"><a href="#package">4.4. Did you get the right package?</a></span></dt> 211 <dt><span class="sect1"><a href="#reading">4.5. Further Reading</a></span></dt> 212 </dl></dd> 213 </dl> 214 </div> 215 <div class="chapter"> 216 <div class="titlepage"><div><div><h1 class="title"> 217 <a name="intro"></a>1.Introduction</h1></div></div></div> 218 <p><code class="computeroutput">bzip2</code> compresses files 219 using the Burrows-Wheeler block-sorting text compression 220 algorithm, and Huffman coding. Compression is generally 221 considerably better than that achieved by more conventional 222 LZ77/LZ78-based compressors, and approaches the performance of 223 the PPM family of statistical compressors.</p> 224 <p><code class="computeroutput">bzip2</code> is built on top of 225 <code class="computeroutput">libbzip2</code>, a flexible library for 226 handling compressed data in the 227 <code class="computeroutput">bzip2</code> format. This manual 228 describes both how to use the program and how to work with the 229 library interface. Most of the manual is devoted to this 230 library, not the program, which is good news if your interest is 231 only in the program.</p> 232 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: bullet; "> 233 <li class="listitem" style="list-style-type: disc"><p><a class="xref" href="#using" title="2.How to use bzip2">How to use bzip2</a> describes how to use 234 <code class="computeroutput">bzip2</code>; this is the only part 235 you need to read if you just want to know how to operate the 236 program.</p></li> 237 <li class="listitem" style="list-style-type: disc"><p><a class="xref" href="#libprog" title="3. Programming with libbzip2">Programming with libbzip2</a> describes the 238 programming interfaces in detail, and</p></li> 239 <li class="listitem" style="list-style-type: disc"><p><a class="xref" href="#misc" title="4.Miscellanea">Miscellanea</a> records some 240 miscellaneous notes which I thought ought to be recorded 241 somewhere.</p></li> 242 </ul></div> 243 </div> 244 <div class="chapter"> 245 <div class="titlepage"><div><div><h1 class="title"> 246 <a name="using"></a>2.How to use bzip2</h1></div></div></div> 247 <div class="toc"> 248 <p><b>Table of Contents</b></p> 249 <dl class="toc"> 250 <dt><span class="sect1"><a href="#name">2.1. NAME</a></span></dt> 251 <dt><span class="sect1"><a href="#synopsis">2.2. SYNOPSIS</a></span></dt> 252 <dt><span class="sect1"><a href="#description">2.3. DESCRIPTION</a></span></dt> 253 <dt><span class="sect1"><a href="#options">2.4. OPTIONS</a></span></dt> 254 <dt><span class="sect1"><a href="#memory-management">2.5. MEMORY MANAGEMENT</a></span></dt> 255 <dt><span class="sect1"><a href="#recovering">2.6. RECOVERING DATA FROM DAMAGED FILES</a></span></dt> 256 <dt><span class="sect1"><a href="#performance">2.7. PERFORMANCE NOTES</a></span></dt> 257 <dt><span class="sect1"><a href="#caveats">2.8. CAVEATS</a></span></dt> 258 <dt><span class="sect1"><a href="#author">2.9. AUTHOR</a></span></dt> 259 </dl> 260 </div> 261 <p>This chapter contains a copy of the 262 <code class="computeroutput">bzip2</code> man page, and nothing 263 else.</p> 264 <div class="sect1"> 265 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 266 <a name="name"></a>2.1.NAME</h2></div></div></div> 267 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: bullet; "> 268 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzip2</code>, 269 <code class="computeroutput">bunzip2</code> - a block-sorting file 270 compressor, v1.0.8</p></li> 271 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzcat</code> - 272 decompresses files to stdout</p></li> 273 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzip2recover</code> - 274 recovers data from damaged bzip2 files</p></li> 275 </ul></div> 276 </div> 277 <div class="sect1"> 278 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 279 <a name="synopsis"></a>2.2.SYNOPSIS</h2></div></div></div> 280 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: bullet; "> 281 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzip2</code> [ 282 -cdfkqstvzVL123456789 ] [ filenames ... ]</p></li> 283 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bunzip2</code> [ 284 -fkvsVL ] [ filenames ... ]</p></li> 285 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzcat</code> [ -s ] [ 286 filenames ... ]</p></li> 287 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzip2recover</code> 288 filename</p></li> 289 </ul></div> 290 </div> 291 <div class="sect1"> 292 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 293 <a name="description"></a>2.3.DESCRIPTION</h2></div></div></div> 294 <p><code class="computeroutput">bzip2</code> compresses files 295 using the Burrows-Wheeler block sorting text compression 296 algorithm, and Huffman coding. Compression is generally 297 considerably better than that achieved by more conventional 298 LZ77/LZ78-based compressors, and approaches the performance of 299 the PPM family of statistical compressors.</p> 300 <p>The command-line options are deliberately very similar to 301 those of GNU <code class="computeroutput">gzip</code>, but they are 302 not identical.</p> 303 <p><code class="computeroutput">bzip2</code> expects a list of 304 file names to accompany the command-line flags. Each file is 305 replaced by a compressed version of itself, with the name 306 <code class="computeroutput">original_name.bz2</code>. Each 307 compressed file has the same modification date, permissions, and, 308 when possible, ownership as the corresponding original, so that 309 these properties can be correctly restored at decompression time. 310 File name handling is naive in the sense that there is no 311 mechanism for preserving original file names, permissions, 312 ownerships or dates in filesystems which lack these concepts, or 313 have serious file name length restrictions, such as 314 MS-DOS.</p> 315 <p><code class="computeroutput">bzip2</code> and 316 <code class="computeroutput">bunzip2</code> will by default not 317 overwrite existing files. If you want this to happen, specify 318 the <code class="computeroutput">-f</code> flag.</p> 319 <p>If no file names are specified, 320 <code class="computeroutput">bzip2</code> compresses from standard 321 input to standard output. In this case, 322 <code class="computeroutput">bzip2</code> will decline to write 323 compressed output to a terminal, as this would be entirely 324 incomprehensible and therefore pointless.</p> 325 <p><code class="computeroutput">bunzip2</code> (or 326 <code class="computeroutput">bzip2 -d</code>) decompresses all 327 specified files. Files which were not created by 328 <code class="computeroutput">bzip2</code> will be detected and 329 ignored, and a warning issued. 330 <code class="computeroutput">bzip2</code> attempts to guess the 331 filename for the decompressed file from that of the compressed 332 file as follows:</p> 333 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: bullet; "> 334 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">filename.bz2 </code> 335 becomes 336 <code class="computeroutput">filename</code></p></li> 337 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">filename.bz </code> 338 becomes 339 <code class="computeroutput">filename</code></p></li> 340 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">filename.tbz2</code> 341 becomes 342 <code class="computeroutput">filename.tar</code></p></li> 343 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">filename.tbz </code> 344 becomes 345 <code class="computeroutput">filename.tar</code></p></li> 346 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">anyothername </code> 347 becomes 348 <code class="computeroutput">anyothername.out</code></p></li> 349 </ul></div> 350 <p>If the file does not end in one of the recognised endings, 351 <code class="computeroutput">.bz2</code>, 352 <code class="computeroutput">.bz</code>, 353 <code class="computeroutput">.tbz2</code> or 354 <code class="computeroutput">.tbz</code>, 355 <code class="computeroutput">bzip2</code> complains that it cannot 356 guess the name of the original file, and uses the original name 357 with <code class="computeroutput">.out</code> appended.</p> 358 <p>As with compression, supplying no filenames causes 359 decompression from standard input to standard output.</p> 360 <p><code class="computeroutput">bunzip2</code> will correctly 361 decompress a file which is the concatenation of two or more 362 compressed files. The result is the concatenation of the 363 corresponding uncompressed files. Integrity testing 364 (<code class="computeroutput">-t</code>) of concatenated compressed 365 files is also supported.</p> 366 <p>You can also compress or decompress files to the standard 367 output by giving the <code class="computeroutput">-c</code> flag. 368 Multiple files may be compressed and decompressed like this. The 369 resulting outputs are fed sequentially to stdout. Compression of 370 multiple files in this manner generates a stream containing 371 multiple compressed file representations. Such a stream can be 372 decompressed correctly only by 373 <code class="computeroutput">bzip2</code> version 0.9.0 or later. 374 Earlier versions of <code class="computeroutput">bzip2</code> will 375 stop after decompressing the first file in the stream.</p> 376 <p><code class="computeroutput">bzcat</code> (or 377 <code class="computeroutput">bzip2 -dc</code>) decompresses all 378 specified files to the standard output.</p> 379 <p><code class="computeroutput">bzip2</code> will read arguments 380 from the environment variables 381 <code class="computeroutput">BZIP2</code> and 382 <code class="computeroutput">BZIP</code>, in that order, and will 383 process them before any arguments read from the command line. 384 This gives a convenient way to supply default arguments.</p> 385 <p>Compression is always performed, even if the compressed 386 file is slightly larger than the original. Files of less than 387 about one hundred bytes tend to get larger, since the compression 388 mechanism has a constant overhead in the region of 50 bytes. 389 Random data (including the output of most file compressors) is 390 coded at about 8.05 bits per byte, giving an expansion of around 391 0.5%.</p> 392 <p>As a self-check for your protection, 393 <code class="computeroutput">bzip2</code> uses 32-bit CRCs to make 394 sure that the decompressed version of a file is identical to the 395 original. This guards against corruption of the compressed data, 396 and against undetected bugs in 397 <code class="computeroutput">bzip2</code> (hopefully very unlikely). 398 The chances of data corruption going undetected is microscopic, 399 about one chance in four billion for each file processed. Be 400 aware, though, that the check occurs upon decompression, so it 401 can only tell you that something is wrong. It can't help you 402 recover the original uncompressed data. You can use 403 <code class="computeroutput">bzip2recover</code> to try to recover 404 data from damaged files.</p> 405 <p>Return values: 0 for a normal exit, 1 for environmental 406 problems (file not found, invalid flags, I/O errors, etc.), 2 407 to indicate a corrupt compressed file, 3 for an internal 408 consistency error (eg, bug) which caused 409 <code class="computeroutput">bzip2</code> to panic.</p> 410 </div> 411 <div class="sect1"> 412 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 413 <a name="options"></a>2.4.OPTIONS</h2></div></div></div> 414 <div class="variablelist"><dl class="variablelist"> 415 <dt><span class="term"><code class="computeroutput">-c --stdout</code></span></dt> 416 <dd><p>Compress or decompress to standard 417 output.</p></dd> 418 <dt><span class="term"><code class="computeroutput">-d --decompress</code></span></dt> 419 <dd><p>Force decompression. 420 <code class="computeroutput">bzip2</code>, 421 <code class="computeroutput">bunzip2</code> and 422 <code class="computeroutput">bzcat</code> are really the same 423 program, and the decision about what actions to take is done on 424 the basis of which name is used. This flag overrides that 425 mechanism, and forces bzip2 to decompress.</p></dd> 426 <dt><span class="term"><code class="computeroutput">-z --compress</code></span></dt> 427 <dd><p>The complement to 428 <code class="computeroutput">-d</code>: forces compression, 429 regardless of the invokation name.</p></dd> 430 <dt><span class="term"><code class="computeroutput">-t --test</code></span></dt> 431 <dd><p>Check integrity of the specified file(s), but 432 don't decompress them. This really performs a trial 433 decompression and throws away the result.</p></dd> 434 <dt><span class="term"><code class="computeroutput">-f --force</code></span></dt> 435 <dd> 436 <p>Force overwrite of output files. Normally, 437 <code class="computeroutput">bzip2</code> will not overwrite 438 existing output files. Also forces 439 <code class="computeroutput">bzip2</code> to break hard links to 440 files, which it otherwise wouldn't do.</p> 441 <p><code class="computeroutput">bzip2</code> normally declines 442 to decompress files which don't have the correct magic header 443 bytes. If forced (<code class="computeroutput">-f</code>), 444 however, it will pass such files through unmodified. This is 445 how GNU <code class="computeroutput">gzip</code> behaves.</p> 446 </dd> 447 <dt><span class="term"><code class="computeroutput">-k --keep</code></span></dt> 448 <dd><p>Keep (don't delete) input files during 449 compression or decompression.</p></dd> 450 <dt><span class="term"><code class="computeroutput">-s --small</code></span></dt> 451 <dd> 452 <p>Reduce memory usage, for compression, 453 decompression and testing. Files are decompressed and tested 454 using a modified algorithm which only requires 2.5 bytes per 455 block byte. This means any file can be decompressed in 2300k 456 of memory, albeit at about half the normal speed.</p> 457 <p>During compression, <code class="computeroutput">-s</code> 458 selects a block size of 200k, which limits memory use to around 459 the same figure, at the expense of your compression ratio. In 460 short, if your machine is low on memory (8 megabytes or less), 461 use <code class="computeroutput">-s</code> for everything. See 462 <a class="xref" href="#memory-management" title="2.5.MEMORY MANAGEMENT">MEMORY MANAGEMENT</a> below.</p> 463 </dd> 464 <dt><span class="term"><code class="computeroutput">-q --quiet</code></span></dt> 465 <dd><p>Suppress non-essential warning messages. 466 Messages pertaining to I/O errors and other critical events 467 will not be suppressed.</p></dd> 468 <dt><span class="term"><code class="computeroutput">-v --verbose</code></span></dt> 469 <dd><p>Verbose mode -- show the compression ratio for 470 each file processed. Further 471 <code class="computeroutput">-v</code>'s increase the verbosity 472 level, spewing out lots of information which is primarily of 473 interest for diagnostic purposes.</p></dd> 474 <dt><span class="term"><code class="computeroutput">-L --license -V --version</code></span></dt> 475 <dd><p>Display the software version, license terms and 476 conditions.</p></dd> 477 <dt><span class="term"><code class="computeroutput">-1</code> (or 478 <code class="computeroutput">--fast</code>) to 479 <code class="computeroutput">-9</code> (or 480 <code class="computeroutput">-best</code>)</span></dt> 481 <dd><p>Set the block size to 100 k, 200 k ... 900 k 482 when compressing. Has no effect when decompressing. See <a class="xref" href="#memory-management" title="2.5.MEMORY MANAGEMENT">MEMORY MANAGEMENT</a> below. The 483 <code class="computeroutput">--fast</code> and 484 <code class="computeroutput">--best</code> aliases are primarily 485 for GNU <code class="computeroutput">gzip</code> compatibility. 486 In particular, <code class="computeroutput">--fast</code> doesn't 487 make things significantly faster. And 488 <code class="computeroutput">--best</code> merely selects the 489 default behaviour.</p></dd> 490 <dt><span class="term"><code class="computeroutput">--</code></span></dt> 491 <dd><p>Treats all subsequent arguments as file names, 492 even if they start with a dash. This is so you can handle 493 files with names beginning with a dash, for example: 494 <code class="computeroutput">bzip2 -- 495 -myfilename</code>.</p></dd> 496 <dt> 497 <span class="term"><code class="computeroutput">--repetitive-fast</code>, </span><span class="term"><code class="computeroutput">--repetitive-best</code></span> 498 </dt> 499 <dd><p>These flags are redundant in versions 0.9.5 and 500 above. They provided some coarse control over the behaviour of 501 the sorting algorithm in earlier versions, which was sometimes 502 useful. 0.9.5 and above have an improved algorithm which 503 renders these flags irrelevant.</p></dd> 504 </dl></div> 505 </div> 506 <div class="sect1"> 507 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 508 <a name="memory-management"></a>2.5.MEMORY MANAGEMENT</h2></div></div></div> 509 <p><code class="computeroutput">bzip2</code> compresses large 510 files in blocks. The block size affects both the compression 511 ratio achieved, and the amount of memory needed for compression 512 and decompression. The flags <code class="computeroutput">-1</code> 513 through <code class="computeroutput">-9</code> specify the block 514 size to be 100,000 bytes through 900,000 bytes (the default) 515 respectively. At decompression time, the block size used for 516 compression is read from the header of the compressed file, and 517 <code class="computeroutput">bunzip2</code> then allocates itself 518 just enough memory to decompress the file. Since block sizes are 519 stored in compressed files, it follows that the flags 520 <code class="computeroutput">-1</code> to 521 <code class="computeroutput">-9</code> are irrelevant to and so 522 ignored during decompression.</p> 523 <p>Compression and decompression requirements, in bytes, can be 524 estimated as:</p> 525 <pre class="programlisting">Compression: 400k + ( 8 x block size ) 526 527 Decompression: 100k + ( 4 x block size ), or 528 100k + ( 2.5 x block size )</pre> 529 <p>Larger block sizes give rapidly diminishing marginal 530 returns. Most of the compression comes from the first two or 531 three hundred k of block size, a fact worth bearing in mind when 532 using <code class="computeroutput">bzip2</code> on small machines. 533 It is also important to appreciate that the decompression memory 534 requirement is set at compression time by the choice of block 535 size.</p> 536 <p>For files compressed with the default 900k block size, 537 <code class="computeroutput">bunzip2</code> will require about 3700 538 kbytes to decompress. To support decompression of any file on a 539 4 megabyte machine, <code class="computeroutput">bunzip2</code> has 540 an option to decompress using approximately half this amount of 541 memory, about 2300 kbytes. Decompression speed is also halved, 542 so you should use this option only where necessary. The relevant 543 flag is <code class="computeroutput">-s</code>.</p> 544 <p>In general, try and use the largest block size memory 545 constraints allow, since that maximises the compression achieved. 546 Compression and decompression speed are virtually unaffected by 547 block size.</p> 548 <p>Another significant point applies to files which fit in a 549 single block -- that means most files you'd encounter using a 550 large block size. The amount of real memory touched is 551 proportional to the size of the file, since the file is smaller 552 than a block. For example, compressing a file 20,000 bytes long 553 with the flag <code class="computeroutput">-9</code> will cause the 554 compressor to allocate around 7600k of memory, but only touch 555 400k + 20000 * 8 = 560 kbytes of it. Similarly, the decompressor 556 will allocate 3700k but only touch 100k + 20000 * 4 = 180 557 kbytes.</p> 558 <p>Here is a table which summarises the maximum memory usage 559 for different block sizes. Also recorded is the total compressed 560 size for 14 files of the Calgary Text Compression Corpus 561 totalling 3,141,622 bytes. This column gives some feel for how 562 compression varies with block size. These figures tend to 563 understate the advantage of larger block sizes for larger files, 564 since the Corpus is dominated by smaller files.</p> 565 <pre class="programlisting"> Compress Decompress Decompress Corpus 566 Flag usage usage -s usage Size 567 568 -1 1200k 500k 350k 914704 569 -2 2000k 900k 600k 877703 570 -3 2800k 1300k 850k 860338 571 -4 3600k 1700k 1100k 846899 572 -5 4400k 2100k 1350k 845160 573 -6 5200k 2500k 1600k 838626 574 -7 6100k 2900k 1850k 834096 575 -8 6800k 3300k 2100k 828642 576 -9 7600k 3700k 2350k 828642</pre> 577 </div> 578 <div class="sect1"> 579 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 580 <a name="recovering"></a>2.6.RECOVERING DATA FROM DAMAGED FILES</h2></div></div></div> 581 <p><code class="computeroutput">bzip2</code> compresses files in 582 blocks, usually 900kbytes long. Each block is handled 583 independently. If a media or transmission error causes a 584 multi-block <code class="computeroutput">.bz2</code> file to become 585 damaged, it may be possible to recover data from the undamaged 586 blocks in the file.</p> 587 <p>The compressed representation of each block is delimited by 588 a 48-bit pattern, which makes it possible to find the block 589 boundaries with reasonable certainty. Each block also carries 590 its own 32-bit CRC, so damaged blocks can be distinguished from 591 undamaged ones.</p> 592 <p><code class="computeroutput">bzip2recover</code> is a simple 593 program whose purpose is to search for blocks in 594 <code class="computeroutput">.bz2</code> files, and write each block 595 out into its own <code class="computeroutput">.bz2</code> file. You 596 can then use <code class="computeroutput">bzip2 -t</code> to test 597 the integrity of the resulting files, and decompress those which 598 are undamaged.</p> 599 <p><code class="computeroutput">bzip2recover</code> takes a 600 single argument, the name of the damaged file, and writes a 601 number of files <code class="computeroutput">rec0001file.bz2</code>, 602 <code class="computeroutput">rec0002file.bz2</code>, etc, containing 603 the extracted blocks. The output filenames are designed so that 604 the use of wildcards in subsequent processing -- for example, 605 <code class="computeroutput">bzip2 -dc rec*file.bz2 > 606 recovered_data</code> -- lists the files in the correct 607 order.</p> 608 <p><code class="computeroutput">bzip2recover</code> should be of 609 most use dealing with large <code class="computeroutput">.bz2</code> 610 files, as these will contain many blocks. It is clearly futile 611 to use it on damaged single-block files, since a damaged block 612 cannot be recovered. If you wish to minimise any potential data 613 loss through media or transmission errors, you might consider 614 compressing with a smaller block size.</p> 615 </div> 616 <div class="sect1"> 617 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 618 <a name="performance"></a>2.7.PERFORMANCE NOTES</h2></div></div></div> 619 <p>The sorting phase of compression gathers together similar 620 strings in the file. Because of this, files containing very long 621 runs of repeated symbols, like "aabaabaabaab ..." (repeated 622 several hundred times) may compress more slowly than normal. 623 Versions 0.9.5 and above fare much better than previous versions 624 in this respect. The ratio between worst-case and average-case 625 compression time is in the region of 10:1. For previous 626 versions, this figure was more like 100:1. You can use the 627 <code class="computeroutput">-vvvv</code> option to monitor progress 628 in great detail, if you want.</p> 629 <p>Decompression speed is unaffected by these 630 phenomena.</p> 631 <p><code class="computeroutput">bzip2</code> usually allocates 632 several megabytes of memory to operate in, and then charges all 633 over it in a fairly random fashion. This means that performance, 634 both for compressing and decompressing, is largely determined by 635 the speed at which your machine can service cache misses. 636 Because of this, small changes to the code to reduce the miss 637 rate have been observed to give disproportionately large 638 performance improvements. I imagine 639 <code class="computeroutput">bzip2</code> will perform best on 640 machines with very large caches.</p> 641 </div> 642 <div class="sect1"> 643 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 644 <a name="caveats"></a>2.8.CAVEATS</h2></div></div></div> 645 <p>I/O error messages are not as helpful as they could be. 646 <code class="computeroutput">bzip2</code> tries hard to detect I/O 647 errors and exit cleanly, but the details of what the problem is 648 sometimes seem rather misleading.</p> 649 <p>This manual page pertains to version 1.0.8 of 650 <code class="computeroutput">bzip2</code>. Compressed data created by 651 this version is entirely forwards and backwards compatible with the 652 previous public releases, versions 0.1pl2, 0.9.0 and 0.9.5, 1.0.0, 653 1.0.1, 1.0.2 and 1.0.3, but with the following exception: 0.9.0 and 654 above can correctly decompress multiple concatenated compressed files. 655 0.1pl2 cannot do this; it will stop after decompressing just the first 656 file in the stream.</p> 657 <p><code class="computeroutput">bzip2recover</code> versions 658 prior to 1.0.2 used 32-bit integers to represent bit positions in 659 compressed files, so it could not handle compressed files more 660 than 512 megabytes long. Versions 1.0.2 and above use 64-bit ints 661 on some platforms which support them (GNU supported targets, and 662 Windows). To establish whether or not 663 <code class="computeroutput">bzip2recover</code> was built with such 664 a limitation, run it without arguments. In any event you can 665 build yourself an unlimited version if you can recompile it with 666 <code class="computeroutput">MaybeUInt64</code> set to be an 667 unsigned 64-bit integer.</p> 668 </div> 669 <div class="sect1"> 670 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 671 <a name="author"></a>2.9.AUTHOR</h2></div></div></div> 672 <p>Julian Seward, 673 <code class="computeroutput">jseward (a] acm.org</code></p> 674 <p>The ideas embodied in 675 <code class="computeroutput">bzip2</code> are due to (at least) the 676 following people: Michael Burrows and David Wheeler (for the 677 block sorting transformation), David Wheeler (again, for the 678 Huffman coder), Peter Fenwick (for the structured coding model in 679 the original <code class="computeroutput">bzip</code>, and many 680 refinements), and Alistair Moffat, Radford Neal and Ian Witten 681 (for the arithmetic coder in the original 682 <code class="computeroutput">bzip</code>). I am much indebted for 683 their help, support and advice. See the manual in the source 684 distribution for pointers to sources of documentation. Christian 685 von Roques encouraged me to look for faster sorting algorithms, 686 so as to speed up compression. Bela Lubkin encouraged me to 687 improve the worst-case compression performance. 688 Donna Robinson XMLised the documentation. 689 Many people sent 690 patches, helped with portability problems, lent machines, gave 691 advice and were generally helpful.</p> 692 </div> 693 </div> 694 <div class="chapter"> 695 <div class="titlepage"><div><div><h1 class="title"> 696 <a name="libprog"></a>3. 697 Programming with <code class="computeroutput">libbzip2</code> 698 </h1></div></div></div> 699 <div class="toc"> 700 <p><b>Table of Contents</b></p> 701 <dl class="toc"> 702 <dt><span class="sect1"><a href="#top-level">3.1. Top-level structure</a></span></dt> 703 <dd><dl> 704 <dt><span class="sect2"><a href="#ll-summary">3.1.1. Low-level summary</a></span></dt> 705 <dt><span class="sect2"><a href="#hl-summary">3.1.2. High-level summary</a></span></dt> 706 <dt><span class="sect2"><a href="#util-fns-summary">3.1.3. Utility functions summary</a></span></dt> 707 </dl></dd> 708 <dt><span class="sect1"><a href="#err-handling">3.2. Error handling</a></span></dt> 709 <dt><span class="sect1"><a href="#low-level">3.3. Low-level interface</a></span></dt> 710 <dd><dl> 711 <dt><span class="sect2"><a href="#bzcompress-init">3.3.1. BZ2_bzCompressInit</a></span></dt> 712 <dt><span class="sect2"><a href="#bzCompress">3.3.2. BZ2_bzCompress</a></span></dt> 713 <dt><span class="sect2"><a href="#bzCompress-end">3.3.3. BZ2_bzCompressEnd</a></span></dt> 714 <dt><span class="sect2"><a href="#bzDecompress-init">3.3.4. BZ2_bzDecompressInit</a></span></dt> 715 <dt><span class="sect2"><a href="#bzDecompress">3.3.5. BZ2_bzDecompress</a></span></dt> 716 <dt><span class="sect2"><a href="#bzDecompress-end">3.3.6. BZ2_bzDecompressEnd</a></span></dt> 717 </dl></dd> 718 <dt><span class="sect1"><a href="#hl-interface">3.4. High-level interface</a></span></dt> 719 <dd><dl> 720 <dt><span class="sect2"><a href="#bzreadopen">3.4.1. BZ2_bzReadOpen</a></span></dt> 721 <dt><span class="sect2"><a href="#bzread">3.4.2. BZ2_bzRead</a></span></dt> 722 <dt><span class="sect2"><a href="#bzreadgetunused">3.4.3. BZ2_bzReadGetUnused</a></span></dt> 723 <dt><span class="sect2"><a href="#bzreadclose">3.4.4. BZ2_bzReadClose</a></span></dt> 724 <dt><span class="sect2"><a href="#bzwriteopen">3.4.5. BZ2_bzWriteOpen</a></span></dt> 725 <dt><span class="sect2"><a href="#bzwrite">3.4.6. BZ2_bzWrite</a></span></dt> 726 <dt><span class="sect2"><a href="#bzwriteclose">3.4.7. BZ2_bzWriteClose</a></span></dt> 727 <dt><span class="sect2"><a href="#embed">3.4.8. Handling embedded compressed data streams</a></span></dt> 728 <dt><span class="sect2"><a href="#std-rdwr">3.4.9. Standard file-reading/writing code</a></span></dt> 729 </dl></dd> 730 <dt><span class="sect1"><a href="#util-fns">3.5. Utility functions</a></span></dt> 731 <dd><dl> 732 <dt><span class="sect2"><a href="#bzbufftobuffcompress">3.5.1. BZ2_bzBuffToBuffCompress</a></span></dt> 733 <dt><span class="sect2"><a href="#bzbufftobuffdecompress">3.5.2. BZ2_bzBuffToBuffDecompress</a></span></dt> 734 </dl></dd> 735 <dt><span class="sect1"><a href="#zlib-compat">3.6. zlib compatibility functions</a></span></dt> 736 <dt><span class="sect1"><a href="#stdio-free">3.7. Using the library in a stdio-free environment</a></span></dt> 737 <dd><dl> 738 <dt><span class="sect2"><a href="#stdio-bye">3.7.1. Getting rid of stdio</a></span></dt> 739 <dt><span class="sect2"><a href="#critical-error">3.7.2. Critical error handling</a></span></dt> 740 </dl></dd> 741 <dt><span class="sect1"><a href="#win-dll">3.8. Making a Windows DLL</a></span></dt> 742 </dl> 743 </div> 744 <p>This chapter describes the programming interface to 745 <code class="computeroutput">libbzip2</code>.</p> 746 <p>For general background information, particularly about 747 memory use and performance aspects, you'd be well advised to read 748 <a class="xref" href="#using" title="2.How to use bzip2">How to use bzip2</a> as well.</p> 749 <div class="sect1"> 750 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 751 <a name="top-level"></a>3.1.Top-level structure</h2></div></div></div> 752 <p><code class="computeroutput">libbzip2</code> is a flexible 753 library for compressing and decompressing data in the 754 <code class="computeroutput">bzip2</code> data format. Although 755 packaged as a single entity, it helps to regard the library as 756 three separate parts: the low level interface, and the high level 757 interface, and some utility functions.</p> 758 <p>The structure of 759 <code class="computeroutput">libbzip2</code>'s interfaces is similar 760 to that of Jean-loup Gailly's and Mark Adler's excellent 761 <code class="computeroutput">zlib</code> library.</p> 762 <p>All externally visible symbols have names beginning 763 <code class="computeroutput">BZ2_</code>. This is new in version 764 1.0. The intention is to minimise pollution of the namespaces of 765 library clients.</p> 766 <p>To use any part of the library, you need to 767 <code class="computeroutput">#include <bzlib.h></code> 768 into your sources.</p> 769 <div class="sect2"> 770 <div class="titlepage"><div><div><h3 class="title"> 771 <a name="ll-summary"></a>3.1.1.Low-level summary</h3></div></div></div> 772 <p>This interface provides services for compressing and 773 decompressing data in memory. There's no provision for dealing 774 with files, streams or any other I/O mechanisms, just straight 775 memory-to-memory work. In fact, this part of the library can be 776 compiled without inclusion of 777 <code class="computeroutput">stdio.h</code>, which may be helpful 778 for embedded applications.</p> 779 <p>The low-level part of the library has no global variables 780 and is therefore thread-safe.</p> 781 <p>Six routines make up the low level interface: 782 <code class="computeroutput">BZ2_bzCompressInit</code>, 783 <code class="computeroutput">BZ2_bzCompress</code>, and 784 <code class="computeroutput">BZ2_bzCompressEnd</code> for 785 compression, and a corresponding trio 786 <code class="computeroutput">BZ2_bzDecompressInit</code>, 787 <code class="computeroutput">BZ2_bzDecompress</code> and 788 <code class="computeroutput">BZ2_bzDecompressEnd</code> for 789 decompression. The <code class="computeroutput">*Init</code> 790 functions allocate memory for compression/decompression and do 791 other initialisations, whilst the 792 <code class="computeroutput">*End</code> functions close down 793 operations and release memory.</p> 794 <p>The real work is done by 795 <code class="computeroutput">BZ2_bzCompress</code> and 796 <code class="computeroutput">BZ2_bzDecompress</code>. These 797 compress and decompress data from a user-supplied input buffer to 798 a user-supplied output buffer. These buffers can be any size; 799 arbitrary quantities of data are handled by making repeated calls 800 to these functions. This is a flexible mechanism allowing a 801 consumer-pull style of activity, or producer-push, or a mixture 802 of both.</p> 803 </div> 804 <div class="sect2"> 805 <div class="titlepage"><div><div><h3 class="title"> 806 <a name="hl-summary"></a>3.1.2.High-level summary</h3></div></div></div> 807 <p>This interface provides some handy wrappers around the 808 low-level interface to facilitate reading and writing 809 <code class="computeroutput">bzip2</code> format files 810 (<code class="computeroutput">.bz2</code> files). The routines 811 provide hooks to facilitate reading files in which the 812 <code class="computeroutput">bzip2</code> data stream is embedded 813 within some larger-scale file structure, or where there are 814 multiple <code class="computeroutput">bzip2</code> data streams 815 concatenated end-to-end.</p> 816 <p>For reading files, 817 <code class="computeroutput">BZ2_bzReadOpen</code>, 818 <code class="computeroutput">BZ2_bzRead</code>, 819 <code class="computeroutput">BZ2_bzReadClose</code> and 820 <code class="computeroutput">BZ2_bzReadGetUnused</code> are 821 supplied. For writing files, 822 <code class="computeroutput">BZ2_bzWriteOpen</code>, 823 <code class="computeroutput">BZ2_bzWrite</code> and 824 <code class="computeroutput">BZ2_bzWriteFinish</code> are 825 available.</p> 826 <p>As with the low-level library, no global variables are used 827 so the library is per se thread-safe. However, if I/O errors 828 occur whilst reading or writing the underlying compressed files, 829 you may have to consult <code class="computeroutput">errno</code> to 830 determine the cause of the error. In that case, you'd need a C 831 library which correctly supports 832 <code class="computeroutput">errno</code> in a multithreaded 833 environment.</p> 834 <p>To make the library a little simpler and more portable, 835 <code class="computeroutput">BZ2_bzReadOpen</code> and 836 <code class="computeroutput">BZ2_bzWriteOpen</code> require you to 837 pass them file handles (<code class="computeroutput">FILE*</code>s) 838 which have previously been opened for reading or writing 839 respectively. That avoids portability problems associated with 840 file operations and file attributes, whilst not being much of an 841 imposition on the programmer.</p> 842 </div> 843 <div class="sect2"> 844 <div class="titlepage"><div><div><h3 class="title"> 845 <a name="util-fns-summary"></a>3.1.3.Utility functions summary</h3></div></div></div> 846 <p>For very simple needs, 847 <code class="computeroutput">BZ2_bzBuffToBuffCompress</code> and 848 <code class="computeroutput">BZ2_bzBuffToBuffDecompress</code> are 849 provided. These compress data in memory from one buffer to 850 another buffer in a single function call. You should assess 851 whether these functions fulfill your memory-to-memory 852 compression/decompression requirements before investing effort in 853 understanding the more general but more complex low-level 854 interface.</p> 855 <p>Yoshioka Tsuneo 856 (<code class="computeroutput">tsuneo (a] rr.iij4u.or.jp</code>) has 857 contributed some functions to give better 858 <code class="computeroutput">zlib</code> compatibility. These 859 functions are <code class="computeroutput">BZ2_bzopen</code>, 860 <code class="computeroutput">BZ2_bzread</code>, 861 <code class="computeroutput">BZ2_bzwrite</code>, 862 <code class="computeroutput">BZ2_bzflush</code>, 863 <code class="computeroutput">BZ2_bzclose</code>, 864 <code class="computeroutput">BZ2_bzerror</code> and 865 <code class="computeroutput">BZ2_bzlibVersion</code>. You may find 866 these functions more convenient for simple file reading and 867 writing, than those in the high-level interface. These functions 868 are not (yet) officially part of the library, and are minimally 869 documented here. If they break, you get to keep all the pieces. 870 I hope to document them properly when time permits.</p> 871 <p>Yoshioka also contributed modifications to allow the 872 library to be built as a Windows DLL.</p> 873 </div> 874 </div> 875 <div class="sect1"> 876 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 877 <a name="err-handling"></a>3.2.Error handling</h2></div></div></div> 878 <p>The library is designed to recover cleanly in all 879 situations, including the worst-case situation of decompressing 880 random data. I'm not 100% sure that it can always do this, so 881 you might want to add a signal handler to catch segmentation 882 violations during decompression if you are feeling especially 883 paranoid. I would be interested in hearing more about the 884 robustness of the library to corrupted compressed data.</p> 885 <p>Version 1.0.3 more robust in this respect than any 886 previous version. Investigations with Valgrind (a tool for detecting 887 problems with memory management) indicate 888 that, at least for the few files I tested, all single-bit errors 889 in the decompressed data are caught properly, with no 890 segmentation faults, no uses of uninitialised data, no out of 891 range reads or writes, and no infinite looping in the decompressor. 892 So it's certainly pretty robust, although 893 I wouldn't claim it to be totally bombproof.</p> 894 <p>The file <code class="computeroutput">bzlib.h</code> contains 895 all definitions needed to use the library. In particular, you 896 should definitely not include 897 <code class="computeroutput">bzlib_private.h</code>.</p> 898 <p>In <code class="computeroutput">bzlib.h</code>, the various 899 return values are defined. The following list is not intended as 900 an exhaustive description of the circumstances in which a given 901 value may be returned -- those descriptions are given later. 902 Rather, it is intended to convey the rough meaning of each return 903 value. The first five actions are normal and not intended to 904 denote an error situation.</p> 905 <div class="variablelist"><dl class="variablelist"> 906 <dt><span class="term"><code class="computeroutput">BZ_OK</code></span></dt> 907 <dd><p>The requested action was completed 908 successfully.</p></dd> 909 <dt><span class="term"><code class="computeroutput">BZ_RUN_OK, BZ_FLUSH_OK, 910 BZ_FINISH_OK</code></span></dt> 911 <dd><p>In 912 <code class="computeroutput">BZ2_bzCompress</code>, the requested 913 flush/finish/nothing-special action was completed 914 successfully.</p></dd> 915 <dt><span class="term"><code class="computeroutput">BZ_STREAM_END</code></span></dt> 916 <dd><p>Compression of data was completed, or the 917 logical stream end was detected during 918 decompression.</p></dd> 919 </dl></div> 920 <p>The following return values indicate an error of some 921 kind.</p> 922 <div class="variablelist"><dl class="variablelist"> 923 <dt><span class="term"><code class="computeroutput">BZ_CONFIG_ERROR</code></span></dt> 924 <dd><p>Indicates that the library has been improperly 925 compiled on your platform -- a major configuration error. 926 Specifically, it means that 927 <code class="computeroutput">sizeof(char)</code>, 928 <code class="computeroutput">sizeof(short)</code> and 929 <code class="computeroutput">sizeof(int)</code> are not 1, 2 and 930 4 respectively, as they should be. Note that the library 931 should still work properly on 64-bit platforms which follow 932 the LP64 programming model -- that is, where 933 <code class="computeroutput">sizeof(long)</code> and 934 <code class="computeroutput">sizeof(void*)</code> are 8. Under 935 LP64, <code class="computeroutput">sizeof(int)</code> is still 4, 936 so <code class="computeroutput">libbzip2</code>, which doesn't 937 use the <code class="computeroutput">long</code> type, is 938 OK.</p></dd> 939 <dt><span class="term"><code class="computeroutput">BZ_SEQUENCE_ERROR</code></span></dt> 940 <dd><p>When using the library, it is important to call 941 the functions in the correct sequence and with data structures 942 (buffers etc) in the correct states. 943 <code class="computeroutput">libbzip2</code> checks as much as it 944 can to ensure this is happening, and returns 945 <code class="computeroutput">BZ_SEQUENCE_ERROR</code> if not. 946 Code which complies precisely with the function semantics, as 947 detailed below, should never receive this value; such an event 948 denotes buggy code which you should 949 investigate.</p></dd> 950 <dt><span class="term"><code class="computeroutput">BZ_PARAM_ERROR</code></span></dt> 951 <dd><p>Returned when a parameter to a function call is 952 out of range or otherwise manifestly incorrect. As with 953 <code class="computeroutput">BZ_SEQUENCE_ERROR</code>, this 954 denotes a bug in the client code. The distinction between 955 <code class="computeroutput">BZ_PARAM_ERROR</code> and 956 <code class="computeroutput">BZ_SEQUENCE_ERROR</code> is a bit 957 hazy, but still worth making.</p></dd> 958 <dt><span class="term"><code class="computeroutput">BZ_MEM_ERROR</code></span></dt> 959 <dd><p>Returned when a request to allocate memory 960 failed. Note that the quantity of memory needed to decompress 961 a stream cannot be determined until the stream's header has 962 been read. So 963 <code class="computeroutput">BZ2_bzDecompress</code> and 964 <code class="computeroutput">BZ2_bzRead</code> may return 965 <code class="computeroutput">BZ_MEM_ERROR</code> even though some 966 of the compressed data has been read. The same is not true 967 for compression; once 968 <code class="computeroutput">BZ2_bzCompressInit</code> or 969 <code class="computeroutput">BZ2_bzWriteOpen</code> have 970 successfully completed, 971 <code class="computeroutput">BZ_MEM_ERROR</code> cannot 972 occur.</p></dd> 973 <dt><span class="term"><code class="computeroutput">BZ_DATA_ERROR</code></span></dt> 974 <dd><p>Returned when a data integrity error is 975 detected during decompression. Most importantly, this means 976 when stored and computed CRCs for the data do not match. This 977 value is also returned upon detection of any other anomaly in 978 the compressed data.</p></dd> 979 <dt><span class="term"><code class="computeroutput">BZ_DATA_ERROR_MAGIC</code></span></dt> 980 <dd><p>As a special case of 981 <code class="computeroutput">BZ_DATA_ERROR</code>, it is 982 sometimes useful to know when the compressed stream does not 983 start with the correct magic bytes (<code class="computeroutput">'B' 'Z' 984 'h'</code>).</p></dd> 985 <dt><span class="term"><code class="computeroutput">BZ_IO_ERROR</code></span></dt> 986 <dd><p>Returned by 987 <code class="computeroutput">BZ2_bzRead</code> and 988 <code class="computeroutput">BZ2_bzWrite</code> when there is an 989 error reading or writing in the compressed file, and by 990 <code class="computeroutput">BZ2_bzReadOpen</code> and 991 <code class="computeroutput">BZ2_bzWriteOpen</code> for attempts 992 to use a file for which the error indicator (viz, 993 <code class="computeroutput">ferror(f)</code>) is set. On 994 receipt of <code class="computeroutput">BZ_IO_ERROR</code>, the 995 caller should consult <code class="computeroutput">errno</code> 996 and/or <code class="computeroutput">perror</code> to acquire 997 operating-system specific information about the 998 problem.</p></dd> 999 <dt><span class="term"><code class="computeroutput">BZ_UNEXPECTED_EOF</code></span></dt> 1000 <dd><p>Returned by 1001 <code class="computeroutput">BZ2_bzRead</code> when the 1002 compressed file finishes before the logical end of stream is 1003 detected.</p></dd> 1004 <dt><span class="term"><code class="computeroutput">BZ_OUTBUFF_FULL</code></span></dt> 1005 <dd><p>Returned by 1006 <code class="computeroutput">BZ2_bzBuffToBuffCompress</code> and 1007 <code class="computeroutput">BZ2_bzBuffToBuffDecompress</code> to 1008 indicate that the output data will not fit into the output 1009 buffer provided.</p></dd> 1010 </dl></div> 1011 </div> 1012 <div class="sect1"> 1013 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 1014 <a name="low-level"></a>3.3.Low-level interface</h2></div></div></div> 1015 <div class="sect2"> 1016 <div class="titlepage"><div><div><h3 class="title"> 1017 <a name="bzcompress-init"></a>3.3.1.BZ2_bzCompressInit</h3></div></div></div> 1018 <pre class="programlisting">typedef struct { 1019 char *next_in; 1020 unsigned int avail_in; 1021 unsigned int total_in_lo32; 1022 unsigned int total_in_hi32; 1023 1024 char *next_out; 1025 unsigned int avail_out; 1026 unsigned int total_out_lo32; 1027 unsigned int total_out_hi32; 1028 1029 void *state; 1030 1031 void *(*bzalloc)(void *,int,int); 1032 void (*bzfree)(void *,void *); 1033 void *opaque; 1034 } bz_stream; 1035 1036 int BZ2_bzCompressInit ( bz_stream *strm, 1037 int blockSize100k, 1038 int verbosity, 1039 int workFactor );</pre> 1040 <p>Prepares for compression. The 1041 <code class="computeroutput">bz_stream</code> structure holds all 1042 data pertaining to the compression activity. A 1043 <code class="computeroutput">bz_stream</code> structure should be 1044 allocated and initialised prior to the call. The fields of 1045 <code class="computeroutput">bz_stream</code> comprise the entirety 1046 of the user-visible data. <code class="computeroutput">state</code> 1047 is a pointer to the private data structures required for 1048 compression.</p> 1049 <p>Custom memory allocators are supported, via fields 1050 <code class="computeroutput">bzalloc</code>, 1051 <code class="computeroutput">bzfree</code>, and 1052 <code class="computeroutput">opaque</code>. The value 1053 <code class="computeroutput">opaque</code> is passed to as the first 1054 argument to all calls to <code class="computeroutput">bzalloc</code> 1055 and <code class="computeroutput">bzfree</code>, but is otherwise 1056 ignored by the library. The call <code class="computeroutput">bzalloc ( 1057 opaque, n, m )</code> is expected to return a pointer 1058 <code class="computeroutput">p</code> to <code class="computeroutput">n * 1059 m</code> bytes of memory, and <code class="computeroutput">bzfree ( 1060 opaque, p )</code> should free that memory.</p> 1061 <p>If you don't want to use a custom memory allocator, set 1062 <code class="computeroutput">bzalloc</code>, 1063 <code class="computeroutput">bzfree</code> and 1064 <code class="computeroutput">opaque</code> to 1065 <code class="computeroutput">NULL</code>, and the library will then 1066 use the standard <code class="computeroutput">malloc</code> / 1067 <code class="computeroutput">free</code> routines.</p> 1068 <p>Before calling 1069 <code class="computeroutput">BZ2_bzCompressInit</code>, fields 1070 <code class="computeroutput">bzalloc</code>, 1071 <code class="computeroutput">bzfree</code> and 1072 <code class="computeroutput">opaque</code> should be filled 1073 appropriately, as just described. Upon return, the internal 1074 state will have been allocated and initialised, and 1075 <code class="computeroutput">total_in_lo32</code>, 1076 <code class="computeroutput">total_in_hi32</code>, 1077 <code class="computeroutput">total_out_lo32</code> and 1078 <code class="computeroutput">total_out_hi32</code> will have been 1079 set to zero. These four fields are used by the library to inform 1080 the caller of the total amount of data passed into and out of the 1081 library, respectively. You should not try to change them. As of 1082 version 1.0, 64-bit counts are maintained, even on 32-bit 1083 platforms, using the <code class="computeroutput">_hi32</code> 1084 fields to store the upper 32 bits of the count. So, for example, 1085 the total amount of data in is <code class="computeroutput">(total_in_hi32 1086 << 32) + total_in_lo32</code>.</p> 1087 <p>Parameter <code class="computeroutput">blockSize100k</code> 1088 specifies the block size to be used for compression. It should 1089 be a value between 1 and 9 inclusive, and the actual block size 1090 used is 100000 x this figure. 9 gives the best compression but 1091 takes most memory.</p> 1092 <p>Parameter <code class="computeroutput">verbosity</code> should 1093 be set to a number between 0 and 4 inclusive. 0 is silent, and 1094 greater numbers give increasingly verbose monitoring/debugging 1095 output. If the library has been compiled with 1096 <code class="computeroutput">-DBZ_NO_STDIO</code>, no such output 1097 will appear for any verbosity setting.</p> 1098 <p>Parameter <code class="computeroutput">workFactor</code> 1099 controls how the compression phase behaves when presented with 1100 worst case, highly repetitive, input data. If compression runs 1101 into difficulties caused by repetitive data, the library switches 1102 from the standard sorting algorithm to a fallback algorithm. The 1103 fallback is slower than the standard algorithm by perhaps a 1104 factor of three, but always behaves reasonably, no matter how bad 1105 the input.</p> 1106 <p>Lower values of <code class="computeroutput">workFactor</code> 1107 reduce the amount of effort the standard algorithm will expend 1108 before resorting to the fallback. You should set this parameter 1109 carefully; too low, and many inputs will be handled by the 1110 fallback algorithm and so compress rather slowly, too high, and 1111 your average-to-worst case compression times can become very 1112 large. The default value of 30 gives reasonable behaviour over a 1113 wide range of circumstances.</p> 1114 <p>Allowable values range from 0 to 250 inclusive. 0 is a 1115 special case, equivalent to using the default value of 30.</p> 1116 <p>Note that the compressed output generated is the same 1117 regardless of whether or not the fallback algorithm is 1118 used.</p> 1119 <p>Be aware also that this parameter may disappear entirely in 1120 future versions of the library. In principle it should be 1121 possible to devise a good way to automatically choose which 1122 algorithm to use. Such a mechanism would render the parameter 1123 obsolete.</p> 1124 <p>Possible return values:</p> 1125 <pre class="programlisting">BZ_CONFIG_ERROR 1126 if the library has been mis-compiled 1127 BZ_PARAM_ERROR 1128 if strm is NULL 1129 or blockSize < 1 or blockSize > 9 1130 or verbosity < 0 or verbosity > 4 1131 or workFactor < 0 or workFactor > 250 1132 BZ_MEM_ERROR 1133 if not enough memory is available 1134 BZ_OK 1135 otherwise</pre> 1136 <p>Allowable next actions:</p> 1137 <pre class="programlisting">BZ2_bzCompress 1138 if BZ_OK is returned 1139 no specific action needed in case of error</pre> 1140 </div> 1141 <div class="sect2"> 1142 <div class="titlepage"><div><div><h3 class="title"> 1143 <a name="bzCompress"></a>3.3.2.BZ2_bzCompress</h3></div></div></div> 1144 <pre class="programlisting">int BZ2_bzCompress ( bz_stream *strm, int action );</pre> 1145 <p>Provides more input and/or output buffer space for the 1146 library. The caller maintains input and output buffers, and 1147 calls <code class="computeroutput">BZ2_bzCompress</code> to transfer 1148 data between them.</p> 1149 <p>Before each call to 1150 <code class="computeroutput">BZ2_bzCompress</code>, 1151 <code class="computeroutput">next_in</code> should point at the data 1152 to be compressed, and <code class="computeroutput">avail_in</code> 1153 should indicate how many bytes the library may read. 1154 <code class="computeroutput">BZ2_bzCompress</code> updates 1155 <code class="computeroutput">next_in</code>, 1156 <code class="computeroutput">avail_in</code> and 1157 <code class="computeroutput">total_in</code> to reflect the number 1158 of bytes it has read.</p> 1159 <p>Similarly, <code class="computeroutput">next_out</code> should 1160 point to a buffer in which the compressed data is to be placed, 1161 with <code class="computeroutput">avail_out</code> indicating how 1162 much output space is available. 1163 <code class="computeroutput">BZ2_bzCompress</code> updates 1164 <code class="computeroutput">next_out</code>, 1165 <code class="computeroutput">avail_out</code> and 1166 <code class="computeroutput">total_out</code> to reflect the number 1167 of bytes output.</p> 1168 <p>You may provide and remove as little or as much data as you 1169 like on each call of 1170 <code class="computeroutput">BZ2_bzCompress</code>. In the limit, 1171 it is acceptable to supply and remove data one byte at a time, 1172 although this would be terribly inefficient. You should always 1173 ensure that at least one byte of output space is available at 1174 each call.</p> 1175 <p>A second purpose of 1176 <code class="computeroutput">BZ2_bzCompress</code> is to request a 1177 change of mode of the compressed stream.</p> 1178 <p>Conceptually, a compressed stream can be in one of four 1179 states: IDLE, RUNNING, FLUSHING and FINISHING. Before 1180 initialisation 1181 (<code class="computeroutput">BZ2_bzCompressInit</code>) and after 1182 termination (<code class="computeroutput">BZ2_bzCompressEnd</code>), 1183 a stream is regarded as IDLE.</p> 1184 <p>Upon initialisation 1185 (<code class="computeroutput">BZ2_bzCompressInit</code>), the stream 1186 is placed in the RUNNING state. Subsequent calls to 1187 <code class="computeroutput">BZ2_bzCompress</code> should pass 1188 <code class="computeroutput">BZ_RUN</code> as the requested action; 1189 other actions are illegal and will result in 1190 <code class="computeroutput">BZ_SEQUENCE_ERROR</code>.</p> 1191 <p>At some point, the calling program will have provided all 1192 the input data it wants to. It will then want to finish up -- in 1193 effect, asking the library to process any data it might have 1194 buffered internally. In this state, 1195 <code class="computeroutput">BZ2_bzCompress</code> will no longer 1196 attempt to read data from 1197 <code class="computeroutput">next_in</code>, but it will want to 1198 write data to <code class="computeroutput">next_out</code>. Because 1199 the output buffer supplied by the user can be arbitrarily small, 1200 the finishing-up operation cannot necessarily be done with a 1201 single call of 1202 <code class="computeroutput">BZ2_bzCompress</code>.</p> 1203 <p>Instead, the calling program passes 1204 <code class="computeroutput">BZ_FINISH</code> as an action to 1205 <code class="computeroutput">BZ2_bzCompress</code>. This changes 1206 the stream's state to FINISHING. Any remaining input (ie, 1207 <code class="computeroutput">next_in[0 .. avail_in-1]</code>) is 1208 compressed and transferred to the output buffer. To do this, 1209 <code class="computeroutput">BZ2_bzCompress</code> must be called 1210 repeatedly until all the output has been consumed. At that 1211 point, <code class="computeroutput">BZ2_bzCompress</code> returns 1212 <code class="computeroutput">BZ_STREAM_END</code>, and the stream's 1213 state is set back to IDLE. 1214 <code class="computeroutput">BZ2_bzCompressEnd</code> should then be 1215 called.</p> 1216 <p>Just to make sure the calling program does not cheat, the 1217 library makes a note of <code class="computeroutput">avail_in</code> 1218 at the time of the first call to 1219 <code class="computeroutput">BZ2_bzCompress</code> which has 1220 <code class="computeroutput">BZ_FINISH</code> as an action (ie, at 1221 the time the program has announced its intention to not supply 1222 any more input). By comparing this value with that of 1223 <code class="computeroutput">avail_in</code> over subsequent calls 1224 to <code class="computeroutput">BZ2_bzCompress</code>, the library 1225 can detect any attempts to slip in more data to compress. Any 1226 calls for which this is detected will return 1227 <code class="computeroutput">BZ_SEQUENCE_ERROR</code>. This 1228 indicates a programming mistake which should be corrected.</p> 1229 <p>Instead of asking to finish, the calling program may ask 1230 <code class="computeroutput">BZ2_bzCompress</code> to take all the 1231 remaining input, compress it and terminate the current 1232 (Burrows-Wheeler) compression block. This could be useful for 1233 error control purposes. The mechanism is analogous to that for 1234 finishing: call <code class="computeroutput">BZ2_bzCompress</code> 1235 with an action of <code class="computeroutput">BZ_FLUSH</code>, 1236 remove output data, and persist with the 1237 <code class="computeroutput">BZ_FLUSH</code> action until the value 1238 <code class="computeroutput">BZ_RUN</code> is returned. As with 1239 finishing, <code class="computeroutput">BZ2_bzCompress</code> 1240 detects any attempt to provide more input data once the flush has 1241 begun.</p> 1242 <p>Once the flush is complete, the stream returns to the 1243 normal RUNNING state.</p> 1244 <p>This all sounds pretty complex, but isn't really. Here's a 1245 table which shows which actions are allowable in each state, what 1246 action will be taken, what the next state is, and what the 1247 non-error return values are. Note that you can't explicitly ask 1248 what state the stream is in, but nor do you need to -- it can be 1249 inferred from the values returned by 1250 <code class="computeroutput">BZ2_bzCompress</code>.</p> 1251 <pre class="programlisting">IDLE/any 1252 Illegal. IDLE state only exists after BZ2_bzCompressEnd or 1253 before BZ2_bzCompressInit. 1254 Return value = BZ_SEQUENCE_ERROR 1255 1256 RUNNING/BZ_RUN 1257 Compress from next_in to next_out as much as possible. 1258 Next state = RUNNING 1259 Return value = BZ_RUN_OK 1260 1261 RUNNING/BZ_FLUSH 1262 Remember current value of next_in. Compress from next_in 1263 to next_out as much as possible, but do not accept any more input. 1264 Next state = FLUSHING 1265 Return value = BZ_FLUSH_OK 1266 1267 RUNNING/BZ_FINISH 1268 Remember current value of next_in. Compress from next_in 1269 to next_out as much as possible, but do not accept any more input. 1270 Next state = FINISHING 1271 Return value = BZ_FINISH_OK 1272 1273 FLUSHING/BZ_FLUSH 1274 Compress from next_in to next_out as much as possible, 1275 but do not accept any more input. 1276 If all the existing input has been used up and all compressed 1277 output has been removed 1278 Next state = RUNNING; Return value = BZ_RUN_OK 1279 else 1280 Next state = FLUSHING; Return value = BZ_FLUSH_OK 1281 1282 FLUSHING/other 1283 Illegal. 1284 Return value = BZ_SEQUENCE_ERROR 1285 1286 FINISHING/BZ_FINISH 1287 Compress from next_in to next_out as much as possible, 1288 but to not accept any more input. 1289 If all the existing input has been used up and all compressed 1290 output has been removed 1291 Next state = IDLE; Return value = BZ_STREAM_END 1292 else 1293 Next state = FINISHING; Return value = BZ_FINISH_OK 1294 1295 FINISHING/other 1296 Illegal. 1297 Return value = BZ_SEQUENCE_ERROR</pre> 1298 <p>That still looks complicated? Well, fair enough. The 1299 usual sequence of calls for compressing a load of data is:</p> 1300 <div class="orderedlist"><ol class="orderedlist" type="1"> 1301 <li class="listitem"><p>Get started with 1302 <code class="computeroutput">BZ2_bzCompressInit</code>.</p></li> 1303 <li class="listitem"><p>Shovel data in and shlurp out its compressed form 1304 using zero or more calls of 1305 <code class="computeroutput">BZ2_bzCompress</code> with action = 1306 <code class="computeroutput">BZ_RUN</code>.</p></li> 1307 <li class="listitem"><p>Finish up. Repeatedly call 1308 <code class="computeroutput">BZ2_bzCompress</code> with action = 1309 <code class="computeroutput">BZ_FINISH</code>, copying out the 1310 compressed output, until 1311 <code class="computeroutput">BZ_STREAM_END</code> is 1312 returned.</p></li> 1313 <li class="listitem"><p>Close up and go home. Call 1314 <code class="computeroutput">BZ2_bzCompressEnd</code>.</p></li> 1315 </ol></div> 1316 <p>If the data you want to compress fits into your input 1317 buffer all at once, you can skip the calls of 1318 <code class="computeroutput">BZ2_bzCompress ( ..., BZ_RUN )</code> 1319 and just do the <code class="computeroutput">BZ2_bzCompress ( ..., BZ_FINISH 1320 )</code> calls.</p> 1321 <p>All required memory is allocated by 1322 <code class="computeroutput">BZ2_bzCompressInit</code>. The 1323 compression library can accept any data at all (obviously). So 1324 you shouldn't get any error return values from the 1325 <code class="computeroutput">BZ2_bzCompress</code> calls. If you 1326 do, they will be 1327 <code class="computeroutput">BZ_SEQUENCE_ERROR</code>, and indicate 1328 a bug in your programming.</p> 1329 <p>Trivial other possible return values:</p> 1330 <pre class="programlisting">BZ_PARAM_ERROR 1331 if strm is NULL, or strm->s is NULL</pre> 1332 </div> 1333 <div class="sect2"> 1334 <div class="titlepage"><div><div><h3 class="title"> 1335 <a name="bzCompress-end"></a>3.3.3.BZ2_bzCompressEnd</h3></div></div></div> 1336 <pre class="programlisting">int BZ2_bzCompressEnd ( bz_stream *strm );</pre> 1337 <p>Releases all memory associated with a compression 1338 stream.</p> 1339 <p>Possible return values:</p> 1340 <pre class="programlisting">BZ_PARAM_ERROR if strm is NULL or strm->s is NULL 1341 BZ_OK otherwise</pre> 1342 </div> 1343 <div class="sect2"> 1344 <div class="titlepage"><div><div><h3 class="title"> 1345 <a name="bzDecompress-init"></a>3.3.4.BZ2_bzDecompressInit</h3></div></div></div> 1346 <pre class="programlisting">int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small );</pre> 1347 <p>Prepares for decompression. As with 1348 <code class="computeroutput">BZ2_bzCompressInit</code>, a 1349 <code class="computeroutput">bz_stream</code> record should be 1350 allocated and initialised before the call. Fields 1351 <code class="computeroutput">bzalloc</code>, 1352 <code class="computeroutput">bzfree</code> and 1353 <code class="computeroutput">opaque</code> should be set if a custom 1354 memory allocator is required, or made 1355 <code class="computeroutput">NULL</code> for the normal 1356 <code class="computeroutput">malloc</code> / 1357 <code class="computeroutput">free</code> routines. Upon return, the 1358 internal state will have been initialised, and 1359 <code class="computeroutput">total_in</code> and 1360 <code class="computeroutput">total_out</code> will be zero.</p> 1361 <p>For the meaning of parameter 1362 <code class="computeroutput">verbosity</code>, see 1363 <code class="computeroutput">BZ2_bzCompressInit</code>.</p> 1364 <p>If <code class="computeroutput">small</code> is nonzero, the 1365 library will use an alternative decompression algorithm which 1366 uses less memory but at the cost of decompressing more slowly 1367 (roughly speaking, half the speed, but the maximum memory 1368 requirement drops to around 2300k). See <a class="xref" href="#using" title="2.How to use bzip2">How to use bzip2</a> 1369 for more information on memory management.</p> 1370 <p>Note that the amount of memory needed to decompress a 1371 stream cannot be determined until the stream's header has been 1372 read, so even if 1373 <code class="computeroutput">BZ2_bzDecompressInit</code> succeeds, a 1374 subsequent <code class="computeroutput">BZ2_bzDecompress</code> 1375 could fail with 1376 <code class="computeroutput">BZ_MEM_ERROR</code>.</p> 1377 <p>Possible return values:</p> 1378 <pre class="programlisting">BZ_CONFIG_ERROR 1379 if the library has been mis-compiled 1380 BZ_PARAM_ERROR 1381 if ( small != 0 && small != 1 ) 1382 or (verbosity <; 0 || verbosity > 4) 1383 BZ_MEM_ERROR 1384 if insufficient memory is available</pre> 1385 <p>Allowable next actions:</p> 1386 <pre class="programlisting">BZ2_bzDecompress 1387 if BZ_OK was returned 1388 no specific action required in case of error</pre> 1389 </div> 1390 <div class="sect2"> 1391 <div class="titlepage"><div><div><h3 class="title"> 1392 <a name="bzDecompress"></a>3.3.5.BZ2_bzDecompress</h3></div></div></div> 1393 <pre class="programlisting">int BZ2_bzDecompress ( bz_stream *strm );</pre> 1394 <p>Provides more input and/out output buffer space for the 1395 library. The caller maintains input and output buffers, and uses 1396 <code class="computeroutput">BZ2_bzDecompress</code> to transfer 1397 data between them.</p> 1398 <p>Before each call to 1399 <code class="computeroutput">BZ2_bzDecompress</code>, 1400 <code class="computeroutput">next_in</code> should point at the 1401 compressed data, and <code class="computeroutput">avail_in</code> 1402 should indicate how many bytes the library may read. 1403 <code class="computeroutput">BZ2_bzDecompress</code> updates 1404 <code class="computeroutput">next_in</code>, 1405 <code class="computeroutput">avail_in</code> and 1406 <code class="computeroutput">total_in</code> to reflect the number 1407 of bytes it has read.</p> 1408 <p>Similarly, <code class="computeroutput">next_out</code> should 1409 point to a buffer in which the uncompressed output is to be 1410 placed, with <code class="computeroutput">avail_out</code> 1411 indicating how much output space is available. 1412 <code class="computeroutput">BZ2_bzCompress</code> updates 1413 <code class="computeroutput">next_out</code>, 1414 <code class="computeroutput">avail_out</code> and 1415 <code class="computeroutput">total_out</code> to reflect the number 1416 of bytes output.</p> 1417 <p>You may provide and remove as little or as much data as you 1418 like on each call of 1419 <code class="computeroutput">BZ2_bzDecompress</code>. In the limit, 1420 it is acceptable to supply and remove data one byte at a time, 1421 although this would be terribly inefficient. You should always 1422 ensure that at least one byte of output space is available at 1423 each call.</p> 1424 <p>Use of <code class="computeroutput">BZ2_bzDecompress</code> is 1425 simpler than 1426 <code class="computeroutput">BZ2_bzCompress</code>.</p> 1427 <p>You should provide input and remove output as described 1428 above, and repeatedly call 1429 <code class="computeroutput">BZ2_bzDecompress</code> until 1430 <code class="computeroutput">BZ_STREAM_END</code> is returned. 1431 Appearance of <code class="computeroutput">BZ_STREAM_END</code> 1432 denotes that <code class="computeroutput">BZ2_bzDecompress</code> 1433 has detected the logical end of the compressed stream. 1434 <code class="computeroutput">BZ2_bzDecompress</code> will not 1435 produce <code class="computeroutput">BZ_STREAM_END</code> until all 1436 output data has been placed into the output buffer, so once 1437 <code class="computeroutput">BZ_STREAM_END</code> appears, you are 1438 guaranteed to have available all the decompressed output, and 1439 <code class="computeroutput">BZ2_bzDecompressEnd</code> can safely 1440 be called.</p> 1441 <p>If case of an error return value, you should call 1442 <code class="computeroutput">BZ2_bzDecompressEnd</code> to clean up 1443 and release memory.</p> 1444 <p>Possible return values:</p> 1445 <pre class="programlisting">BZ_PARAM_ERROR 1446 if strm is NULL or strm->s is NULL 1447 or strm->avail_out < 1 1448 BZ_DATA_ERROR 1449 if a data integrity error is detected in the compressed stream 1450 BZ_DATA_ERROR_MAGIC 1451 if the compressed stream doesn't begin with the right magic bytes 1452 BZ_MEM_ERROR 1453 if there wasn't enough memory available 1454 BZ_STREAM_END 1455 if the logical end of the data stream was detected and all 1456 output in has been consumed, eg s-->avail_out > 0 1457 BZ_OK 1458 otherwise</pre> 1459 <p>Allowable next actions:</p> 1460 <pre class="programlisting">BZ2_bzDecompress 1461 if BZ_OK was returned 1462 BZ2_bzDecompressEnd 1463 otherwise</pre> 1464 </div> 1465 <div class="sect2"> 1466 <div class="titlepage"><div><div><h3 class="title"> 1467 <a name="bzDecompress-end"></a>3.3.6.BZ2_bzDecompressEnd</h3></div></div></div> 1468 <pre class="programlisting">int BZ2_bzDecompressEnd ( bz_stream *strm );</pre> 1469 <p>Releases all memory associated with a decompression 1470 stream.</p> 1471 <p>Possible return values:</p> 1472 <pre class="programlisting">BZ_PARAM_ERROR 1473 if strm is NULL or strm->s is NULL 1474 BZ_OK 1475 otherwise</pre> 1476 <p>Allowable next actions:</p> 1477 <pre class="programlisting"> None.</pre> 1478 </div> 1479 </div> 1480 <div class="sect1"> 1481 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 1482 <a name="hl-interface"></a>3.4.High-level interface</h2></div></div></div> 1483 <p>This interface provides functions for reading and writing 1484 <code class="computeroutput">bzip2</code> format files. First, some 1485 general points.</p> 1486 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: bullet; "> 1487 <li class="listitem" style="list-style-type: disc"><p>All of the functions take an 1488 <code class="computeroutput">int*</code> first argument, 1489 <code class="computeroutput">bzerror</code>. After each call, 1490 <code class="computeroutput">bzerror</code> should be consulted 1491 first to determine the outcome of the call. If 1492 <code class="computeroutput">bzerror</code> is 1493 <code class="computeroutput">BZ_OK</code>, the call completed 1494 successfully, and only then should the return value of the 1495 function (if any) be consulted. If 1496 <code class="computeroutput">bzerror</code> is 1497 <code class="computeroutput">BZ_IO_ERROR</code>, there was an 1498 error reading/writing the underlying compressed file, and you 1499 should then consult <code class="computeroutput">errno</code> / 1500 <code class="computeroutput">perror</code> to determine the cause 1501 of the difficulty. <code class="computeroutput">bzerror</code> 1502 may also be set to various other values; precise details are 1503 given on a per-function basis below.</p></li> 1504 <li class="listitem" style="list-style-type: disc"><p>If <code class="computeroutput">bzerror</code> indicates 1505 an error (ie, anything except 1506 <code class="computeroutput">BZ_OK</code> and 1507 <code class="computeroutput">BZ_STREAM_END</code>), you should 1508 immediately call 1509 <code class="computeroutput">BZ2_bzReadClose</code> (or 1510 <code class="computeroutput">BZ2_bzWriteClose</code>, depending on 1511 whether you are attempting to read or to write) to free up all 1512 resources associated with the stream. Once an error has been 1513 indicated, behaviour of all calls except 1514 <code class="computeroutput">BZ2_bzReadClose</code> 1515 (<code class="computeroutput">BZ2_bzWriteClose</code>) is 1516 undefined. The implication is that (1) 1517 <code class="computeroutput">bzerror</code> should be checked 1518 after each call, and (2) if 1519 <code class="computeroutput">bzerror</code> indicates an error, 1520 <code class="computeroutput">BZ2_bzReadClose</code> 1521 (<code class="computeroutput">BZ2_bzWriteClose</code>) should then 1522 be called to clean up.</p></li> 1523 <li class="listitem" style="list-style-type: disc"><p>The <code class="computeroutput">FILE*</code> arguments 1524 passed to <code class="computeroutput">BZ2_bzReadOpen</code> / 1525 <code class="computeroutput">BZ2_bzWriteOpen</code> should be set 1526 to binary mode. Most Unix systems will do this by default, but 1527 other platforms, including Windows and Mac, will not. If you 1528 omit this, you may encounter problems when moving code to new 1529 platforms.</p></li> 1530 <li class="listitem" style="list-style-type: disc"><p>Memory allocation requests are handled by 1531 <code class="computeroutput">malloc</code> / 1532 <code class="computeroutput">free</code>. At present there is no 1533 facility for user-defined memory allocators in the file I/O 1534 functions (could easily be added, though).</p></li> 1535 </ul></div> 1536 <div class="sect2"> 1537 <div class="titlepage"><div><div><h3 class="title"> 1538 <a name="bzreadopen"></a>3.4.1.BZ2_bzReadOpen</h3></div></div></div> 1539 <pre class="programlisting">typedef void BZFILE; 1540 1541 BZFILE *BZ2_bzReadOpen( int *bzerror, FILE *f, 1542 int verbosity, int small, 1543 void *unused, int nUnused );</pre> 1544 <p>Prepare to read compressed data from file handle 1545 <code class="computeroutput">f</code>. 1546 <code class="computeroutput">f</code> should refer to a file which 1547 has been opened for reading, and for which the error indicator 1548 (<code class="computeroutput">ferror(f)</code>)is not set. If 1549 <code class="computeroutput">small</code> is 1, the library will try 1550 to decompress using less memory, at the expense of speed.</p> 1551 <p>For reasons explained below, 1552 <code class="computeroutput">BZ2_bzRead</code> will decompress the 1553 <code class="computeroutput">nUnused</code> bytes starting at 1554 <code class="computeroutput">unused</code>, before starting to read 1555 from the file <code class="computeroutput">f</code>. At most 1556 <code class="computeroutput">BZ_MAX_UNUSED</code> bytes may be 1557 supplied like this. If this facility is not required, you should 1558 pass <code class="computeroutput">NULL</code> and 1559 <code class="computeroutput">0</code> for 1560 <code class="computeroutput">unused</code> and 1561 n<code class="computeroutput">Unused</code> respectively.</p> 1562 <p>For the meaning of parameters 1563 <code class="computeroutput">small</code> and 1564 <code class="computeroutput">verbosity</code>, see 1565 <code class="computeroutput">BZ2_bzDecompressInit</code>.</p> 1566 <p>The amount of memory needed to decompress a file cannot be 1567 determined until the file's header has been read. So it is 1568 possible that <code class="computeroutput">BZ2_bzReadOpen</code> 1569 returns <code class="computeroutput">BZ_OK</code> but a subsequent 1570 call of <code class="computeroutput">BZ2_bzRead</code> will return 1571 <code class="computeroutput">BZ_MEM_ERROR</code>.</p> 1572 <p>Possible assignments to 1573 <code class="computeroutput">bzerror</code>:</p> 1574 <pre class="programlisting">BZ_CONFIG_ERROR 1575 if the library has been mis-compiled 1576 BZ_PARAM_ERROR 1577 if f is NULL 1578 or small is neither 0 nor 1 1579 or ( unused == NULL && nUnused != 0 ) 1580 or ( unused != NULL && !(0 <= nUnused <= BZ_MAX_UNUSED) ) 1581 BZ_IO_ERROR 1582 if ferror(f) is nonzero 1583 BZ_MEM_ERROR 1584 if insufficient memory is available 1585 BZ_OK 1586 otherwise.</pre> 1587 <p>Possible return values:</p> 1588 <pre class="programlisting">Pointer to an abstract BZFILE 1589 if bzerror is BZ_OK 1590 NULL 1591 otherwise</pre> 1592 <p>Allowable next actions:</p> 1593 <pre class="programlisting">BZ2_bzRead 1594 if bzerror is BZ_OK 1595 BZ2_bzClose 1596 otherwise</pre> 1597 </div> 1598 <div class="sect2"> 1599 <div class="titlepage"><div><div><h3 class="title"> 1600 <a name="bzread"></a>3.4.2.BZ2_bzRead</h3></div></div></div> 1601 <pre class="programlisting">int BZ2_bzRead ( int *bzerror, BZFILE *b, void *buf, int len );</pre> 1602 <p>Reads up to <code class="computeroutput">len</code> 1603 (uncompressed) bytes from the compressed file 1604 <code class="computeroutput">b</code> into the buffer 1605 <code class="computeroutput">buf</code>. If the read was 1606 successful, <code class="computeroutput">bzerror</code> is set to 1607 <code class="computeroutput">BZ_OK</code> and the number of bytes 1608 read is returned. If the logical end-of-stream was detected, 1609 <code class="computeroutput">bzerror</code> will be set to 1610 <code class="computeroutput">BZ_STREAM_END</code>, and the number of 1611 bytes read is returned. All other 1612 <code class="computeroutput">bzerror</code> values denote an 1613 error.</p> 1614 <p><code class="computeroutput">BZ2_bzRead</code> will supply 1615 <code class="computeroutput">len</code> bytes, unless the logical 1616 stream end is detected or an error occurs. Because of this, it 1617 is possible to detect the stream end by observing when the number 1618 of bytes returned is less than the number requested. 1619 Nevertheless, this is regarded as inadvisable; you should instead 1620 check <code class="computeroutput">bzerror</code> after every call 1621 and watch out for 1622 <code class="computeroutput">BZ_STREAM_END</code>.</p> 1623 <p>Internally, <code class="computeroutput">BZ2_bzRead</code> 1624 copies data from the compressed file in chunks of size 1625 <code class="computeroutput">BZ_MAX_UNUSED</code> bytes before 1626 decompressing it. If the file contains more bytes than strictly 1627 needed to reach the logical end-of-stream, 1628 <code class="computeroutput">BZ2_bzRead</code> will almost certainly 1629 read some of the trailing data before signalling 1630 <code class="computeroutput">BZ_SEQUENCE_END</code>. To collect the 1631 read but unused data once 1632 <code class="computeroutput">BZ_SEQUENCE_END</code> has appeared, 1633 call <code class="computeroutput">BZ2_bzReadGetUnused</code> 1634 immediately before 1635 <code class="computeroutput">BZ2_bzReadClose</code>.</p> 1636 <p>Possible assignments to 1637 <code class="computeroutput">bzerror</code>:</p> 1638 <pre class="programlisting">BZ_PARAM_ERROR 1639 if b is NULL or buf is NULL or len < 0 1640 BZ_SEQUENCE_ERROR 1641 if b was opened with BZ2_bzWriteOpen 1642 BZ_IO_ERROR 1643 if there is an error reading from the compressed file 1644 BZ_UNEXPECTED_EOF 1645 if the compressed file ended before 1646 the logical end-of-stream was detected 1647 BZ_DATA_ERROR 1648 if a data integrity error was detected in the compressed stream 1649 BZ_DATA_ERROR_MAGIC 1650 if the stream does not begin with the requisite header bytes 1651 (ie, is not a bzip2 data file). This is really 1652 a special case of BZ_DATA_ERROR. 1653 BZ_MEM_ERROR 1654 if insufficient memory was available 1655 BZ_STREAM_END 1656 if the logical end of stream was detected. 1657 BZ_OK 1658 otherwise.</pre> 1659 <p>Possible return values:</p> 1660 <pre class="programlisting">number of bytes read 1661 if bzerror is BZ_OK or BZ_STREAM_END 1662 undefined 1663 otherwise</pre> 1664 <p>Allowable next actions:</p> 1665 <pre class="programlisting">collect data from buf, then BZ2_bzRead or BZ2_bzReadClose 1666 if bzerror is BZ_OK 1667 collect data from buf, then BZ2_bzReadClose or BZ2_bzReadGetUnused 1668 if bzerror is BZ_SEQUENCE_END 1669 BZ2_bzReadClose 1670 otherwise</pre> 1671 </div> 1672 <div class="sect2"> 1673 <div class="titlepage"><div><div><h3 class="title"> 1674 <a name="bzreadgetunused"></a>3.4.3.BZ2_bzReadGetUnused</h3></div></div></div> 1675 <pre class="programlisting">void BZ2_bzReadGetUnused( int* bzerror, BZFILE *b, 1676 void** unused, int* nUnused );</pre> 1677 <p>Returns data which was read from the compressed file but 1678 was not needed to get to the logical end-of-stream. 1679 <code class="computeroutput">*unused</code> is set to the address of 1680 the data, and <code class="computeroutput">*nUnused</code> to the 1681 number of bytes. <code class="computeroutput">*nUnused</code> will 1682 be set to a value between <code class="computeroutput">0</code> and 1683 <code class="computeroutput">BZ_MAX_UNUSED</code> inclusive.</p> 1684 <p>This function may only be called once 1685 <code class="computeroutput">BZ2_bzRead</code> has signalled 1686 <code class="computeroutput">BZ_STREAM_END</code> but before 1687 <code class="computeroutput">BZ2_bzReadClose</code>.</p> 1688 <p>Possible assignments to 1689 <code class="computeroutput">bzerror</code>:</p> 1690 <pre class="programlisting">BZ_PARAM_ERROR 1691 if b is NULL 1692 or unused is NULL or nUnused is NULL 1693 BZ_SEQUENCE_ERROR 1694 if BZ_STREAM_END has not been signalled 1695 or if b was opened with BZ2_bzWriteOpen 1696 BZ_OK 1697 otherwise</pre> 1698 <p>Allowable next actions:</p> 1699 <pre class="programlisting">BZ2_bzReadClose</pre> 1700 </div> 1701 <div class="sect2"> 1702 <div class="titlepage"><div><div><h3 class="title"> 1703 <a name="bzreadclose"></a>3.4.4.BZ2_bzReadClose</h3></div></div></div> 1704 <pre class="programlisting">void BZ2_bzReadClose ( int *bzerror, BZFILE *b );</pre> 1705 <p>Releases all memory pertaining to the compressed file 1706 <code class="computeroutput">b</code>. 1707 <code class="computeroutput">BZ2_bzReadClose</code> does not call 1708 <code class="computeroutput">fclose</code> on the underlying file 1709 handle, so you should do that yourself if appropriate. 1710 <code class="computeroutput">BZ2_bzReadClose</code> should be called 1711 to clean up after all error situations.</p> 1712 <p>Possible assignments to 1713 <code class="computeroutput">bzerror</code>:</p> 1714 <pre class="programlisting">BZ_SEQUENCE_ERROR 1715 if b was opened with BZ2_bzOpenWrite 1716 BZ_OK 1717 otherwise</pre> 1718 <p>Allowable next actions:</p> 1719 <pre class="programlisting">none</pre> 1720 </div> 1721 <div class="sect2"> 1722 <div class="titlepage"><div><div><h3 class="title"> 1723 <a name="bzwriteopen"></a>3.4.5.BZ2_bzWriteOpen</h3></div></div></div> 1724 <pre class="programlisting">BZFILE *BZ2_bzWriteOpen( int *bzerror, FILE *f, 1725 int blockSize100k, int verbosity, 1726 int workFactor );</pre> 1727 <p>Prepare to write compressed data to file handle 1728 <code class="computeroutput">f</code>. 1729 <code class="computeroutput">f</code> should refer to a file which 1730 has been opened for writing, and for which the error indicator 1731 (<code class="computeroutput">ferror(f)</code>)is not set.</p> 1732 <p>For the meaning of parameters 1733 <code class="computeroutput">blockSize100k</code>, 1734 <code class="computeroutput">verbosity</code> and 1735 <code class="computeroutput">workFactor</code>, see 1736 <code class="computeroutput">BZ2_bzCompressInit</code>.</p> 1737 <p>All required memory is allocated at this stage, so if the 1738 call completes successfully, 1739 <code class="computeroutput">BZ_MEM_ERROR</code> cannot be signalled 1740 by a subsequent call to 1741 <code class="computeroutput">BZ2_bzWrite</code>.</p> 1742 <p>Possible assignments to 1743 <code class="computeroutput">bzerror</code>:</p> 1744 <pre class="programlisting">BZ_CONFIG_ERROR 1745 if the library has been mis-compiled 1746 BZ_PARAM_ERROR 1747 if f is NULL 1748 or blockSize100k < 1 or blockSize100k > 9 1749 BZ_IO_ERROR 1750 if ferror(f) is nonzero 1751 BZ_MEM_ERROR 1752 if insufficient memory is available 1753 BZ_OK 1754 otherwise</pre> 1755 <p>Possible return values:</p> 1756 <pre class="programlisting">Pointer to an abstract BZFILE 1757 if bzerror is BZ_OK 1758 NULL 1759 otherwise</pre> 1760 <p>Allowable next actions:</p> 1761 <pre class="programlisting">BZ2_bzWrite 1762 if bzerror is BZ_OK 1763 (you could go directly to BZ2_bzWriteClose, but this would be pretty pointless) 1764 BZ2_bzWriteClose 1765 otherwise</pre> 1766 </div> 1767 <div class="sect2"> 1768 <div class="titlepage"><div><div><h3 class="title"> 1769 <a name="bzwrite"></a>3.4.6.BZ2_bzWrite</h3></div></div></div> 1770 <pre class="programlisting">void BZ2_bzWrite ( int *bzerror, BZFILE *b, void *buf, int len );</pre> 1771 <p>Absorbs <code class="computeroutput">len</code> bytes from the 1772 buffer <code class="computeroutput">buf</code>, eventually to be 1773 compressed and written to the file.</p> 1774 <p>Possible assignments to 1775 <code class="computeroutput">bzerror</code>:</p> 1776 <pre class="programlisting">BZ_PARAM_ERROR 1777 if b is NULL or buf is NULL or len < 0 1778 BZ_SEQUENCE_ERROR 1779 if b was opened with BZ2_bzReadOpen 1780 BZ_IO_ERROR 1781 if there is an error writing the compressed file. 1782 BZ_OK 1783 otherwise</pre> 1784 </div> 1785 <div class="sect2"> 1786 <div class="titlepage"><div><div><h3 class="title"> 1787 <a name="bzwriteclose"></a>3.4.7.BZ2_bzWriteClose</h3></div></div></div> 1788 <pre class="programlisting">void BZ2_bzWriteClose( int *bzerror, BZFILE* f, 1789 int abandon, 1790 unsigned int* nbytes_in, 1791 unsigned int* nbytes_out ); 1792 1793 void BZ2_bzWriteClose64( int *bzerror, BZFILE* f, 1794 int abandon, 1795 unsigned int* nbytes_in_lo32, 1796 unsigned int* nbytes_in_hi32, 1797 unsigned int* nbytes_out_lo32, 1798 unsigned int* nbytes_out_hi32 );</pre> 1799 <p>Compresses and flushes to the compressed file all data so 1800 far supplied by <code class="computeroutput">BZ2_bzWrite</code>. 1801 The logical end-of-stream markers are also written, so subsequent 1802 calls to <code class="computeroutput">BZ2_bzWrite</code> are 1803 illegal. All memory associated with the compressed file 1804 <code class="computeroutput">b</code> is released. 1805 <code class="computeroutput">fflush</code> is called on the 1806 compressed file, but it is not 1807 <code class="computeroutput">fclose</code>'d.</p> 1808 <p>If <code class="computeroutput">BZ2_bzWriteClose</code> is 1809 called to clean up after an error, the only action is to release 1810 the memory. The library records the error codes issued by 1811 previous calls, so this situation will be detected automatically. 1812 There is no attempt to complete the compression operation, nor to 1813 <code class="computeroutput">fflush</code> the compressed file. You 1814 can force this behaviour to happen even in the case of no error, 1815 by passing a nonzero value to 1816 <code class="computeroutput">abandon</code>.</p> 1817 <p>If <code class="computeroutput">nbytes_in</code> is non-null, 1818 <code class="computeroutput">*nbytes_in</code> will be set to be the 1819 total volume of uncompressed data handled. Similarly, 1820 <code class="computeroutput">nbytes_out</code> will be set to the 1821 total volume of compressed data written. For compatibility with 1822 older versions of the library, 1823 <code class="computeroutput">BZ2_bzWriteClose</code> only yields the 1824 lower 32 bits of these counts. Use 1825 <code class="computeroutput">BZ2_bzWriteClose64</code> if you want 1826 the full 64 bit counts. These two functions are otherwise 1827 absolutely identical.</p> 1828 <p>Possible assignments to 1829 <code class="computeroutput">bzerror</code>:</p> 1830 <pre class="programlisting">BZ_SEQUENCE_ERROR 1831 if b was opened with BZ2_bzReadOpen 1832 BZ_IO_ERROR 1833 if there is an error writing the compressed file 1834 BZ_OK 1835 otherwise</pre> 1836 </div> 1837 <div class="sect2"> 1838 <div class="titlepage"><div><div><h3 class="title"> 1839 <a name="embed"></a>3.4.8.Handling embedded compressed data streams</h3></div></div></div> 1840 <p>The high-level library facilitates use of 1841 <code class="computeroutput">bzip2</code> data streams which form 1842 some part of a surrounding, larger data stream.</p> 1843 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: bullet; "> 1844 <li class="listitem" style="list-style-type: disc"><p>For writing, the library takes an open file handle, 1845 writes compressed data to it, 1846 <code class="computeroutput">fflush</code>es it but does not 1847 <code class="computeroutput">fclose</code> it. The calling 1848 application can write its own data before and after the 1849 compressed data stream, using that same file handle.</p></li> 1850 <li class="listitem" style="list-style-type: disc"><p>Reading is more complex, and the facilities are not as 1851 general as they could be since generality is hard to reconcile 1852 with efficiency. <code class="computeroutput">BZ2_bzRead</code> 1853 reads from the compressed file in blocks of size 1854 <code class="computeroutput">BZ_MAX_UNUSED</code> bytes, and in 1855 doing so probably will overshoot the logical end of compressed 1856 stream. To recover this data once decompression has ended, 1857 call <code class="computeroutput">BZ2_bzReadGetUnused</code> after 1858 the last call of <code class="computeroutput">BZ2_bzRead</code> 1859 (the one returning 1860 <code class="computeroutput">BZ_STREAM_END</code>) but before 1861 calling 1862 <code class="computeroutput">BZ2_bzReadClose</code>.</p></li> 1863 </ul></div> 1864 <p>This mechanism makes it easy to decompress multiple 1865 <code class="computeroutput">bzip2</code> streams placed end-to-end. 1866 As the end of one stream, when 1867 <code class="computeroutput">BZ2_bzRead</code> returns 1868 <code class="computeroutput">BZ_STREAM_END</code>, call 1869 <code class="computeroutput">BZ2_bzReadGetUnused</code> to collect 1870 the unused data (copy it into your own buffer somewhere). That 1871 data forms the start of the next compressed stream. To start 1872 uncompressing that next stream, call 1873 <code class="computeroutput">BZ2_bzReadOpen</code> again, feeding in 1874 the unused data via the <code class="computeroutput">unused</code> / 1875 <code class="computeroutput">nUnused</code> parameters. Keep doing 1876 this until <code class="computeroutput">BZ_STREAM_END</code> return 1877 coincides with the physical end of file 1878 (<code class="computeroutput">feof(f)</code>). In this situation 1879 <code class="computeroutput">BZ2_bzReadGetUnused</code> will of 1880 course return no data.</p> 1881 <p>This should give some feel for how the high-level interface 1882 can be used. If you require extra flexibility, you'll have to 1883 bite the bullet and get to grips with the low-level 1884 interface.</p> 1885 </div> 1886 <div class="sect2"> 1887 <div class="titlepage"><div><div><h3 class="title"> 1888 <a name="std-rdwr"></a>3.4.9.Standard file-reading/writing code</h3></div></div></div> 1889 <p>Here's how you'd write data to a compressed file:</p> 1890 <pre class="programlisting">FILE* f; 1891 BZFILE* b; 1892 int nBuf; 1893 char buf[ /* whatever size you like */ ]; 1894 int bzerror; 1895 int nWritten; 1896 1897 f = fopen ( "myfile.bz2", "w" ); 1898 if ( !f ) { 1899 /* handle error */ 1900 } 1901 b = BZ2_bzWriteOpen( &bzerror, f, 9 ); 1902 if (bzerror != BZ_OK) { 1903 BZ2_bzWriteClose ( b ); 1904 /* handle error */ 1905 } 1906 1907 while ( /* condition */ ) { 1908 /* get data to write into buf, and set nBuf appropriately */ 1909 nWritten = BZ2_bzWrite ( &bzerror, b, buf, nBuf ); 1910 if (bzerror == BZ_IO_ERROR) { 1911 BZ2_bzWriteClose ( &bzerror, b ); 1912 /* handle error */ 1913 } 1914 } 1915 1916 BZ2_bzWriteClose( &bzerror, b ); 1917 if (bzerror == BZ_IO_ERROR) { 1918 /* handle error */ 1919 }</pre> 1920 <p>And to read from a compressed file:</p> 1921 <pre class="programlisting">FILE* f; 1922 BZFILE* b; 1923 int nBuf; 1924 char buf[ /* whatever size you like */ ]; 1925 int bzerror; 1926 int nWritten; 1927 1928 f = fopen ( "myfile.bz2", "r" ); 1929 if ( !f ) { 1930 /* handle error */ 1931 } 1932 b = BZ2_bzReadOpen ( &bzerror, f, 0, NULL, 0 ); 1933 if ( bzerror != BZ_OK ) { 1934 BZ2_bzReadClose ( &bzerror, b ); 1935 /* handle error */ 1936 } 1937 1938 bzerror = BZ_OK; 1939 while ( bzerror == BZ_OK && /* arbitrary other conditions */) { 1940 nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf */ ); 1941 if ( bzerror == BZ_OK ) { 1942 /* do something with buf[0 .. nBuf-1] */ 1943 } 1944 } 1945 if ( bzerror != BZ_STREAM_END ) { 1946 BZ2_bzReadClose ( &bzerror, b ); 1947 /* handle error */ 1948 } else { 1949 BZ2_bzReadClose ( &bzerror, b ); 1950 }</pre> 1951 </div> 1952 </div> 1953 <div class="sect1"> 1954 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 1955 <a name="util-fns"></a>3.5.Utility functions</h2></div></div></div> 1956 <div class="sect2"> 1957 <div class="titlepage"><div><div><h3 class="title"> 1958 <a name="bzbufftobuffcompress"></a>3.5.1.BZ2_bzBuffToBuffCompress</h3></div></div></div> 1959 <pre class="programlisting">int BZ2_bzBuffToBuffCompress( char* dest, 1960 unsigned int* destLen, 1961 char* source, 1962 unsigned int sourceLen, 1963 int blockSize100k, 1964 int verbosity, 1965 int workFactor );</pre> 1966 <p>Attempts to compress the data in <code class="computeroutput">source[0 1967 .. sourceLen-1]</code> into the destination buffer, 1968 <code class="computeroutput">dest[0 .. *destLen-1]</code>. If the 1969 destination buffer is big enough, 1970 <code class="computeroutput">*destLen</code> is set to the size of 1971 the compressed data, and <code class="computeroutput">BZ_OK</code> 1972 is returned. If the compressed data won't fit, 1973 <code class="computeroutput">*destLen</code> is unchanged, and 1974 <code class="computeroutput">BZ_OUTBUFF_FULL</code> is 1975 returned.</p> 1976 <p>Compression in this manner is a one-shot event, done with a 1977 single call to this function. The resulting compressed data is a 1978 complete <code class="computeroutput">bzip2</code> format data 1979 stream. There is no mechanism for making additional calls to 1980 provide extra input data. If you want that kind of mechanism, 1981 use the low-level interface.</p> 1982 <p>For the meaning of parameters 1983 <code class="computeroutput">blockSize100k</code>, 1984 <code class="computeroutput">verbosity</code> and 1985 <code class="computeroutput">workFactor</code>, see 1986 <code class="computeroutput">BZ2_bzCompressInit</code>.</p> 1987 <p>To guarantee that the compressed data will fit in its 1988 buffer, allocate an output buffer of size 1% larger than the 1989 uncompressed data, plus six hundred extra bytes.</p> 1990 <p><code class="computeroutput">BZ2_bzBuffToBuffDecompress</code> 1991 will not write data at or beyond 1992 <code class="computeroutput">dest[*destLen]</code>, even in case of 1993 buffer overflow.</p> 1994 <p>Possible return values:</p> 1995 <pre class="programlisting">BZ_CONFIG_ERROR 1996 if the library has been mis-compiled 1997 BZ_PARAM_ERROR 1998 if dest is NULL or destLen is NULL 1999 or blockSize100k < 1 or blockSize100k > 9 2000 or verbosity < 0 or verbosity > 4 2001 or workFactor < 0 or workFactor > 250 2002 BZ_MEM_ERROR 2003 if insufficient memory is available 2004 BZ_OUTBUFF_FULL 2005 if the size of the compressed data exceeds *destLen 2006 BZ_OK 2007 otherwise</pre> 2008 </div> 2009 <div class="sect2"> 2010 <div class="titlepage"><div><div><h3 class="title"> 2011 <a name="bzbufftobuffdecompress"></a>3.5.2.BZ2_bzBuffToBuffDecompress</h3></div></div></div> 2012 <pre class="programlisting">int BZ2_bzBuffToBuffDecompress( char* dest, 2013 unsigned int* destLen, 2014 char* source, 2015 unsigned int sourceLen, 2016 int small, 2017 int verbosity );</pre> 2018 <p>Attempts to decompress the data in <code class="computeroutput">source[0 2019 .. sourceLen-1]</code> into the destination buffer, 2020 <code class="computeroutput">dest[0 .. *destLen-1]</code>. If the 2021 destination buffer is big enough, 2022 <code class="computeroutput">*destLen</code> is set to the size of 2023 the uncompressed data, and <code class="computeroutput">BZ_OK</code> 2024 is returned. If the compressed data won't fit, 2025 <code class="computeroutput">*destLen</code> is unchanged, and 2026 <code class="computeroutput">BZ_OUTBUFF_FULL</code> is 2027 returned.</p> 2028 <p><code class="computeroutput">source</code> is assumed to hold 2029 a complete <code class="computeroutput">bzip2</code> format data 2030 stream. 2031 <code class="computeroutput">BZ2_bzBuffToBuffDecompress</code> tries 2032 to decompress the entirety of the stream into the output 2033 buffer.</p> 2034 <p>For the meaning of parameters 2035 <code class="computeroutput">small</code> and 2036 <code class="computeroutput">verbosity</code>, see 2037 <code class="computeroutput">BZ2_bzDecompressInit</code>.</p> 2038 <p>Because the compression ratio of the compressed data cannot 2039 be known in advance, there is no easy way to guarantee that the 2040 output buffer will be big enough. You may of course make 2041 arrangements in your code to record the size of the uncompressed 2042 data, but such a mechanism is beyond the scope of this 2043 library.</p> 2044 <p><code class="computeroutput">BZ2_bzBuffToBuffDecompress</code> 2045 will not write data at or beyond 2046 <code class="computeroutput">dest[*destLen]</code>, even in case of 2047 buffer overflow.</p> 2048 <p>Possible return values:</p> 2049 <pre class="programlisting">BZ_CONFIG_ERROR 2050 if the library has been mis-compiled 2051 BZ_PARAM_ERROR 2052 if dest is NULL or destLen is NULL 2053 or small != 0 && small != 1 2054 or verbosity < 0 or verbosity > 4 2055 BZ_MEM_ERROR 2056 if insufficient memory is available 2057 BZ_OUTBUFF_FULL 2058 if the size of the compressed data exceeds *destLen 2059 BZ_DATA_ERROR 2060 if a data integrity error was detected in the compressed data 2061 BZ_DATA_ERROR_MAGIC 2062 if the compressed data doesn't begin with the right magic bytes 2063 BZ_UNEXPECTED_EOF 2064 if the compressed data ends unexpectedly 2065 BZ_OK 2066 otherwise</pre> 2067 </div> 2068 </div> 2069 <div class="sect1"> 2070 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 2071 <a name="zlib-compat"></a>3.6.zlib compatibility functions</h2></div></div></div> 2072 <p>Yoshioka Tsuneo has contributed some functions to give 2073 better <code class="computeroutput">zlib</code> compatibility. 2074 These functions are <code class="computeroutput">BZ2_bzopen</code>, 2075 <code class="computeroutput">BZ2_bzread</code>, 2076 <code class="computeroutput">BZ2_bzwrite</code>, 2077 <code class="computeroutput">BZ2_bzflush</code>, 2078 <code class="computeroutput">BZ2_bzclose</code>, 2079 <code class="computeroutput">BZ2_bzerror</code> and 2080 <code class="computeroutput">BZ2_bzlibVersion</code>. These 2081 functions are not (yet) officially part of the library. If they 2082 break, you get to keep all the pieces. Nevertheless, I think 2083 they work ok.</p> 2084 <pre class="programlisting">typedef void BZFILE; 2085 2086 const char * BZ2_bzlibVersion ( void );</pre> 2087 <p>Returns a string indicating the library version.</p> 2088 <pre class="programlisting">BZFILE * BZ2_bzopen ( const char *path, const char *mode ); 2089 BZFILE * BZ2_bzdopen ( int fd, const char *mode );</pre> 2090 <p>Opens a <code class="computeroutput">.bz2</code> file for 2091 reading or writing, using either its name or a pre-existing file 2092 descriptor. Analogous to <code class="computeroutput">fopen</code> 2093 and <code class="computeroutput">fdopen</code>.</p> 2094 <pre class="programlisting">int BZ2_bzread ( BZFILE* b, void* buf, int len ); 2095 int BZ2_bzwrite ( BZFILE* b, void* buf, int len );</pre> 2096 <p>Reads/writes data from/to a previously opened 2097 <code class="computeroutput">BZFILE</code>. Analogous to 2098 <code class="computeroutput">fread</code> and 2099 <code class="computeroutput">fwrite</code>.</p> 2100 <pre class="programlisting">int BZ2_bzflush ( BZFILE* b ); 2101 void BZ2_bzclose ( BZFILE* b );</pre> 2102 <p>Flushes/closes a <code class="computeroutput">BZFILE</code>. 2103 <code class="computeroutput">BZ2_bzflush</code> doesn't actually do 2104 anything. Analogous to <code class="computeroutput">fflush</code> 2105 and <code class="computeroutput">fclose</code>.</p> 2106 <pre class="programlisting">const char * BZ2_bzerror ( BZFILE *b, int *errnum )</pre> 2107 <p>Returns a string describing the more recent error status of 2108 <code class="computeroutput">b</code>, and also sets 2109 <code class="computeroutput">*errnum</code> to its numerical 2110 value.</p> 2111 </div> 2112 <div class="sect1"> 2113 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 2114 <a name="stdio-free"></a>3.7.Using the library in a stdio-free environment</h2></div></div></div> 2115 <div class="sect2"> 2116 <div class="titlepage"><div><div><h3 class="title"> 2117 <a name="stdio-bye"></a>3.7.1.Getting rid of stdio</h3></div></div></div> 2118 <p>In a deeply embedded application, you might want to use 2119 just the memory-to-memory functions. You can do this 2120 conveniently by compiling the library with preprocessor symbol 2121 <code class="computeroutput">BZ_NO_STDIO</code> defined. Doing this 2122 gives you a library containing only the following eight 2123 functions:</p> 2124 <p><code class="computeroutput">BZ2_bzCompressInit</code>, 2125 <code class="computeroutput">BZ2_bzCompress</code>, 2126 <code class="computeroutput">BZ2_bzCompressEnd</code> 2127 <code class="computeroutput">BZ2_bzDecompressInit</code>, 2128 <code class="computeroutput">BZ2_bzDecompress</code>, 2129 <code class="computeroutput">BZ2_bzDecompressEnd</code> 2130 <code class="computeroutput">BZ2_bzBuffToBuffCompress</code>, 2131 <code class="computeroutput">BZ2_bzBuffToBuffDecompress</code></p> 2132 <p>When compiled like this, all functions will ignore 2133 <code class="computeroutput">verbosity</code> settings.</p> 2134 </div> 2135 <div class="sect2"> 2136 <div class="titlepage"><div><div><h3 class="title"> 2137 <a name="critical-error"></a>3.7.2.Critical error handling</h3></div></div></div> 2138 <p><code class="computeroutput">libbzip2</code> contains a number 2139 of internal assertion checks which should, needless to say, never 2140 be activated. Nevertheless, if an assertion should fail, 2141 behaviour depends on whether or not the library was compiled with 2142 <code class="computeroutput">BZ_NO_STDIO</code> set.</p> 2143 <p>For a normal compile, an assertion failure yields the 2144 message:</p> 2145 <div class="blockquote"><blockquote class="blockquote"> 2146 <p>bzip2/libbzip2: internal error number N.</p> 2147 <p>This is a bug in bzip2/libbzip2, 1.0.8 of 13 July 2019. 2148 Please report it to: bzip2-devel (a] sourceware.org. If this happened 2149 when you were using some program which uses libbzip2 as a 2150 component, you should also report this bug to the author(s) 2151 of that program. Please make an effort to report this bug; 2152 timely and accurate bug reports eventually lead to higher 2153 quality software. Thanks. 2154 </p> 2155 </blockquote></div> 2156 <p>where <code class="computeroutput">N</code> is some error code 2157 number. If <code class="computeroutput">N == 1007</code>, it also 2158 prints some extra text advising the reader that unreliable memory 2159 is often associated with internal error 1007. (This is a 2160 frequently-observed-phenomenon with versions 1.0.0/1.0.1).</p> 2161 <p><code class="computeroutput">exit(3)</code> is then 2162 called.</p> 2163 <p>For a <code class="computeroutput">stdio</code>-free library, 2164 assertion failures result in a call to a function declared 2165 as:</p> 2166 <pre class="programlisting">extern void bz_internal_error ( int errcode );</pre> 2167 <p>The relevant code is passed as a parameter. You should 2168 supply such a function.</p> 2169 <p>In either case, once an assertion failure has occurred, any 2170 <code class="computeroutput">bz_stream</code> records involved can 2171 be regarded as invalid. You should not attempt to resume normal 2172 operation with them.</p> 2173 <p>You may, of course, change critical error handling to suit 2174 your needs. As I said above, critical errors indicate bugs in 2175 the library and should not occur. All "normal" error situations 2176 are indicated via error return codes from functions, and can be 2177 recovered from.</p> 2178 </div> 2179 </div> 2180 <div class="sect1"> 2181 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 2182 <a name="win-dll"></a>3.8.Making a Windows DLL</h2></div></div></div> 2183 <p>Everything related to Windows has been contributed by 2184 Yoshioka Tsuneo 2185 (<code class="computeroutput">tsuneo (a] rr.iij4u.or.jp</code>), so 2186 you should send your queries to him (but please Cc: 2187 <code class="computeroutput">bzip2-devel (a] sourceware.org</code>).</p> 2188 <p>My vague understanding of what to do is: using Visual C++ 2189 5.0, open the project file 2190 <code class="computeroutput">libbz2.dsp</code>, and build. That's 2191 all.</p> 2192 <p>If you can't open the project file for some reason, make a 2193 new one, naming these files: 2194 <code class="computeroutput">blocksort.c</code>, 2195 <code class="computeroutput">bzlib.c</code>, 2196 <code class="computeroutput">compress.c</code>, 2197 <code class="computeroutput">crctable.c</code>, 2198 <code class="computeroutput">decompress.c</code>, 2199 <code class="computeroutput">huffman.c</code>, 2200 <code class="computeroutput">randtable.c</code> and 2201 <code class="computeroutput">libbz2.def</code>. You will also need 2202 to name the header files <code class="computeroutput">bzlib.h</code> 2203 and <code class="computeroutput">bzlib_private.h</code>.</p> 2204 <p>If you don't use VC++, you may need to define the 2205 proprocessor symbol 2206 <code class="computeroutput">_WIN32</code>.</p> 2207 <p>Finally, <code class="computeroutput">dlltest.c</code> is a 2208 sample program using the DLL. It has a project file, 2209 <code class="computeroutput">dlltest.dsp</code>.</p> 2210 <p>If you just want a makefile for Visual C, have a look at 2211 <code class="computeroutput">makefile.msc</code>.</p> 2212 <p>Be aware that if you compile 2213 <code class="computeroutput">bzip2</code> itself on Win32, you must 2214 set <code class="computeroutput">BZ_UNIX</code> to 0 and 2215 <code class="computeroutput">BZ_LCCWIN32</code> to 1, in the file 2216 <code class="computeroutput">bzip2.c</code>, before compiling. 2217 Otherwise the resulting binary won't work correctly.</p> 2218 <p>I haven't tried any of this stuff myself, but it all looks 2219 plausible.</p> 2220 </div> 2221 </div> 2222 <div class="chapter"> 2223 <div class="titlepage"><div><div><h1 class="title"> 2224 <a name="misc"></a>4.Miscellanea</h1></div></div></div> 2225 <div class="toc"> 2226 <p><b>Table of Contents</b></p> 2227 <dl class="toc"> 2228 <dt><span class="sect1"><a href="#limits">4.1. Limitations of the compressed file format</a></span></dt> 2229 <dt><span class="sect1"><a href="#port-issues">4.2. Portability issues</a></span></dt> 2230 <dt><span class="sect1"><a href="#bugs">4.3. Reporting bugs</a></span></dt> 2231 <dt><span class="sect1"><a href="#package">4.4. Did you get the right package?</a></span></dt> 2232 <dt><span class="sect1"><a href="#reading">4.5. Further Reading</a></span></dt> 2233 </dl> 2234 </div> 2235 <p>These are just some random thoughts of mine. Your mileage 2236 may vary.</p> 2237 <div class="sect1"> 2238 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 2239 <a name="limits"></a>4.1.Limitations of the compressed file format</h2></div></div></div> 2240 <p><code class="computeroutput">bzip2-1.0.X</code>, 2241 <code class="computeroutput">0.9.5</code> and 2242 <code class="computeroutput">0.9.0</code> use exactly the same file 2243 format as the original version, 2244 <code class="computeroutput">bzip2-0.1</code>. This decision was 2245 made in the interests of stability. Creating yet another 2246 incompatible compressed file format would create further 2247 confusion and disruption for users.</p> 2248 <p>Nevertheless, this is not a painless decision. Development 2249 work since the release of 2250 <code class="computeroutput">bzip2-0.1</code> in August 1997 has 2251 shown complexities in the file format which slow down 2252 decompression and, in retrospect, are unnecessary. These 2253 are:</p> 2254 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: bullet; "> 2255 <li class="listitem" style="list-style-type: disc"><p>The run-length encoder, which is the first of the 2256 compression transformations, is entirely irrelevant. The 2257 original purpose was to protect the sorting algorithm from the 2258 very worst case input: a string of repeated symbols. But 2259 algorithm steps Q6a and Q6b in the original Burrows-Wheeler 2260 technical report (SRC-124) show how repeats can be handled 2261 without difficulty in block sorting.</p></li> 2262 <li class="listitem" style="list-style-type: disc"> 2263 <p>The randomisation mechanism doesn't really need to be 2264 there. Udi Manber and Gene Myers published a suffix array 2265 construction algorithm a few years back, which can be employed 2266 to sort any block, no matter how repetitive, in O(N log N) 2267 time. Subsequent work by Kunihiko Sadakane has produced a 2268 derivative O(N (log N)^2) algorithm which usually outperforms 2269 the Manber-Myers algorithm.</p> 2270 <p>I could have changed to Sadakane's algorithm, but I find 2271 it to be slower than <code class="computeroutput">bzip2</code>'s 2272 existing algorithm for most inputs, and the randomisation 2273 mechanism protects adequately against bad cases. I didn't 2274 think it was a good tradeoff to make. Partly this is due to 2275 the fact that I was not flooded with email complaints about 2276 <code class="computeroutput">bzip2-0.1</code>'s performance on 2277 repetitive data, so perhaps it isn't a problem for real 2278 inputs.</p> 2279 <p>Probably the best long-term solution, and the one I have 2280 incorporated into 0.9.5 and above, is to use the existing 2281 sorting algorithm initially, and fall back to a O(N (log N)^2) 2282 algorithm if the standard algorithm gets into 2283 difficulties.</p> 2284 </li> 2285 <li class="listitem" style="list-style-type: disc"><p>The compressed file format was never designed to be 2286 handled by a library, and I have had to jump though some hoops 2287 to produce an efficient implementation of decompression. It's 2288 a bit hairy. Try passing 2289 <code class="computeroutput">decompress.c</code> through the C 2290 preprocessor and you'll see what I mean. Much of this 2291 complexity could have been avoided if the compressed size of 2292 each block of data was recorded in the data stream.</p></li> 2293 <li class="listitem" style="list-style-type: disc"><p>An Adler-32 checksum, rather than a CRC32 checksum, 2294 would be faster to compute.</p></li> 2295 </ul></div> 2296 <p>It would be fair to say that the 2297 <code class="computeroutput">bzip2</code> format was frozen before I 2298 properly and fully understood the performance consequences of 2299 doing so.</p> 2300 <p>Improvements which I was able to incorporate into 0.9.0, 2301 despite using the same file format, are:</p> 2302 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: bullet; "> 2303 <li class="listitem" style="list-style-type: disc"><p>Single array implementation of the inverse BWT. This 2304 significantly speeds up decompression, presumably because it 2305 reduces the number of cache misses.</p></li> 2306 <li class="listitem" style="list-style-type: disc"><p>Faster inverse MTF transform for large MTF values. 2307 The new implementation is based on the notion of sliding blocks 2308 of values.</p></li> 2309 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzip2-0.9.0</code> now reads 2310 and writes files with <code class="computeroutput">fread</code> 2311 and <code class="computeroutput">fwrite</code>; version 0.1 used 2312 <code class="computeroutput">putc</code> and 2313 <code class="computeroutput">getc</code>. Duh! Well, you live 2314 and learn.</p></li> 2315 </ul></div> 2316 <p>Further ahead, it would be nice to be able to do random 2317 access into files. This will require some careful design of 2318 compressed file formats.</p> 2319 </div> 2320 <div class="sect1"> 2321 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 2322 <a name="port-issues"></a>4.2.Portability issues</h2></div></div></div> 2323 <p>After some consideration, I have decided not to use GNU 2324 <code class="computeroutput">autoconf</code> to configure 0.9.5 or 2325 1.0.</p> 2326 <p><code class="computeroutput">autoconf</code>, admirable and 2327 wonderful though it is, mainly assists with portability problems 2328 between Unix-like platforms. But 2329 <code class="computeroutput">bzip2</code> doesn't have much in the 2330 way of portability problems on Unix; most of the difficulties 2331 appear when porting to the Mac, or to Microsoft's operating 2332 systems. <code class="computeroutput">autoconf</code> doesn't help 2333 in those cases, and brings in a whole load of new 2334 complexity.</p> 2335 <p>Most people should be able to compile the library and 2336 program under Unix straight out-of-the-box, so to speak, 2337 especially if you have a version of GNU C available.</p> 2338 <p>There are a couple of 2339 <code class="computeroutput">__inline__</code> directives in the 2340 code. GNU C (<code class="computeroutput">gcc</code>) should be 2341 able to handle them. If you're not using GNU C, your C compiler 2342 shouldn't see them at all. If your compiler does, for some 2343 reason, see them and doesn't like them, just 2344 <code class="computeroutput">#define</code> 2345 <code class="computeroutput">__inline__</code> to be 2346 <code class="computeroutput">/* */</code>. One easy way to do this 2347 is to compile with the flag 2348 <code class="computeroutput">-D__inline__=</code>, which should be 2349 understood by most Unix compilers.</p> 2350 <p>If you still have difficulties, try compiling with the 2351 macro <code class="computeroutput">BZ_STRICT_ANSI</code> defined. 2352 This should enable you to build the library in a strictly ANSI 2353 compliant environment. Building the program itself like this is 2354 dangerous and not supported, since you remove 2355 <code class="computeroutput">bzip2</code>'s checks against 2356 compressing directories, symbolic links, devices, and other 2357 not-really-a-file entities. This could cause filesystem 2358 corruption!</p> 2359 <p>One other thing: if you create a 2360 <code class="computeroutput">bzip2</code> binary for public distribution, 2361 please consider linking it statically (<code class="computeroutput">gcc 2362 -static</code>). This avoids all sorts of library-version 2363 issues that others may encounter later on.</p> 2364 <p>If you build <code class="computeroutput">bzip2</code> on 2365 Win32, you must set <code class="computeroutput">BZ_UNIX</code> to 0 2366 and <code class="computeroutput">BZ_LCCWIN32</code> to 1, in the 2367 file <code class="computeroutput">bzip2.c</code>, before compiling. 2368 Otherwise the resulting binary won't work correctly.</p> 2369 </div> 2370 <div class="sect1"> 2371 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 2372 <a name="bugs"></a>4.3.Reporting bugs</h2></div></div></div> 2373 <p>I tried pretty hard to make sure 2374 <code class="computeroutput">bzip2</code> is bug free, both by 2375 design and by testing. Hopefully you'll never need to read this 2376 section for real.</p> 2377 <p>Nevertheless, if <code class="computeroutput">bzip2</code> dies 2378 with a segmentation fault, a bus error or an internal assertion 2379 failure, it will ask you to email me a bug report. Experience from 2380 years of feedback of bzip2 users indicates that almost all these 2381 problems can be traced to either compiler bugs or hardware 2382 problems.</p> 2383 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: bullet; "> 2384 <li class="listitem" style="list-style-type: disc"> 2385 <p>Recompile the program with no optimisation, and 2386 see if it works. And/or try a different compiler. I heard all 2387 sorts of stories about various flavours of GNU C (and other 2388 compilers) generating bad code for 2389 <code class="computeroutput">bzip2</code>, and I've run across two 2390 such examples myself.</p> 2391 <p>2.7.X versions of GNU C are known to generate bad code 2392 from time to time, at high optimisation levels. If you get 2393 problems, try using the flags 2394 <code class="computeroutput">-O2</code> 2395 <code class="computeroutput">-fomit-frame-pointer</code> 2396 <code class="computeroutput">-fno-strength-reduce</code>. You 2397 should specifically <span class="emphasis"><em>not</em></span> use 2398 <code class="computeroutput">-funroll-loops</code>.</p> 2399 <p>You may notice that the Makefile runs six tests as part 2400 of the build process. If the program passes all of these, it's 2401 a pretty good (but not 100%) indication that the compiler has 2402 done its job correctly.</p> 2403 </li> 2404 <li class="listitem" style="list-style-type: disc"> 2405 <p>If <code class="computeroutput">bzip2</code> 2406 crashes randomly, and the crashes are not repeatable, you may 2407 have a flaky memory subsystem. 2408 <code class="computeroutput">bzip2</code> really hammers your 2409 memory hierarchy, and if it's a bit marginal, you may get these 2410 problems. Ditto if your disk or I/O subsystem is slowly 2411 failing. Yup, this really does happen.</p> 2412 <p>Try using a different machine of the same type, and see 2413 if you can repeat the problem.</p> 2414 </li> 2415 <li class="listitem" style="list-style-type: disc"><p>This isn't really a bug, but ... If 2416 <code class="computeroutput">bzip2</code> tells you your file is 2417 corrupted on decompression, and you obtained the file via FTP, 2418 there is a possibility that you forgot to tell FTP to do a 2419 binary mode transfer. That absolutely will cause the file to 2420 be non-decompressible. You'll have to transfer it 2421 again.</p></li> 2422 </ul></div> 2423 <p>If you've incorporated 2424 <code class="computeroutput">libbzip2</code> into your own program 2425 and are getting problems, please, please, please, check that the 2426 parameters you are passing in calls to the library, are correct, 2427 and in accordance with what the documentation says is allowable. 2428 I have tried to make the library robust against such problems, 2429 but I'm sure I haven't succeeded.</p> 2430 <p>Finally, if the above comments don't help, you'll have to 2431 send me a bug report. Now, it's just amazing how many people 2432 will send me a bug report saying something like:</p> 2433 <pre class="programlisting">bzip2 crashed with segmentation fault on my machine</pre> 2434 <p>and absolutely nothing else. Needless to say, a such a 2435 report is <span class="emphasis"><em>totally, utterly, completely and 2436 comprehensively 100% useless; a waste of your time, my time, and 2437 net bandwidth</em></span>. With no details at all, there's no way 2438 I can possibly begin to figure out what the problem is.</p> 2439 <p>The rules of the game are: facts, facts, facts. Don't omit 2440 them because "oh, they won't be relevant". At the bare 2441 minimum:</p> 2442 <pre class="programlisting">Machine type. Operating system version. 2443 Exact version of bzip2 (do bzip2 -V). 2444 Exact version of the compiler used. 2445 Flags passed to the compiler.</pre> 2446 <p>However, the most important single thing that will help me 2447 is the file that you were trying to compress or decompress at the 2448 time the problem happened. Without that, my ability to do 2449 anything more than speculate about the cause, is limited.</p> 2450 </div> 2451 <div class="sect1"> 2452 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 2453 <a name="package"></a>4.4.Did you get the right package?</h2></div></div></div> 2454 <p><code class="computeroutput">bzip2</code> is a resource hog. 2455 It soaks up large amounts of CPU cycles and memory. Also, it 2456 gives very large latencies. In the worst case, you can feed many 2457 megabytes of uncompressed data into the library before getting 2458 any compressed output, so this probably rules out applications 2459 requiring interactive behaviour.</p> 2460 <p>These aren't faults of my implementation, I hope, but more 2461 an intrinsic property of the Burrows-Wheeler transform 2462 (unfortunately). Maybe this isn't what you want.</p> 2463 <p>If you want a compressor and/or library which is faster, 2464 uses less memory but gets pretty good compression, and has 2465 minimal latency, consider Jean-loup Gailly's and Mark Adler's 2466 work, <code class="computeroutput">zlib-1.2.1</code> and 2467 <code class="computeroutput">gzip-1.2.4</code>. Look for them at 2468 <a class="ulink" href="http://www.zlib.org" target="_top">http://www.zlib.org</a> and 2469 <a class="ulink" href="http://www.gzip.org" target="_top">http://www.gzip.org</a> 2470 respectively.</p> 2471 <p>For something faster and lighter still, you might try Markus F 2472 X J Oberhumer's <code class="computeroutput">LZO</code> real-time 2473 compression/decompression library, at 2474 <a class="ulink" href="http://www.oberhumer.com/opensource" target="_top">http://www.oberhumer.com/opensource</a>.</p> 2475 </div> 2476 <div class="sect1"> 2477 <div class="titlepage"><div><div><h2 class="title" style="clear: both"> 2478 <a name="reading"></a>4.5.Further Reading</h2></div></div></div> 2479 <p><code class="computeroutput">bzip2</code> is not research 2480 work, in the sense that it doesn't present any new ideas. 2481 Rather, it's an engineering exercise based on existing 2482 ideas.</p> 2483 <p>Four documents describe essentially all the ideas behind 2484 <code class="computeroutput">bzip2</code>:</p> 2485 <div class="literallayout"><p>MichaelBurrowsandD.J.Wheeler:<br> 2486 "Ablock-sortinglosslessdatacompressionalgorithm"<br> 2487 10thMay1994.<br> 2488 DigitalSRCResearchReport124.<br> 2489 ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz<br> 2490 Ifyouhavetroublefindingit,trysearchingatthe<br> 2491 NewZealandDigitalLibrary,http://www.nzdl.org.<br> 2492 <br> 2493 DanielS.HirschbergandDebraA.LeLewer<br> 2494 "EfficientDecodingofPrefixCodes"<br> 2495 CommunicationsoftheACM,April1990,Vol33,Number4.<br> 2496 Youmightbeabletogetanelectroniccopyofthis<br> 2497 fromtheACMDigitalLibrary.<br> 2498 <br> 2499 DavidJ.Wheeler<br> 2500 Programbred3.candaccompanyingdocumentbred3.ps.<br> 2501 Thiscontainstheideabehindthemulti-tableHuffmancodingscheme.<br> 2502 ftp://ftp.cl.cam.ac.uk/users/djw3/<br> 2503 <br> 2504 JonL.BentleyandRobertSedgewick<br> 2505 "FastAlgorithmsforSortingandSearchingStrings"<br> 2506 AvailablefromSedgewick'swebpage,<br> 2507 www.cs.princeton.edu/~rs<br> 2508 </p></div> 2509 <p>The following paper gives valuable additional insights into 2510 the algorithm, but is not immediately the basis of any code used 2511 in bzip2.</p> 2512 <div class="literallayout"><p>PeterFenwick:<br> 2513 BlockSortingTextCompression<br> 2514 Proceedingsofthe19thAustralasianComputerScienceConference,<br> 2515 Melbourne,Australia.Jan31-Feb2,1996.<br> 2516 ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps</p></div> 2517 <p>Kunihiko Sadakane's sorting algorithm, mentioned above, is 2518 available from:</p> 2519 <div class="literallayout"><p>http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz<br> 2520 </p></div> 2521 <p>The Manber-Myers suffix array construction algorithm is 2522 described in a paper available from:</p> 2523 <div class="literallayout"><p>http://www.cs.arizona.edu/people/gene/PAPERS/suffix.ps<br> 2524 </p></div> 2525 <p>Finally, the following papers document some 2526 investigations I made into the performance of sorting 2527 and decompression algorithms:</p> 2528 <div class="literallayout"><p>JulianSeward<br> 2529 OnthePerformanceofBWTSortingAlgorithms<br> 2530 ProceedingsoftheIEEEDataCompressionConference2000<br> 2531 Snowbird,Utah.28-30March2000.<br> 2532 <br> 2533 JulianSeward<br> 2534 Space-timeTradeoffsintheInverseB-WTransform<br> 2535 ProceedingsoftheIEEEDataCompressionConference2001<br> 2536 Snowbird,Utah.27-29March2001.<br> 2537 </p></div> 2538 </div> 2539 </div> 2540 </div></body> 2541 </html> 2542
Indexes created Tue Mar 03 05:31:39 UTC 2026