From 8f6f4773bc78ee15978657d78a7b44d462bb4416 Mon Sep 17 00:00:00 2001 From: Jay Berkenbilt Date: Sat, 10 Oct 2009 17:43:06 +0000 Subject: remove files not needed for building git-svn-id: svn+q:///qpdf/trunk@768 71b93d88-0707-0410-a8cf-f5a4172ac649 --- external-libs/zlib/doc/algorithm.txt | 209 ----------------------------------- external-libs/zlib/doc/txtvsbin.txt | 107 ------------------ 2 files changed, 316 deletions(-) delete mode 100644 external-libs/zlib/doc/algorithm.txt delete mode 100644 external-libs/zlib/doc/txtvsbin.txt (limited to 'external-libs/zlib/doc') diff --git a/external-libs/zlib/doc/algorithm.txt b/external-libs/zlib/doc/algorithm.txt deleted file mode 100644 index b022dde3..00000000 --- a/external-libs/zlib/doc/algorithm.txt +++ /dev/null @@ -1,209 +0,0 @@ -1. Compression algorithm (deflate) - -The deflation algorithm used by gzip (also zip and zlib) is a variation of -LZ77 (Lempel-Ziv 1977, see reference below). It finds duplicated strings in -the input data. The second occurrence of a string is replaced by a -pointer to the previous string, in the form of a pair (distance, -length). Distances are limited to 32K bytes, and lengths are limited -to 258 bytes. When a string does not occur anywhere in the previous -32K bytes, it is emitted as a sequence of literal bytes. (In this -description, `string' must be taken as an arbitrary sequence of bytes, -and is not restricted to printable characters.) - -Literals or match lengths are compressed with one Huffman tree, and -match distances are compressed with another tree. The trees are stored -in a compact form at the start of each block. The blocks can have any -size (except that the compressed data for one block must fit in -available memory). A block is terminated when deflate() determines that -it would be useful to start another block with fresh trees. (This is -somewhat similar to the behavior of LZW-based _compress_.) - -Duplicated strings are found using a hash table. All input strings of -length 3 are inserted in the hash table. A hash index is computed for -the next 3 bytes. If the hash chain for this index is not empty, all -strings in the chain are compared with the current input string, and -the longest match is selected. - -The hash chains are searched starting with the most recent strings, to -favor small distances and thus take advantage of the Huffman encoding. -The hash chains are singly linked. There are no deletions from the -hash chains, the algorithm simply discards matches that are too old. - -To avoid a worst-case situation, very long hash chains are arbitrarily -truncated at a certain length, determined by a runtime option (level -parameter of deflateInit). So deflate() does not always find the longest -possible match but generally finds a match which is long enough. - -deflate() also defers the selection of matches with a lazy evaluation -mechanism. After a match of length N has been found, deflate() searches for -a longer match at the next input byte. If a longer match is found, the -previous match is truncated to a length of one (thus producing a single -literal byte) and the process of lazy evaluation begins again. Otherwise, -the original match is kept, and the next match search is attempted only N -steps later. - -The lazy match evaluation is also subject to a runtime parameter. If -the current match is long enough, deflate() reduces the search for a longer -match, thus speeding up the whole process. If compression ratio is more -important than speed, deflate() attempts a complete second search even if -the first match is already long enough. - -The lazy match evaluation is not performed for the fastest compression -modes (level parameter 1 to 3). For these fast modes, new strings -are inserted in the hash table only when no match was found, or -when the match is not too long. This degrades the compression ratio -but saves time since there are both fewer insertions and fewer searches. - - -2. Decompression algorithm (inflate) - -2.1 Introduction - -The key question is how to represent a Huffman code (or any prefix code) so -that you can decode fast. The most important characteristic is that shorter -codes are much more common than longer codes, so pay attention to decoding the -short codes fast, and let the long codes take longer to decode. - -inflate() sets up a first level table that covers some number of bits of -input less than the length of longest code. It gets that many bits from the -stream, and looks it up in the table. The table will tell if the next -code is that many bits or less and how many, and if it is, it will tell -the value, else it will point to the next level table for which inflate() -grabs more bits and tries to decode a longer code. - -How many bits to make the first lookup is a tradeoff between the time it -takes to decode and the time it takes to build the table. If building the -table took no time (and if you had infinite memory), then there would only -be a first level table to cover all the way to the longest code. However, -building the table ends up taking a lot longer for more bits since short -codes are replicated many times in such a table. What inflate() does is -simply to make the number of bits in the first table a variable, and then -to set that variable for the maximum speed. - -For inflate, which has 286 possible codes for the literal/length tree, the size -of the first table is nine bits. Also the distance trees have 30 possible -values, and the size of the first table is six bits. Note that for each of -those cases, the table ended up one bit longer than the ``average'' code -length, i.e. the code length of an approximately flat code which would be a -little more than eight bits for 286 symbols and a little less than five bits -for 30 symbols. - - -2.2 More details on the inflate table lookup - -Ok, you want to know what this cleverly obfuscated inflate tree actually -looks like. You are correct that it's not a Huffman tree. It is simply a -lookup table for the first, let's say, nine bits of a Huffman symbol. The -symbol could be as short as one bit or as long as 15 bits. If a particular -symbol is shorter than nine bits, then that symbol's translation is duplicated -in all those entries that start with that symbol's bits. For example, if the -symbol is four bits, then it's duplicated 32 times in a nine-bit table. If a -symbol is nine bits long, it appears in the table once. - -If the symbol is longer than nine bits, then that entry in the table points -to another similar table for the remaining bits. Again, there are duplicated -entries as needed. The idea is that most of the time the symbol will be short -and there will only be one table look up. (That's whole idea behind data -compression in the first place.) For the less frequent long symbols, there -will be two lookups. If you had a compression method with really long -symbols, you could have as many levels of lookups as is efficient. For -inflate, two is enough. - -So a table entry either points to another table (in which case nine bits in -the above example are gobbled), or it contains the translation for the symbol -and the number of bits to gobble. Then you start again with the next -ungobbled bit. - -You may wonder: why not just have one lookup table for how ever many bits the -longest symbol is? The reason is that if you do that, you end up spending -more time filling in duplicate symbol entries than you do actually decoding. -At least for deflate's output that generates new trees every several 10's of -kbytes. You can imagine that filling in a 2^15 entry table for a 15-bit code -would take too long if you're only decoding several thousand symbols. At the -other extreme, you could make a new table for every bit in the code. In fact, -that's essentially a Huffman tree. But then you spend two much time -traversing the tree while decoding, even for short symbols. - -So the number of bits for the first lookup table is a trade of the time to -fill out the table vs. the time spent looking at the second level and above of -the table. - -Here is an example, scaled down: - -The code being decoded, with 10 symbols, from 1 to 6 bits long: - -A: 0 -B: 10 -C: 1100 -D: 11010 -E: 11011 -F: 11100 -G: 11101 -H: 11110 -I: 111110 -J: 111111 - -Let's make the first table three bits long (eight entries): - -000: A,1 -001: A,1 -010: A,1 -011: A,1 -100: B,2 -101: B,2 -110: -> table X (gobble 3 bits) -111: -> table Y (gobble 3 bits) - -Each entry is what the bits decode as and how many bits that is, i.e. how -many bits to gobble. Or the entry points to another table, with the number of -bits to gobble implicit in the size of the table. - -Table X is two bits long since the longest code starting with 110 is five bits -long: - -00: C,1 -01: C,1 -10: D,2 -11: E,2 - -Table Y is three bits long since the longest code starting with 111 is six -bits long: - -000: F,2 -001: F,2 -010: G,2 -011: G,2 -100: H,2 -101: H,2 -110: I,3 -111: J,3 - -So what we have here are three tables with a total of 20 entries that had to -be constructed. That's compared to 64 entries for a single table. Or -compared to 16 entries for a Huffman tree (six two entry tables and one four -entry table). Assuming that the code ideally represents the probability of -the symbols, it takes on the average 1.25 lookups per symbol. That's compared -to one lookup for the single table, or 1.66 lookups per symbol for the -Huffman tree. - -There, I think that gives you a picture of what's going on. For inflate, the -meaning of a particular symbol is often more than just a letter. It can be a -byte (a "literal"), or it can be either a length or a distance which -indicates a base value and a number of bits to fetch after the code that is -added to the base value. Or it might be the special end-of-block code. The -data structures created in inftrees.c try to encode all that information -compactly in the tables. - - -Jean-loup Gailly Mark Adler -jloup@gzip.org madler@alumni.caltech.edu - - -References: - -[LZ77] Ziv J., Lempel A., ``A Universal Algorithm for Sequential Data -Compression,'' IEEE Transactions on Information Theory, Vol. 23, No. 3, -pp. 337-343. - -``DEFLATE Compressed Data Format Specification'' available in -http://www.ietf.org/rfc/rfc1951.txt diff --git a/external-libs/zlib/doc/txtvsbin.txt b/external-libs/zlib/doc/txtvsbin.txt deleted file mode 100644 index 3d0f0634..00000000 --- a/external-libs/zlib/doc/txtvsbin.txt +++ /dev/null @@ -1,107 +0,0 @@ -A Fast Method for Identifying Plain Text Files -============================================== - - -Introduction ------------- - -Given a file coming from an unknown source, it is sometimes desirable -to find out whether the format of that file is plain text. Although -this may appear like a simple task, a fully accurate detection of the -file type requires heavy-duty semantic analysis on the file contents. -It is, however, possible to obtain satisfactory results by employing -various heuristics. - -Previous versions of PKZip and other zip-compatible compression tools -were using a crude detection scheme: if more than 80% (4/5) of the bytes -found in a certain buffer are within the range [7..127], the file is -labeled as plain text, otherwise it is labeled as binary. A prominent -limitation of this scheme is the restriction to Latin-based alphabets. -Other alphabets, like Greek, Cyrillic or Asian, make extensive use of -the bytes within the range [128..255], and texts using these alphabets -are most often misidentified by this scheme; in other words, the rate -of false negatives is sometimes too high, which means that the recall -is low. Another weakness of this scheme is a reduced precision, due to -the false positives that may occur when binary files containing large -amounts of textual characters are misidentified as plain text. - -In this article we propose a new, simple detection scheme that features -a much increased precision and a near-100% recall. This scheme is -designed to work on ASCII, Unicode and other ASCII-derived alphabets, -and it handles single-byte encodings (ISO-8859, MacRoman, KOI8, etc.) -and variable-sized encodings (ISO-2022, UTF-8, etc.). Wider encodings -(UCS-2/UTF-16 and UCS-4/UTF-32) are not handled, however. - - -The Algorithm -------------- - -The algorithm works by dividing the set of bytecodes [0..255] into three -categories: -- The white list of textual bytecodes: - 9 (TAB), 10 (LF), 13 (CR), 32 (SPACE) to 255. -- The gray list of tolerated bytecodes: - 7 (BEL), 8 (BS), 11 (VT), 12 (FF), 26 (SUB), 27 (ESC). -- The black list of undesired, non-textual bytecodes: - 0 (NUL) to 6, 14 to 31. - -If a file contains at least one byte that belongs to the white list and -no byte that belongs to the black list, then the file is categorized as -plain text; otherwise, it is categorized as binary. (The boundary case, -when the file is empty, automatically falls into the latter category.) - - -Rationale ---------- - -The idea behind this algorithm relies on two observations. - -The first observation is that, although the full range of 7-bit codes -[0..127] is properly specified by the ASCII standard, most control -characters in the range [0..31] are not used in practice. The only -widely-used, almost universally-portable control codes are 9 (TAB), -10 (LF) and 13 (CR). There are a few more control codes that are -recognized on a reduced range of platforms and text viewers/editors: -7 (BEL), 8 (BS), 11 (VT), 12 (FF), 26 (SUB) and 27 (ESC); but these -codes are rarely (if ever) used alone, without being accompanied by -some printable text. Even the newer, portable text formats such as -XML avoid using control characters outside the list mentioned here. - -The second observation is that most of the binary files tend to contain -control characters, especially 0 (NUL). Even though the older text -detection schemes observe the presence of non-ASCII codes from the range -[128..255], the precision rarely has to suffer if this upper range is -labeled as textual, because the files that are genuinely binary tend to -contain both control characters and codes from the upper range. On the -other hand, the upper range needs to be labeled as textual, because it -is used by virtually all ASCII extensions. In particular, this range is -used for encoding non-Latin scripts. - -Since there is no counting involved, other than simply observing the -presence or the absence of some byte values, the algorithm produces -consistent results, regardless what alphabet encoding is being used. -(If counting were involved, it could be possible to obtain different -results on a text encoded, say, using ISO-8859-16 versus UTF-8.) - -There is an extra category of plain text files that are "polluted" with -one or more black-listed codes, either by mistake or by peculiar design -considerations. In such cases, a scheme that tolerates a small fraction -of black-listed codes would provide an increased recall (i.e. more true -positives). This, however, incurs a reduced precision overall, since -false positives are more likely to appear in binary files that contain -large chunks of textual data. Furthermore, "polluted" plain text should -be regarded as binary by general-purpose text detection schemes, because -general-purpose text processing algorithms might not be applicable. -Under this premise, it is safe to say that our detection method provides -a near-100% recall. - -Experiments have been run on many files coming from various platforms -and applications. We tried plain text files, system logs, source code, -formatted office documents, compiled object code, etc. The results -confirm the optimistic assumptions about the capabilities of this -algorithm. - - --- -Cosmin Truta -Last updated: 2006-May-28 -- cgit v1.2.3-70-g09d2