Using liblzma securely ---------------------- 0. Introduction This document discusses how to use liblzma securely. There are issues that don't apply to zlib or libbzip2, so reading this document is strongly recommended even for those who are very familiar with zlib or libbzip2. While making liblzma itself as secure as possible is essential, it's out of scope of this document. 1. Memory usage The memory usage of liblzma varies a lot. 1.1. Problem sources 1.1.1. Block coder The memory requirements of Block encoder depend on the used filters and their settings. The memory requirements of the Block decoder depend on the which filters and with which filter settings the Block was encoded. Usually the memory requirements of a decoder are equal or less than the requirements of the encoder with the same settings. While the typical memory requirements to decode a Block is from a few hundred kilobytes to tens of megabytes, a maliciously constructed files can require a lot more RAM to decode. With the current filters, the maximum amount is about 7 GiB. If you use multi-threaded decoding, every Block can require this amount of RAM, thus a four-threaded decoder could suddenly try to allocate 28 GiB of RAM. If you don't limit the maximum memory usage in any way, and there are no resource limits set on the operating system side, one malicious input file can run the system out of memory, or at least make it swap badly for a long time. This is exceptionally bad on servers e.g. email server doing virus scanning on incoming messages. 1.1.2. Metadata decoder Multi-Block .lzma files contain at least one Metadata Block. Externally the Metadata Blocks are similar to Data Blocks, so all the issues mentioned about memory usage of Data Blocks applies to Metadata Blocks too. The uncompressed content of Metadata Blocks contain information about the Stream as a whole, and optionally some Extra Records. The information about the Stream is kept in liblzma's internal data structures in RAM. Extra Records can contain arbitrary data. They are not interpreted by liblzma, but liblzma will provide them to the application in uninterpreted form if the application wishes so. Usually the Uncompressed Size of a Metadata Block is small. Even on extreme cases, it shouldn't be much bigger than a few megabytes. Once the Metadata has been parsed into native data structures in liblzma, it usually takes a little more memory than in the encoded form. For all normal files, this is no problem, since the resulting memory usage won't be too much. The problem is that a maliciously constructed Metadata Block can contain huge amount of "information", which liblzma will try to store in its internal data structures. This may cause liblzma to allocate all the available RAM unless some kind of resource usage limits are applied. Note that the Extra Records in Metadata are always parsed but, but memory is allocated for them only if the application has requested liblzma to provide the Extra Records to the application. 1.2. Solutions If you need to decode files from untrusted sources (most people do), you must limit the memory usage to avoid denial of service (DoS) conditions caused by malicious input files. The first step is to find out how much memory you are allowed consume at maximum. This may be a hardcoded constant or derived from the available RAM; whatever is appropriate in the application. The simplest solution is to use setrlimit() if the kernel supports RLIMIT_AS, which limits the memory usage of the whole process. For more portable and fine-grained limiting, you can use memory limiter functions found from . 1.2.1. Encoder lzma_memory_usage() will give you a rough estimate about the memory usage of the given filter chain. To dramatically simplify the internal implementation, this function doesn't take into account all the small helper data structures needed in various places; only the structures with significant memory usage are taken into account. Still, the accuracy of this function should be well within a mebibyte. The Subblock filter is a special case. If a Subfilter has been specified, it isn't taken into account when lzma_memory_usage() calculates the memory usage. You need to calculate the memory usage of the Subfilter separately. Keeping track of Blocks in a Multi-Block Stream takes a few dozen bytes of RAM per Block (size of the lzma_index structure plus overhead of malloc()). It isn't a good idea to put tens of thousands of Blocks into a Stream unless you have a very good reason to do so (compressed dictionary could be an example of such situation). Also keep the number and sizes of Extra Records sane. If you produce the list of Extra Records automatically from some untrusted source, you should not only validate the content of these Records, but also their memory usage. 1.2.2. Decoder A single-threaded decoder should simply use a memory limiter and indicate an error if it runs out of memory. Memory-limiting with multi-threaded decoding is tricky. The simple solution is to divide the maximum allowed memory usage with the maximum allowed threads, and give each Block decoder their own independent lzma_memory_limiter. The drawback is that if one Block needs notably more RAM than any other Block, the decoder will run out of memory when in reality there would be plenty of free RAM. An attractive alternative would be using shared lzma_memory_limiter. Depending on the application and the expected type of input, this may either be the best solution or a source of hard-to-repeat problems. Consider the following requirements: - You use a maximum of n threads. - x(i) is the decoder memory requirements of the Block number i in an expected input Stream. - The memory limiter is set to higher value than the sum of n highest values x(i). (If you are better at explaining the above conditions, please contribute your improved version.) If the above conditions aren't met, it is possible that the decoding will fail unpredictably. That is, on the same machine using the same settings, the decoding may sometimes succeed and sometimes fail. This is because sometimes threads may run so that the Blocks with highest memory usage are tried to be decoded at the same time. Most .lzma files have all the Blocks encoded with identical settings, or at least the memory usage won't vary dramatically. That's why most multi-threaded decoders probably want to use the simple "separate lzma_memory_limiter for each thread" solution, possibly falling back to single-threaded mode in case the per-thread memory limits aren't enough in multi-threaded mode. FIXME: Memory usage of Stream info. [ ] 2. Huge uncompressed output 2.1. Data Blocks Decoding a tiny .lzma file can produce huge amount of uncompressed output. There is an example file of 45 bytes, which decodes to 64 PiB (that's 2^56 bytes). Uncompressing such a file to disk is likely to fill even a bigger disk array. If the data is written to a pipe, it may not fill the disk, but would still take very long time to finish. To avoid denial of service conditions caused by huge amount of uncompressed output, applications using liblzma should use some method to limit the amount of output produced. The exact method depends on the application. All valid .lzma Streams make it possible to find out the uncompressed size of the Stream without actually uncompressing the data. This information is available in at least one of the Metadata Blocks. Once the uncompressed size is parsed, the decoder can verify that it doesn't exceed certain limits (e.g. available disk space). When the uncompressed size is known, the decoder can actively keep track of the amount of output produced so far, and that it doesn't exceed the known uncompressed size. If it does exceed, the file is known to be corrupt and an error should be indicated without continuing to decode the rest of the file. Unfortunately, finding the uncompressed size beforehand is often possible only in non-streamed mode, because the needed information could be in the Footer Metdata Block, which (obviously) is at the end of the Stream. In purely streamed mode decoding, one may need to use some rough arbitrary limits to prevent the problems described in the beginning of this section. 2.2. Metadata Metadata is stored in Metadata Blocks, which are very similar to Data Blocks. Thus, the uncompressed size can be huge just like with Data Blocks. The difference is, that the contents of Metadata Blocks aren't given to the application as is, but parsed by liblzma. Still, reading through a huge Metadata can take very long time, effectively creating a denial of service like piping decoded a Data Block to another process would do. At first it would seem that using a memory limiter would prevent this issue as a side effect. But it does so only if the application requests liblzma to allocate the Extra Records and provide them to the application. If Extra Records aren't requested, they aren't allocated either. Still, the Extra Records are being read through to validate that the Metadata is in proper format. The solution is to limit the Uncompressed Size of a Metadata Block to some relatively large value. This will make liblzma to give an error when the given limit is reached.