QSHAPE_README.html revision 1.1.1.1.4.2
1 1.1.1.1.4.2 matt <!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN" 2 1.1.1.1.4.2 matt "http://www.w3.org/TR/html4/loose.dtd"> 3 1.1.1.1.4.2 matt 4 1.1.1.1.4.2 matt <html> 5 1.1.1.1.4.2 matt 6 1.1.1.1.4.2 matt <head> 7 1.1.1.1.4.2 matt 8 1.1.1.1.4.2 matt <title>Postfix Bottleneck Analysis</title> 9 1.1.1.1.4.2 matt 10 1.1.1.1.4.2 matt <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> 11 1.1.1.1.4.2 matt 12 1.1.1.1.4.2 matt </head> 13 1.1.1.1.4.2 matt 14 1.1.1.1.4.2 matt <body> 15 1.1.1.1.4.2 matt 16 1.1.1.1.4.2 matt <h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix Bottleneck Analysis</h1> 17 1.1.1.1.4.2 matt 18 1.1.1.1.4.2 matt <hr> 19 1.1.1.1.4.2 matt 20 1.1.1.1.4.2 matt <h2>Purpose of this document </h2> 21 1.1.1.1.4.2 matt 22 1.1.1.1.4.2 matt <p> This document is an introduction to Postfix queue congestion analysis. 23 1.1.1.1.4.2 matt It explains how the <a href="qshape.1.html">qshape(1)</a> program can help to track down the 24 1.1.1.1.4.2 matt reason for queue congestion. <a href="qshape.1.html">qshape(1)</a> is bundled with Postfix 25 1.1.1.1.4.2 matt 2.1 and later source code, under the "auxiliary" directory. This 26 1.1.1.1.4.2 matt document describes <a href="qshape.1.html">qshape(1)</a> as bundled with Postfix 2.4. </p> 27 1.1.1.1.4.2 matt 28 1.1.1.1.4.2 matt <p> This document covers the following topics: </p> 29 1.1.1.1.4.2 matt 30 1.1.1.1.4.2 matt <ul> 31 1.1.1.1.4.2 matt 32 1.1.1.1.4.2 matt <li><a href="#qshape">Introducing the qshape tool</a> 33 1.1.1.1.4.2 matt 34 1.1.1.1.4.2 matt <li><a href="#trouble_shooting">Trouble shooting with qshape</a> 35 1.1.1.1.4.2 matt 36 1.1.1.1.4.2 matt <li><a href="#healthy">Example 1: Healthy queue</a> 37 1.1.1.1.4.2 matt 38 1.1.1.1.4.2 matt <li><a href="#dictionary_bounce">Example 2: Deferred queue full of 39 1.1.1.1.4.2 matt dictionary attack bounces</a></li> 40 1.1.1.1.4.2 matt 41 1.1.1.1.4.2 matt <li><a href="#active_congestion">Example 3: Congestion in the active 42 1.1.1.1.4.2 matt queue</a></li> 43 1.1.1.1.4.2 matt 44 1.1.1.1.4.2 matt <li><a href="#backlog">Example 4: High volume destination backlog</a> 45 1.1.1.1.4.2 matt 46 1.1.1.1.4.2 matt <li><a href="#queues">Postfix queue directories</a> 47 1.1.1.1.4.2 matt 48 1.1.1.1.4.2 matt <ul> 49 1.1.1.1.4.2 matt 50 1.1.1.1.4.2 matt <li> <a href="#maildrop_queue"> The "maildrop" queue </a> 51 1.1.1.1.4.2 matt 52 1.1.1.1.4.2 matt <li> <a href="#hold_queue"> The "hold" queue </a> 53 1.1.1.1.4.2 matt 54 1.1.1.1.4.2 matt <li> <a href="#incoming_queue"> The "incoming" queue </a> 55 1.1.1.1.4.2 matt 56 1.1.1.1.4.2 matt <li> <a href="#active_queue"> The "active" queue </a> 57 1.1.1.1.4.2 matt 58 1.1.1.1.4.2 matt <li> <a href="#deferred_queue"> The "deferred" queue </a> 59 1.1.1.1.4.2 matt 60 1.1.1.1.4.2 matt </ul> 61 1.1.1.1.4.2 matt 62 1.1.1.1.4.2 matt <li><a href="#credits">Credits</a> 63 1.1.1.1.4.2 matt 64 1.1.1.1.4.2 matt </ul> 65 1.1.1.1.4.2 matt 66 1.1.1.1.4.2 matt <h2><a name="qshape">Introducing the qshape tool</a></h2> 67 1.1.1.1.4.2 matt 68 1.1.1.1.4.2 matt <p> When mail is draining slowly or the queue is unexpectedly large, 69 1.1.1.1.4.2 matt run <a href="qshape.1.html">qshape(1)</a> as the super-user (root) to help zero in on the problem. 70 1.1.1.1.4.2 matt The <a href="qshape.1.html">qshape(1)</a> program displays a tabular view of the Postfix queue 71 1.1.1.1.4.2 matt contents. </p> 72 1.1.1.1.4.2 matt 73 1.1.1.1.4.2 matt <ul> 74 1.1.1.1.4.2 matt 75 1.1.1.1.4.2 matt <li> <p> On the horizontal axis, it displays the queue age with 76 1.1.1.1.4.2 matt fine granularity for recent messages and (geometrically) less fine 77 1.1.1.1.4.2 matt granularity for older messages. </p> 78 1.1.1.1.4.2 matt 79 1.1.1.1.4.2 matt <li> <p> The vertical axis displays the destination (or with the 80 1.1.1.1.4.2 matt "-s" switch the sender) domain. Domains with the most messages are 81 1.1.1.1.4.2 matt listed first. </p> 82 1.1.1.1.4.2 matt 83 1.1.1.1.4.2 matt </ul> 84 1.1.1.1.4.2 matt 85 1.1.1.1.4.2 matt <p> For example, in the output below we see the top 10 lines of 86 1.1.1.1.4.2 matt the (mostly forged) sender domain distribution for captured spam 87 1.1.1.1.4.2 matt in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>: </p> 88 1.1.1.1.4.2 matt 89 1.1.1.1.4.2 matt <blockquote> 90 1.1.1.1.4.2 matt <pre> 91 1.1.1.1.4.2 matt $ qshape -s hold | head 92 1.1.1.1.4.2 matt T 5 10 20 40 80 160 320 640 1280 1280+ 93 1.1.1.1.4.2 matt TOTAL 486 0 0 1 0 0 2 4 20 40 419 94 1.1.1.1.4.2 matt yahoo.com 14 0 0 1 0 0 0 0 1 0 12 95 1.1.1.1.4.2 matt extremepricecuts.net 13 0 0 0 0 0 0 0 2 0 11 96 1.1.1.1.4.2 matt ms35.hinet.net 12 0 0 0 0 0 0 0 0 1 11 97 1.1.1.1.4.2 matt winnersdaily.net 12 0 0 0 0 0 0 0 2 0 10 98 1.1.1.1.4.2 matt hotmail.com 11 0 0 0 0 0 0 0 0 1 10 99 1.1.1.1.4.2 matt worldnet.fr 6 0 0 0 0 0 0 0 0 0 6 100 1.1.1.1.4.2 matt ms41.hinet.net 6 0 0 0 0 0 0 0 0 0 6 101 1.1.1.1.4.2 matt osn.de 5 0 0 0 0 0 1 0 0 0 4 102 1.1.1.1.4.2 matt </pre> 103 1.1.1.1.4.2 matt </blockquote> 104 1.1.1.1.4.2 matt 105 1.1.1.1.4.2 matt <ul> 106 1.1.1.1.4.2 matt 107 1.1.1.1.4.2 matt <li> <p> The "T" column shows the total (in this case sender) count 108 1.1.1.1.4.2 matt for each domain. The columns with numbers above them, show counts 109 1.1.1.1.4.2 matt for messages aged fewer than that many minutes, but not younger 110 1.1.1.1.4.2 matt than the age limit for the previous column. The row labeled "TOTAL" 111 1.1.1.1.4.2 matt shows the total count for all domains. </p> 112 1.1.1.1.4.2 matt 113 1.1.1.1.4.2 matt <li> <p> In this example, there are 14 messages allegedly from 114 1.1.1.1.4.2 matt yahoo.com, 1 between 10 and 20 minutes old, 1 between 320 and 640 115 1.1.1.1.4.2 matt minutes old and 12 older than 1280 minutes (1440 minutes in a day). 116 1.1.1.1.4.2 matt </p> 117 1.1.1.1.4.2 matt 118 1.1.1.1.4.2 matt </ul> 119 1.1.1.1.4.2 matt 120 1.1.1.1.4.2 matt <p> When the output is a terminal intermediate results showing the top 20 121 1.1.1.1.4.2 matt domains (-n option) are displayed after every 1000 messages (-N option) 122 1.1.1.1.4.2 matt and the final output also shows only the top 20 domains. This makes 123 1.1.1.1.4.2 matt qshape useful even when the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> is very large and it may 124 1.1.1.1.4.2 matt otherwise take prohibitively long to read the entire <a href="QSHAPE_README.html#deferred_queue">deferred queue</a>. </p> 125 1.1.1.1.4.2 matt 126 1.1.1.1.4.2 matt <p> By default, qshape shows statistics for the union of both the 127 1.1.1.1.4.2 matt <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queues</a> which are the most relevant queues to 128 1.1.1.1.4.2 matt look at when analyzing performance. </p> 129 1.1.1.1.4.2 matt 130 1.1.1.1.4.2 matt <p> One can request an alternate list of queues: </p> 131 1.1.1.1.4.2 matt 132 1.1.1.1.4.2 matt <blockquote> 133 1.1.1.1.4.2 matt <pre> 134 1.1.1.1.4.2 matt $ qshape deferred 135 1.1.1.1.4.2 matt $ qshape incoming active deferred 136 1.1.1.1.4.2 matt </pre> 137 1.1.1.1.4.2 matt </blockquote> 138 1.1.1.1.4.2 matt 139 1.1.1.1.4.2 matt <p> this will show the age distribution of the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> or 140 1.1.1.1.4.2 matt the union of the incoming active and <a href="QSHAPE_README.html#deferred_queue">deferred queues</a>. </p> 141 1.1.1.1.4.2 matt 142 1.1.1.1.4.2 matt <p> Command line options control the number of display "buckets", 143 1.1.1.1.4.2 matt the age limit for the smallest bucket, display of parent domain 144 1.1.1.1.4.2 matt counts and so on. The "-h" option outputs a summary of the available 145 1.1.1.1.4.2 matt switches. </p> 146 1.1.1.1.4.2 matt 147 1.1.1.1.4.2 matt <h2><a name="trouble_shooting">Trouble shooting with qshape</a> 148 1.1.1.1.4.2 matt </h2> 149 1.1.1.1.4.2 matt 150 1.1.1.1.4.2 matt <p> Large numbers in the qshape output represent a large number of 151 1.1.1.1.4.2 matt messages that are destined to (or alleged to come from) a particular 152 1.1.1.1.4.2 matt domain. It should be possible to tell at a glance which domains 153 1.1.1.1.4.2 matt dominate the queue sender or recipient counts, approximately when 154 1.1.1.1.4.2 matt a burst of mail started, and when it stopped. </p> 155 1.1.1.1.4.2 matt 156 1.1.1.1.4.2 matt <p> The problem destinations or sender domains appear near the top 157 1.1.1.1.4.2 matt left corner of the output table. Remember that the <a href="QSHAPE_README.html#active_queue">active queue</a> 158 1.1.1.1.4.2 matt can accommodate up to 20000 ($<a href="postconf.5.html#qmgr_message_active_limit">qmgr_message_active_limit</a>) messages. 159 1.1.1.1.4.2 matt To check whether this limit has been reached, use: </p> 160 1.1.1.1.4.2 matt 161 1.1.1.1.4.2 matt <blockquote> 162 1.1.1.1.4.2 matt <pre> 163 1.1.1.1.4.2 matt $ qshape -s active <i>(show sender statistics)</i> 164 1.1.1.1.4.2 matt </pre> 165 1.1.1.1.4.2 matt </blockquote> 166 1.1.1.1.4.2 matt 167 1.1.1.1.4.2 matt <p> If the total sender count is below 20000 the <a href="QSHAPE_README.html#active_queue">active queue</a> is 168 1.1.1.1.4.2 matt not yet saturated, any high volume sender domains show near the 169 1.1.1.1.4.2 matt top of the output. 170 1.1.1.1.4.2 matt 171 1.1.1.1.4.2 matt <p> With <a href="qmgr.8.html">oqmgr(8)</a> the <a href="QSHAPE_README.html#active_queue">active queue</a> is also limited to at most 20000 172 1.1.1.1.4.2 matt recipient addresses ($<a href="postconf.5.html#qmgr_message_recipient_limit">qmgr_message_recipient_limit</a>). To check for 173 1.1.1.1.4.2 matt exhaustion of this limit use: </p> 174 1.1.1.1.4.2 matt 175 1.1.1.1.4.2 matt <blockquote> 176 1.1.1.1.4.2 matt <pre> 177 1.1.1.1.4.2 matt $ qshape active <i>(show recipient statistics)</i> 178 1.1.1.1.4.2 matt </pre> 179 1.1.1.1.4.2 matt </blockquote> 180 1.1.1.1.4.2 matt 181 1.1.1.1.4.2 matt <p> Having found the high volume domains, it is often useful to 182 1.1.1.1.4.2 matt search the logs for recent messages pertaining to the domains in 183 1.1.1.1.4.2 matt question. </p> 184 1.1.1.1.4.2 matt 185 1.1.1.1.4.2 matt <blockquote> 186 1.1.1.1.4.2 matt <pre> 187 1.1.1.1.4.2 matt # Find deliveries to example.com 188 1.1.1.1.4.2 matt # 189 1.1.1.1.4.2 matt $ tail -10000 /var/log/maillog | 190 1.1.1.1.4.2 matt egrep -i ': to=<.*@example\.com>,' | 191 1.1.1.1.4.2 matt less 192 1.1.1.1.4.2 matt 193 1.1.1.1.4.2 matt # Find messages from example.com 194 1.1.1.1.4.2 matt # 195 1.1.1.1.4.2 matt $ tail -10000 /var/log/maillog | 196 1.1.1.1.4.2 matt egrep -i ': from=<.*@example\.com>,' | 197 1.1.1.1.4.2 matt less 198 1.1.1.1.4.2 matt </pre> 199 1.1.1.1.4.2 matt </blockquote> 200 1.1.1.1.4.2 matt 201 1.1.1.1.4.2 matt <p> You may want to drill in on some specific queue ids: </p> 202 1.1.1.1.4.2 matt 203 1.1.1.1.4.2 matt <blockquote> 204 1.1.1.1.4.2 matt <pre> 205 1.1.1.1.4.2 matt # Find all messages for a specific queue id. 206 1.1.1.1.4.2 matt # 207 1.1.1.1.4.2 matt $ tail -10000 /var/log/maillog | egrep ': 2B2173FF68: ' 208 1.1.1.1.4.2 matt </pre> 209 1.1.1.1.4.2 matt </blockquote> 210 1.1.1.1.4.2 matt 211 1.1.1.1.4.2 matt <p> Also look for queue manager warning messages in the log. These 212 1.1.1.1.4.2 matt warnings can suggest strategies to reduce congestion. </p> 213 1.1.1.1.4.2 matt 214 1.1.1.1.4.2 matt <blockquote> 215 1.1.1.1.4.2 matt <pre> 216 1.1.1.1.4.2 matt $ egrep 'qmgr.*(panic|fatal|error|warning):' /var/log/maillog 217 1.1.1.1.4.2 matt </pre> 218 1.1.1.1.4.2 matt </blockquote> 219 1.1.1.1.4.2 matt 220 1.1.1.1.4.2 matt <p> When all else fails try the Postfix mailing list for help, but 221 1.1.1.1.4.2 matt please don't forget to include the top 10 or 20 lines of <a href="qshape.1.html">qshape(1)</a> 222 1.1.1.1.4.2 matt output. </p> 223 1.1.1.1.4.2 matt 224 1.1.1.1.4.2 matt <h2><a name="healthy">Example 1: Healthy queue</a></h2> 225 1.1.1.1.4.2 matt 226 1.1.1.1.4.2 matt <p> When looking at just the <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queues</a>, under 227 1.1.1.1.4.2 matt normal conditions (no congestion) the <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queues</a> 228 1.1.1.1.4.2 matt are nearly empty. Mail leaves the system almost as quickly as it 229 1.1.1.1.4.2 matt comes in or is deferred without congestion in the <a href="QSHAPE_README.html#active_queue">active queue</a>. 230 1.1.1.1.4.2 matt </p> 231 1.1.1.1.4.2 matt 232 1.1.1.1.4.2 matt <blockquote> 233 1.1.1.1.4.2 matt <pre> 234 1.1.1.1.4.2 matt $ qshape <i>(show <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queue</a> status)</i> 235 1.1.1.1.4.2 matt 236 1.1.1.1.4.2 matt T 5 10 20 40 80 160 320 640 1280 1280+ 237 1.1.1.1.4.2 matt TOTAL 5 0 0 0 1 0 0 0 1 1 2 238 1.1.1.1.4.2 matt meri.uwasa.fi 5 0 0 0 1 0 0 0 1 1 2 239 1.1.1.1.4.2 matt </pre> 240 1.1.1.1.4.2 matt </blockquote> 241 1.1.1.1.4.2 matt 242 1.1.1.1.4.2 matt <p> If one looks at the two queues separately, the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> 243 1.1.1.1.4.2 matt is empty or perhaps briefly has one or two messages, while the 244 1.1.1.1.4.2 matt <a href="QSHAPE_README.html#active_queue">active queue</a> holds more messages and for a somewhat longer time: 245 1.1.1.1.4.2 matt </p> 246 1.1.1.1.4.2 matt 247 1.1.1.1.4.2 matt <blockquote> 248 1.1.1.1.4.2 matt <pre> 249 1.1.1.1.4.2 matt $ qshape incoming 250 1.1.1.1.4.2 matt 251 1.1.1.1.4.2 matt T 5 10 20 40 80 160 320 640 1280 1280+ 252 1.1.1.1.4.2 matt TOTAL 0 0 0 0 0 0 0 0 0 0 0 253 1.1.1.1.4.2 matt 254 1.1.1.1.4.2 matt $ qshape active 255 1.1.1.1.4.2 matt 256 1.1.1.1.4.2 matt T 5 10 20 40 80 160 320 640 1280 1280+ 257 1.1.1.1.4.2 matt TOTAL 5 0 0 0 1 0 0 0 1 1 2 258 1.1.1.1.4.2 matt meri.uwasa.fi 5 0 0 0 1 0 0 0 1 1 2 259 1.1.1.1.4.2 matt </pre> 260 1.1.1.1.4.2 matt </blockquote> 261 1.1.1.1.4.2 matt 262 1.1.1.1.4.2 matt <h2><a name="dictionary_bounce">Example 2: Deferred queue full of 263 1.1.1.1.4.2 matt dictionary attack bounces</a></h2> 264 1.1.1.1.4.2 matt 265 1.1.1.1.4.2 matt <p> This is from a server where recipient validation is not yet 266 1.1.1.1.4.2 matt available for some of the <a href="VIRTUAL_README.html#canonical">hosted domains</a>. Dictionary attacks on 267 1.1.1.1.4.2 matt the unvalidated domains result in bounce backscatter. The bounces 268 1.1.1.1.4.2 matt dominate the queue, but with proper tuning they do not saturate the 269 1.1.1.1.4.2 matt <a href="QSHAPE_README.html#incoming_queue">incoming</a> or <a href="QSHAPE_README.html#active_queue">active queues</a>. The high volume of deferred mail is not 270 1.1.1.1.4.2 matt a direct cause for alarm. </p> 271 1.1.1.1.4.2 matt 272 1.1.1.1.4.2 matt <blockquote> 273 1.1.1.1.4.2 matt <pre> 274 1.1.1.1.4.2 matt $ qshape deferred | head 275 1.1.1.1.4.2 matt 276 1.1.1.1.4.2 matt T 5 10 20 40 80 160 320 640 1280 1280+ 277 1.1.1.1.4.2 matt TOTAL 2234 4 2 5 9 31 57 108 201 464 1353 278 1.1.1.1.4.2 matt heyhihellothere.com 207 0 0 1 1 6 6 8 25 68 92 279 1.1.1.1.4.2 matt pleazerzoneprod.com 105 0 0 0 0 0 0 0 5 44 56 280 1.1.1.1.4.2 matt groups.msn.com 63 2 1 2 4 4 14 14 14 8 0 281 1.1.1.1.4.2 matt orion.toppoint.de 49 0 0 0 1 0 2 4 3 16 23 282 1.1.1.1.4.2 matt kali.com.cn 46 0 0 0 0 1 0 2 6 12 25 283 1.1.1.1.4.2 matt meri.uwasa.fi 44 0 0 0 0 1 0 2 8 11 22 284 1.1.1.1.4.2 matt gjr.paknet.com.pk 43 1 0 0 1 1 3 3 6 12 16 285 1.1.1.1.4.2 matt aristotle.algonet.se 41 0 0 0 0 0 1 2 11 12 15 286 1.1.1.1.4.2 matt </pre> 287 1.1.1.1.4.2 matt </blockquote> 288 1.1.1.1.4.2 matt 289 1.1.1.1.4.2 matt <p> The domains shown are mostly bulk-mailers and all the volume 290 1.1.1.1.4.2 matt is the tail end of the time distribution, showing that short term 291 1.1.1.1.4.2 matt arrival rates are moderate. Larger numbers and lower message ages 292 1.1.1.1.4.2 matt are more indicative of current trouble. Old mail still going nowhere 293 1.1.1.1.4.2 matt is largely harmless so long as the active and <a href="QSHAPE_README.html#incoming_queue">incoming queues</a> are 294 1.1.1.1.4.2 matt short. We can also see that the groups.msn.com undeliverables are 295 1.1.1.1.4.2 matt low rate steady stream rather than a concentrated dictionary attack 296 1.1.1.1.4.2 matt that is now over. </p> 297 1.1.1.1.4.2 matt 298 1.1.1.1.4.2 matt <blockquote> 299 1.1.1.1.4.2 matt <pre> 300 1.1.1.1.4.2 matt $ qshape -s deferred | head 301 1.1.1.1.4.2 matt 302 1.1.1.1.4.2 matt T 5 10 20 40 80 160 320 640 1280 1280+ 303 1.1.1.1.4.2 matt TOTAL 2193 4 4 5 8 33 56 104 205 465 1309 304 1.1.1.1.4.2 matt MAILER-DAEMON 1709 4 4 5 8 33 55 101 198 452 849 305 1.1.1.1.4.2 matt example.com 263 0 0 0 0 0 0 0 0 2 261 306 1.1.1.1.4.2 matt example.org 209 0 0 0 0 0 1 3 6 11 188 307 1.1.1.1.4.2 matt example.net 6 0 0 0 0 0 0 0 0 0 6 308 1.1.1.1.4.2 matt example.edu 3 0 0 0 0 0 0 0 0 0 3 309 1.1.1.1.4.2 matt example.gov 2 0 0 0 0 0 0 0 1 0 1 310 1.1.1.1.4.2 matt example.mil 1 0 0 0 0 0 0 0 0 0 1 311 1.1.1.1.4.2 matt </pre> 312 1.1.1.1.4.2 matt </blockquote> 313 1.1.1.1.4.2 matt 314 1.1.1.1.4.2 matt <p> Looking at the sender distribution, we see that as expected 315 1.1.1.1.4.2 matt most of the messages are bounces. </p> 316 1.1.1.1.4.2 matt 317 1.1.1.1.4.2 matt <h2><a name="active_congestion">Example 3: Congestion in the active 318 1.1.1.1.4.2 matt queue</a></h2> 319 1.1.1.1.4.2 matt 320 1.1.1.1.4.2 matt <p> This example is taken from a Feb 2004 discussion on the Postfix 321 1.1.1.1.4.2 matt Users list. Congestion was reported with the active and incoming 322 1.1.1.1.4.2 matt queues large and not shrinking despite very large delivery agent 323 1.1.1.1.4.2 matt process limits. The thread is archived at: 324 1.1.1.1.4.2 matt <a href="http://groups.google.com/groups?th=636626c645f5bbde">http://groups.google.com/groups?th=636626c645f5bbde</a> </p> 325 1.1.1.1.4.2 matt 326 1.1.1.1.4.2 matt <p> Using an older version of <a href="qshape.1.html">qshape(1)</a> it was quickly determined 327 1.1.1.1.4.2 matt that all the messages were for just a few destinations: </p> 328 1.1.1.1.4.2 matt 329 1.1.1.1.4.2 matt <blockquote> 330 1.1.1.1.4.2 matt <pre> 331 1.1.1.1.4.2 matt $ qshape <i>(show <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queue</a> status)</i> 332 1.1.1.1.4.2 matt 333 1.1.1.1.4.2 matt T A 5 10 20 40 80 160 320 320+ 334 1.1.1.1.4.2 matt TOTAL 11775 9996 0 0 1 1 42 94 221 1420 335 1.1.1.1.4.2 matt user.sourceforge.net 7678 7678 0 0 0 0 0 0 0 0 336 1.1.1.1.4.2 matt lists.sourceforge.net 2313 2313 0 0 0 0 0 0 0 0 337 1.1.1.1.4.2 matt gzd.gotdns.com 102 0 0 0 0 0 0 0 2 100 338 1.1.1.1.4.2 matt </pre> 339 1.1.1.1.4.2 matt </blockquote> 340 1.1.1.1.4.2 matt 341 1.1.1.1.4.2 matt <p> The "A" column showed the count of messages in the <a href="QSHAPE_README.html#active_queue">active queue</a>, 342 1.1.1.1.4.2 matt and the numbered columns showed totals for the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a>. At 343 1.1.1.1.4.2 matt 10000 messages (Postfix 1.x <a href="QSHAPE_README.html#active_queue">active queue</a> size limit) the active 344 1.1.1.1.4.2 matt queue is full. The incoming was growing rapidly. </p> 345 1.1.1.1.4.2 matt 346 1.1.1.1.4.2 matt <p> With the trouble destinations clearly identified, the administrator 347 1.1.1.1.4.2 matt quickly found and fixed the problem. It is substantially harder to 348 1.1.1.1.4.2 matt glean the same information from the logs. While a careful reading 349 1.1.1.1.4.2 matt of <a href="mailq.1.html">mailq(1)</a> output should yield similar results, it is much harder 350 1.1.1.1.4.2 matt to gauge the magnitude of the problem by looking at the queue 351 1.1.1.1.4.2 matt one message at a time. </p> 352 1.1.1.1.4.2 matt 353 1.1.1.1.4.2 matt <h2><a name="backlog">Example 4: High volume destination backlog</a></h2> 354 1.1.1.1.4.2 matt 355 1.1.1.1.4.2 matt <p> When a site you send a lot of email to is down or slow, mail 356 1.1.1.1.4.2 matt messages will rapidly build up in the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a>, or worse, in 357 1.1.1.1.4.2 matt the <a href="QSHAPE_README.html#active_queue">active queue</a>. The qshape output will show large numbers for 358 1.1.1.1.4.2 matt the destination domain in all age buckets that overlap the starting 359 1.1.1.1.4.2 matt time of the problem: </p> 360 1.1.1.1.4.2 matt 361 1.1.1.1.4.2 matt <blockquote> 362 1.1.1.1.4.2 matt <pre> 363 1.1.1.1.4.2 matt $ qshape deferred | head 364 1.1.1.1.4.2 matt 365 1.1.1.1.4.2 matt T 5 10 20 40 80 160 320 640 1280 1280+ 366 1.1.1.1.4.2 matt TOTAL 5000 200 200 400 800 1600 1000 200 200 200 200 367 1.1.1.1.4.2 matt highvolume.com 4000 160 160 320 640 1280 1440 0 0 0 0 368 1.1.1.1.4.2 matt ... 369 1.1.1.1.4.2 matt </pre> 370 1.1.1.1.4.2 matt </blockquote> 371 1.1.1.1.4.2 matt 372 1.1.1.1.4.2 matt <p> Here the "highvolume.com" destination is continuing to accumulate 373 1.1.1.1.4.2 matt deferred mail. The <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queues</a> are fine, but the 374 1.1.1.1.4.2 matt <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> started growing some time between 1 and 2 hours ago 375 1.1.1.1.4.2 matt and continues to grow. </p> 376 1.1.1.1.4.2 matt 377 1.1.1.1.4.2 matt <p> If the high volume destination is not down, but is instead 378 1.1.1.1.4.2 matt slow, one might see similar congestion in the <a href="QSHAPE_README.html#active_queue">active queue</a>. Active 379 1.1.1.1.4.2 matt queue congestion is a greater cause for alarm; one might need to 380 1.1.1.1.4.2 matt take measures to ensure that the mail is deferred instead or even 381 1.1.1.1.4.2 matt add an <a href="access.5.html">access(5)</a> rule asking the sender to try again later. </p> 382 1.1.1.1.4.2 matt 383 1.1.1.1.4.2 matt <p> If a high volume destination exhibits frequent bursts of consecutive 384 1.1.1.1.4.2 matt connections refused by all MX hosts or "421 Server busy errors", it 385 1.1.1.1.4.2 matt is possible for the queue manager to mark the destination as "dead" 386 1.1.1.1.4.2 matt despite the transient nature of the errors. The destination will be 387 1.1.1.1.4.2 matt retried again after the expiration of a $<a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> timer. 388 1.1.1.1.4.2 matt If the error bursts are frequent enough it may be that only a small 389 1.1.1.1.4.2 matt quantity of email is delivered before the destination is again marked 390 1.1.1.1.4.2 matt "dead". In some cases enabling static (not on demand) connection 391 1.1.1.1.4.2 matt caching by listing the appropriate nexthop domain in a table included in 392 1.1.1.1.4.2 matt "<a href="postconf.5.html#smtp_connection_cache_destinations">smtp_connection_cache_destinations</a>" may help to reduce the error rate, 393 1.1.1.1.4.2 matt because most messages will re-use existing connections. </p> 394 1.1.1.1.4.2 matt 395 1.1.1.1.4.2 matt <p> The MTA that has been observed most frequently to exhibit such 396 1.1.1.1.4.2 matt bursts of errors is Microsoft Exchange, which refuses connections 397 1.1.1.1.4.2 matt under load. Some proxy virus scanners in front of the Exchange 398 1.1.1.1.4.2 matt server propagate the refused connection to the client as a "421" 399 1.1.1.1.4.2 matt error. </p> 400 1.1.1.1.4.2 matt 401 1.1.1.1.4.2 matt <p> Note that it is now possible to configure Postfix to exhibit similarly 402 1.1.1.1.4.2 matt erratic behavior by misconfiguring the <a href="anvil.8.html">anvil(8)</a> service. Do not use 403 1.1.1.1.4.2 matt <a href="anvil.8.html">anvil(8)</a> for steady-state rate limiting, its purpose is (unintentional) 404 1.1.1.1.4.2 matt DoS prevention and the rate limits set should be very generous! </p> 405 1.1.1.1.4.2 matt 406 1.1.1.1.4.2 matt <p> If one finds oneself needing to deliver a high volume of mail to a 407 1.1.1.1.4.2 matt destination that exhibits frequent brief bursts of errors and connection 408 1.1.1.1.4.2 matt caching does not solve the problem, there is a subtle workaround. </p> 409 1.1.1.1.4.2 matt 410 1.1.1.1.4.2 matt <ul> 411 1.1.1.1.4.2 matt 412 1.1.1.1.4.2 matt <li> <p> Postfix version 2.5 and later: </p> 413 1.1.1.1.4.2 matt 414 1.1.1.1.4.2 matt <ul> 415 1.1.1.1.4.2 matt 416 1.1.1.1.4.2 matt <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" transport 417 1.1.1.1.4.2 matt for the destination in question. In the example below we will call 418 1.1.1.1.4.2 matt it "fragile". </p> 419 1.1.1.1.4.2 matt 420 1.1.1.1.4.2 matt <li> <p> In <a href="master.5.html">master.cf</a> configure a reasonable process limit for the 421 1.1.1.1.4.2 matt cloned smtp transport (a number in the 10-20 range is typical). </p> 422 1.1.1.1.4.2 matt 423 1.1.1.1.4.2 matt <li> <p> IMPORTANT!!! In <a href="postconf.5.html">main.cf</a> configure a large per-destination 424 1.1.1.1.4.2 matt pseudo-cohort failure limit for the cloned smtp transport. </p> 425 1.1.1.1.4.2 matt 426 1.1.1.1.4.2 matt <pre> 427 1.1.1.1.4.2 matt /etc/postfix/<a href="postconf.5.html">main.cf</a>: 428 1.1.1.1.4.2 matt <a href="postconf.5.html#transport_maps">transport_maps</a> = hash:/etc/postfix/transport 429 1.1.1.1.4.2 matt fragile_destination_concurrency_failed_cohort_limit = 100 430 1.1.1.1.4.2 matt fragile_destination_concurrency_limit = 20 431 1.1.1.1.4.2 matt 432 1.1.1.1.4.2 matt /etc/postfix/transport: 433 1.1.1.1.4.2 matt example.com fragile: 434 1.1.1.1.4.2 matt 435 1.1.1.1.4.2 matt /etc/postfix/<a href="master.5.html">master.cf</a>: 436 1.1.1.1.4.2 matt # service type private unpriv chroot wakeup maxproc command 437 1.1.1.1.4.2 matt fragile unix - - n - 20 smtp 438 1.1.1.1.4.2 matt </pre> 439 1.1.1.1.4.2 matt 440 1.1.1.1.4.2 matt <p> See also the documentation for 441 1.1.1.1.4.2 matt <a href="postconf.5.html#default_destination_concurrency_failed_cohort_limit">default_destination_concurrency_failed_cohort_limit</a> and 442 1.1.1.1.4.2 matt <a href="postconf.5.html#default_destination_concurrency_limit">default_destination_concurrency_limit</a>. </p> 443 1.1.1.1.4.2 matt 444 1.1.1.1.4.2 matt </ul> 445 1.1.1.1.4.2 matt 446 1.1.1.1.4.2 matt <li> <p> Earlier Postfix versions: </p> 447 1.1.1.1.4.2 matt 448 1.1.1.1.4.2 matt <ul> 449 1.1.1.1.4.2 matt 450 1.1.1.1.4.2 matt <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" 451 1.1.1.1.4.2 matt transport for the destination in question. In the example below 452 1.1.1.1.4.2 matt we will call it "fragile". </p> 453 1.1.1.1.4.2 matt 454 1.1.1.1.4.2 matt <li> <p> In <a href="master.5.html">master.cf</a> configure a reasonable process limit for the 455 1.1.1.1.4.2 matt transport (a number in the 10-20 range is typical). </p> 456 1.1.1.1.4.2 matt 457 1.1.1.1.4.2 matt <li> <p> IMPORTANT!!! In <a href="postconf.5.html">main.cf</a> configure a very large initial 458 1.1.1.1.4.2 matt and destination concurrency limit for this transport (say 2000). </p> 459 1.1.1.1.4.2 matt 460 1.1.1.1.4.2 matt <pre> 461 1.1.1.1.4.2 matt /etc/postfix/<a href="postconf.5.html">main.cf</a>: 462 1.1.1.1.4.2 matt <a href="postconf.5.html#transport_maps">transport_maps</a> = hash:/etc/postfix/transport 463 1.1.1.1.4.2 matt <a href="postconf.5.html#initial_destination_concurrency">initial_destination_concurrency</a> = 2000 464 1.1.1.1.4.2 matt fragile_destination_concurrency_limit = 2000 465 1.1.1.1.4.2 matt 466 1.1.1.1.4.2 matt /etc/postfix/transport: 467 1.1.1.1.4.2 matt example.com fragile: 468 1.1.1.1.4.2 matt 469 1.1.1.1.4.2 matt /etc/postfix/<a href="master.5.html">master.cf</a>: 470 1.1.1.1.4.2 matt # service type private unpriv chroot wakeup maxproc command 471 1.1.1.1.4.2 matt fragile unix - - n - 20 smtp 472 1.1.1.1.4.2 matt </pre> 473 1.1.1.1.4.2 matt 474 1.1.1.1.4.2 matt <p> See also the documentation for <a href="postconf.5.html#default_destination_concurrency_limit">default_destination_concurrency_limit</a>. 475 1.1.1.1.4.2 matt </p> 476 1.1.1.1.4.2 matt 477 1.1.1.1.4.2 matt </ul> 478 1.1.1.1.4.2 matt 479 1.1.1.1.4.2 matt </ul> 480 1.1.1.1.4.2 matt 481 1.1.1.1.4.2 matt <p> The effect of this configuration is that up to 2000 482 1.1.1.1.4.2 matt consecutive errors are tolerated without marking the destination 483 1.1.1.1.4.2 matt dead, while the total concurrency remains reasonable (10-20 484 1.1.1.1.4.2 matt processes). This trick is only for a very specialized situation: 485 1.1.1.1.4.2 matt high volume delivery into a channel with multi-error bursts 486 1.1.1.1.4.2 matt that is capable of high throughput, but is repeatedly throttled by 487 1.1.1.1.4.2 matt the bursts of errors. </p> 488 1.1.1.1.4.2 matt 489 1.1.1.1.4.2 matt <p> When a destination is unable to handle the load even after the 490 1.1.1.1.4.2 matt Postfix process limit is reduced to 1, a desperate measure is to 491 1.1.1.1.4.2 matt insert brief delays between delivery attempts. </p> 492 1.1.1.1.4.2 matt 493 1.1.1.1.4.2 matt <ul> 494 1.1.1.1.4.2 matt 495 1.1.1.1.4.2 matt <li> <p> Postfix version 2.5 and later: </p> 496 1.1.1.1.4.2 matt 497 1.1.1.1.4.2 matt <ul> 498 1.1.1.1.4.2 matt 499 1.1.1.1.4.2 matt <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" transport 500 1.1.1.1.4.2 matt for the problem destination. In the example below we call it "slow". 501 1.1.1.1.4.2 matt </p> 502 1.1.1.1.4.2 matt 503 1.1.1.1.4.2 matt <li> <p> In <a href="postconf.5.html">main.cf</a> configure a short delay between deliveries to 504 1.1.1.1.4.2 matt the same destination. </p> 505 1.1.1.1.4.2 matt 506 1.1.1.1.4.2 matt <pre> 507 1.1.1.1.4.2 matt /etc/postfix/<a href="postconf.5.html">main.cf</a>: 508 1.1.1.1.4.2 matt <a href="postconf.5.html#transport_maps">transport_maps</a> = hash:/etc/postfix/transport 509 1.1.1.1.4.2 matt slow_destination_rate_delay = 1 510 1.1.1.1.4.2 matt 511 1.1.1.1.4.2 matt /etc/postfix/transport: 512 1.1.1.1.4.2 matt example.com slow: 513 1.1.1.1.4.2 matt 514 1.1.1.1.4.2 matt /etc/postfix/<a href="master.5.html">master.cf</a>: 515 1.1.1.1.4.2 matt # service type private unpriv chroot wakeup maxproc command 516 1.1.1.1.4.2 matt slow unix - - n - - smtp 517 1.1.1.1.4.2 matt </pre> 518 1.1.1.1.4.2 matt 519 1.1.1.1.4.2 matt </ul> 520 1.1.1.1.4.2 matt 521 1.1.1.1.4.2 matt <p> See also the documentation for <a href="postconf.5.html#default_destination_rate_delay">default_destination_rate_delay</a>. </p> 522 1.1.1.1.4.2 matt 523 1.1.1.1.4.2 matt <p> This solution forces the Postfix <a href="smtp.8.html">smtp(8)</a> client to wait for 524 1.1.1.1.4.2 matt $slow_destination_rate_delay seconds between deliveries to the same 525 1.1.1.1.4.2 matt destination. </p> 526 1.1.1.1.4.2 matt 527 1.1.1.1.4.2 matt <li> <p> Earlier Postfix versions: </p> 528 1.1.1.1.4.2 matt 529 1.1.1.1.4.2 matt <ul> 530 1.1.1.1.4.2 matt 531 1.1.1.1.4.2 matt <li> <p> In the transport map entry for the problem destination, 532 1.1.1.1.4.2 matt specify a dead host as the primary nexthop. </p> 533 1.1.1.1.4.2 matt 534 1.1.1.1.4.2 matt <li> <p> In the <a href="master.5.html">master.cf</a> entry for the transport specify the 535 1.1.1.1.4.2 matt problem destination as the <a href="postconf.5.html#fallback_relay">fallback_relay</a> and specify a small 536 1.1.1.1.4.2 matt <a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a> value. </p> 537 1.1.1.1.4.2 matt 538 1.1.1.1.4.2 matt <pre> 539 1.1.1.1.4.2 matt /etc/postfix/<a href="postconf.5.html">main.cf</a>: 540 1.1.1.1.4.2 matt <a href="postconf.5.html#transport_maps">transport_maps</a> = hash:/etc/postfix/transport 541 1.1.1.1.4.2 matt 542 1.1.1.1.4.2 matt /etc/postfix/transport: 543 1.1.1.1.4.2 matt example.com slow:[dead.host] 544 1.1.1.1.4.2 matt 545 1.1.1.1.4.2 matt /etc/postfix/<a href="master.5.html">master.cf</a>: 546 1.1.1.1.4.2 matt # service type private unpriv chroot wakeup maxproc command 547 1.1.1.1.4.2 matt slow unix - - n - 1 smtp 548 1.1.1.1.4.2 matt -o <a href="postconf.5.html#fallback_relay">fallback_relay</a>=problem.example.com 549 1.1.1.1.4.2 matt -o <a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a>=1 550 1.1.1.1.4.2 matt -o <a href="postconf.5.html#smtp_connection_cache_on_demand">smtp_connection_cache_on_demand</a>=no 551 1.1.1.1.4.2 matt </pre> 552 1.1.1.1.4.2 matt 553 1.1.1.1.4.2 matt </ul> 554 1.1.1.1.4.2 matt 555 1.1.1.1.4.2 matt <p> This solution forces the Postfix <a href="smtp.8.html">smtp(8)</a> client to wait for 556 1.1.1.1.4.2 matt $<a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a> seconds between deliveries. The connection 557 1.1.1.1.4.2 matt caching feature is disabled to prevent the client from skipping 558 1.1.1.1.4.2 matt over the dead host. </p> 559 1.1.1.1.4.2 matt 560 1.1.1.1.4.2 matt </ul> 561 1.1.1.1.4.2 matt 562 1.1.1.1.4.2 matt <h2><a name="queues">Postfix queue directories</a></h2> 563 1.1.1.1.4.2 matt 564 1.1.1.1.4.2 matt <p> The following sections describe Postfix queues: their purpose, 565 1.1.1.1.4.2 matt what normal behavior looks like, and how to diagnose abnormal 566 1.1.1.1.4.2 matt behavior. </p> 567 1.1.1.1.4.2 matt 568 1.1.1.1.4.2 matt <h3> <a name="maildrop_queue"> The "maildrop" queue </a> </h3> 569 1.1.1.1.4.2 matt 570 1.1.1.1.4.2 matt <p> Messages that have been submitted via the Postfix <a href="sendmail.1.html">sendmail(1)</a> 571 1.1.1.1.4.2 matt command, but not yet brought into the main Postfix queue by the 572 1.1.1.1.4.2 matt <a href="pickup.8.html">pickup(8)</a> service, await processing in the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a>. Messages 573 1.1.1.1.4.2 matt can be added to the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> even when the Postfix system 574 1.1.1.1.4.2 matt is not running. They will begin to be processed once Postfix is 575 1.1.1.1.4.2 matt started. </p> 576 1.1.1.1.4.2 matt 577 1.1.1.1.4.2 matt <p> The "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> is drained by the single threaded <a href="pickup.8.html">pickup(8)</a> 578 1.1.1.1.4.2 matt service scanning the queue directory periodically or when notified 579 1.1.1.1.4.2 matt of new message arrival by the <a href="postdrop.1.html">postdrop(1)</a> program. The <a href="postdrop.1.html">postdrop(1)</a> 580 1.1.1.1.4.2 matt program is a setgid helper that allows the unprivileged Postfix 581 1.1.1.1.4.2 matt <a href="sendmail.1.html">sendmail(1)</a> program to inject mail into the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> and 582 1.1.1.1.4.2 matt to notify the <a href="pickup.8.html">pickup(8)</a> service of its arrival. </p> 583 1.1.1.1.4.2 matt 584 1.1.1.1.4.2 matt <p> All mail that enters the main Postfix queue does so via the 585 1.1.1.1.4.2 matt <a href="cleanup.8.html">cleanup(8)</a> service. The cleanup service is responsible for envelope 586 1.1.1.1.4.2 matt and header rewriting, header and body regular expression checks, 587 1.1.1.1.4.2 matt automatic bcc recipient processing, milter content processing, and 588 1.1.1.1.4.2 matt reliable insertion of the message into the Postfix "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>. </p> 589 1.1.1.1.4.2 matt 590 1.1.1.1.4.2 matt <p> In the absence of excessive CPU consumption in <a href="cleanup.8.html">cleanup(8)</a> header 591 1.1.1.1.4.2 matt or body regular expression checks or other software consuming all 592 1.1.1.1.4.2 matt available CPU resources, Postfix performance is disk I/O bound. 593 1.1.1.1.4.2 matt The rate at which the <a href="pickup.8.html">pickup(8)</a> service can inject messages into 594 1.1.1.1.4.2 matt the queue is largely determined by disk access times, since the 595 1.1.1.1.4.2 matt <a href="cleanup.8.html">cleanup(8)</a> service must commit the message to stable storage before 596 1.1.1.1.4.2 matt returning success. The same is true of the <a href="postdrop.1.html">postdrop(1)</a> program 597 1.1.1.1.4.2 matt writing the message to the "maildrop" directory. </p> 598 1.1.1.1.4.2 matt 599 1.1.1.1.4.2 matt <p> As the pickup service is single threaded, it can only deliver 600 1.1.1.1.4.2 matt one message at a time at a rate that does not exceed the reciprocal 601 1.1.1.1.4.2 matt disk I/O latency (+ CPU if not negligible) of the cleanup service. 602 1.1.1.1.4.2 matt </p> 603 1.1.1.1.4.2 matt 604 1.1.1.1.4.2 matt <p> Congestion in this queue is indicative of an excessive local message 605 1.1.1.1.4.2 matt submission rate or perhaps excessive CPU consumption in the <a href="cleanup.8.html">cleanup(8)</a> 606 1.1.1.1.4.2 matt service due to excessive <a href="postconf.5.html#body_checks">body_checks</a>, or (Postfix ≥ 2.3) high latency 607 1.1.1.1.4.2 matt milters. </p> 608 1.1.1.1.4.2 matt 609 1.1.1.1.4.2 matt <p> Note, that once the <a href="QSHAPE_README.html#active_queue">active queue</a> is full, the cleanup service 610 1.1.1.1.4.2 matt will attempt to slow down message injection by pausing $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> 611 1.1.1.1.4.2 matt for each message. In this case "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> congestion may be 612 1.1.1.1.4.2 matt a consequence of congestion downstream, rather than a problem in 613 1.1.1.1.4.2 matt its own right. </p> 614 1.1.1.1.4.2 matt 615 1.1.1.1.4.2 matt <p> Note, you should not attempt to deliver large volumes of mail via 616 1.1.1.1.4.2 matt the <a href="pickup.8.html">pickup(8)</a> service. High volume sites should avoid using "simple" 617 1.1.1.1.4.2 matt content filters that re-inject scanned mail via Postfix <a href="sendmail.1.html">sendmail(1)</a> 618 1.1.1.1.4.2 matt and <a href="postdrop.1.html">postdrop(1)</a>. </p> 619 1.1.1.1.4.2 matt 620 1.1.1.1.4.2 matt <p> A high arrival rate of locally submitted mail may be an indication 621 1.1.1.1.4.2 matt of an uncaught forwarding loop, or a run-away notification program. 622 1.1.1.1.4.2 matt Try to keep the volume of local mail injection to a moderate level. 623 1.1.1.1.4.2 matt </p> 624 1.1.1.1.4.2 matt 625 1.1.1.1.4.2 matt <p> The "postsuper -r" command can place selected messages into 626 1.1.1.1.4.2 matt the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> for reprocessing. This is most useful for 627 1.1.1.1.4.2 matt resetting any stale <a href="postconf.5.html#content_filter">content_filter</a> settings. Requeuing a large number 628 1.1.1.1.4.2 matt of messages using "postsuper -r" can clearly cause a spike in the 629 1.1.1.1.4.2 matt size of the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a>. </p> 630 1.1.1.1.4.2 matt 631 1.1.1.1.4.2 matt <h3> <a name="hold_queue"> The "hold" queue </a> </h3> 632 1.1.1.1.4.2 matt 633 1.1.1.1.4.2 matt <p> The administrator can define "smtpd" <a href="access.5.html">access(5)</a> policies, or 634 1.1.1.1.4.2 matt <a href="cleanup.8.html">cleanup(8)</a> header/body checks that cause messages to be automatically 635 1.1.1.1.4.2 matt diverted from normal processing and placed indefinitely in the 636 1.1.1.1.4.2 matt "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>. Messages placed in the "hold" queue stay there until 637 1.1.1.1.4.2 matt the administrator intervenes. No periodic delivery attempts are 638 1.1.1.1.4.2 matt made for messages in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>. The <a href="postsuper.1.html">postsuper(1)</a> command 639 1.1.1.1.4.2 matt can be used to manually release messages into the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. 640 1.1.1.1.4.2 matt </p> 641 1.1.1.1.4.2 matt 642 1.1.1.1.4.2 matt <p> Messages can potentially stay in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> longer than 643 1.1.1.1.4.2 matt $<a href="postconf.5.html#maximal_queue_lifetime">maximal_queue_lifetime</a>. If such "old" messages need to be released from 644 1.1.1.1.4.2 matt the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>, they should typically be moved into the "maildrop" 645 1.1.1.1.4.2 matt queue using "postsuper -r", so that the message gets a new timestamp and 646 1.1.1.1.4.2 matt is given more than one opportunity to be delivered. Messages that are 647 1.1.1.1.4.2 matt "young" can be moved directly into the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> using 648 1.1.1.1.4.2 matt "postsuper -H". </p> 649 1.1.1.1.4.2 matt 650 1.1.1.1.4.2 matt <p> The "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> plays little role in Postfix performance, and 651 1.1.1.1.4.2 matt monitoring of the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> is typically more closely motivated 652 1.1.1.1.4.2 matt by tracking spam and malware, than by performance issues. </p> 653 1.1.1.1.4.2 matt 654 1.1.1.1.4.2 matt <h3> <a name="incoming_queue"> The "incoming" queue </a> </h3> 655 1.1.1.1.4.2 matt 656 1.1.1.1.4.2 matt <p> All new mail entering the Postfix queue is written by the 657 1.1.1.1.4.2 matt <a href="cleanup.8.html">cleanup(8)</a> service into the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>. New queue files are 658 1.1.1.1.4.2 matt created owned by the "postfix" user with an access bitmask (or 659 1.1.1.1.4.2 matt mode) of 0600. Once a queue file is ready for further processing 660 1.1.1.1.4.2 matt the <a href="cleanup.8.html">cleanup(8)</a> service changes the queue file mode to 0700 and 661 1.1.1.1.4.2 matt notifies the queue manager of new mail arrival. The queue manager 662 1.1.1.1.4.2 matt ignores incomplete queue files whose mode is 0600, as these are 663 1.1.1.1.4.2 matt still being written by cleanup. </p> 664 1.1.1.1.4.2 matt 665 1.1.1.1.4.2 matt <p> The queue manager scans the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> bringing any new 666 1.1.1.1.4.2 matt mail into the "<a href="QSHAPE_README.html#active_queue">active" queue</a> if the active queue resource limits 667 1.1.1.1.4.2 matt have not been exceeded. By default, the <a href="QSHAPE_README.html#active_queue">active queue</a> accommodates 668 1.1.1.1.4.2 matt at most 20000 messages. Once the <a href="QSHAPE_README.html#active_queue">active queue</a> message limit is 669 1.1.1.1.4.2 matt reached, the queue manager stops scanning the incoming (and deferred, 670 1.1.1.1.4.2 matt see below) queue. </p> 671 1.1.1.1.4.2 matt 672 1.1.1.1.4.2 matt <p> Under normal conditions the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> is nearly empty (has 673 1.1.1.1.4.2 matt only mode 0600 files), with the queue manager able to import new 674 1.1.1.1.4.2 matt messages into the <a href="QSHAPE_README.html#active_queue">active queue</a> as soon as they become available. 675 1.1.1.1.4.2 matt </p> 676 1.1.1.1.4.2 matt 677 1.1.1.1.4.2 matt <p> The <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> grows when the message input rate spikes 678 1.1.1.1.4.2 matt above the rate at which the queue manager can import messages into 679 1.1.1.1.4.2 matt the <a href="QSHAPE_README.html#active_queue">active queue</a>. The main factors slowing down the queue manager 680 1.1.1.1.4.2 matt are disk I/O and lookup queries to the trivial-rewrite service. If the queue 681 1.1.1.1.4.2 matt manager is routinely not keeping up, consider not using "slow" 682 1.1.1.1.4.2 matt lookup services (MySQL, LDAP, ...) for transport lookups or speeding 683 1.1.1.1.4.2 matt up the hosts that provide the lookup service. If the problem is I/O 684 1.1.1.1.4.2 matt starvation, consider striping the queue over more disks, faster controllers 685 1.1.1.1.4.2 matt with a battery write cache, or other hardware improvements. At the very 686 1.1.1.1.4.2 matt least, make sure that the queue directory is mounted with the "noatime" 687 1.1.1.1.4.2 matt option if applicable to the underlying filesystem. </p> 688 1.1.1.1.4.2 matt 689 1.1.1.1.4.2 matt <p> The <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> parameter is used to clamp the input rate 690 1.1.1.1.4.2 matt when the queue manager starts to fall behind. The <a href="cleanup.8.html">cleanup(8)</a> service 691 1.1.1.1.4.2 matt will pause for $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> seconds before creating a new queue 692 1.1.1.1.4.2 matt file if it cannot obtain a "token" from the queue manager. </p> 693 1.1.1.1.4.2 matt 694 1.1.1.1.4.2 matt <p> Since the number of <a href="cleanup.8.html">cleanup(8)</a> processes is limited in most 695 1.1.1.1.4.2 matt cases by the SMTP server concurrency, the input rate can exceed 696 1.1.1.1.4.2 matt the output rate by at most "SMTP connection count" / $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> 697 1.1.1.1.4.2 matt messages per second. </p> 698 1.1.1.1.4.2 matt 699 1.1.1.1.4.2 matt <p> With a default process limit of 100, and an <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> of 700 1.1.1.1.4.2 matt 1s, the coupling is strong enough to limit a single run-away injector 701 1.1.1.1.4.2 matt to 1 message per second, but is not strong enough to deflect an 702 1.1.1.1.4.2 matt excessive input rate from many sources at the same time. </p> 703 1.1.1.1.4.2 matt 704 1.1.1.1.4.2 matt <p> If a server is being hammered from multiple directions, consider 705 1.1.1.1.4.2 matt raising the <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> to 10 seconds, but only if the incoming 706 1.1.1.1.4.2 matt queue is growing even while the <a href="QSHAPE_README.html#active_queue">active queue</a> is not full and the 707 1.1.1.1.4.2 matt trivial-rewrite service is using a fast transport lookup mechanism. 708 1.1.1.1.4.2 matt </p> 709 1.1.1.1.4.2 matt 710 1.1.1.1.4.2 matt <h3> <a name="active_queue"> The "active" queue </a> </h3> 711 1.1.1.1.4.2 matt 712 1.1.1.1.4.2 matt <p> The queue manager is a delivery agent scheduler; it works to 713 1.1.1.1.4.2 matt ensure fast and fair delivery of mail to all destinations within 714 1.1.1.1.4.2 matt designated resource limits. </p> 715 1.1.1.1.4.2 matt 716 1.1.1.1.4.2 matt <p> The <a href="QSHAPE_README.html#active_queue">active queue</a> is somewhat analogous to an operating system's 717 1.1.1.1.4.2 matt process run queue. Messages in the <a href="QSHAPE_README.html#active_queue">active queue</a> are ready to be 718 1.1.1.1.4.2 matt sent (runnable), but are not necessarily in the process of being 719 1.1.1.1.4.2 matt sent (running). </p> 720 1.1.1.1.4.2 matt 721 1.1.1.1.4.2 matt <p> While most Postfix administrators think of the "<a href="QSHAPE_README.html#active_queue">active" queue</a> 722 1.1.1.1.4.2 matt as a directory on disk, the real "<a href="QSHAPE_README.html#active_queue">active" queue</a> is a set of data 723 1.1.1.1.4.2 matt structures in the memory of the queue manager process. </p> 724 1.1.1.1.4.2 matt 725 1.1.1.1.4.2 matt <p> Messages in the "<a href="QSHAPE_README.html#maildrop_queue">maildrop"</a>, "<a href="QSHAPE_README.html#hold_queue">hold"</a>, "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "deferred" 726 1.1.1.1.4.2 matt queues (see below) do not occupy memory; they are safely stored on 727 1.1.1.1.4.2 matt disk waiting for their turn to be processed. The envelope information 728 1.1.1.1.4.2 matt for messages in the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is managed in memory, allowing 729 1.1.1.1.4.2 matt the queue manager to do global scheduling, allocating available 730 1.1.1.1.4.2 matt delivery agent processes to an appropriate message in the active 731 1.1.1.1.4.2 matt queue. </p> 732 1.1.1.1.4.2 matt 733 1.1.1.1.4.2 matt <p> Within the <a href="QSHAPE_README.html#active_queue">active queue</a>, (multi-recipient) messages are broken 734 1.1.1.1.4.2 matt up into groups of recipients that share the same transport/nexthop 735 1.1.1.1.4.2 matt combination; the group size is capped by the transport's recipient 736 1.1.1.1.4.2 matt concurrency limit. </p> 737 1.1.1.1.4.2 matt 738 1.1.1.1.4.2 matt <p> Multiple recipient groups (from one or more messages) are queued 739 1.1.1.1.4.2 matt for delivery grouped by transport/nexthop combination. The 740 1.1.1.1.4.2 matt <b>destination</b> concurrency limit for the transports caps the number 741 1.1.1.1.4.2 matt of simultaneous delivery attempts for each nexthop. Transports with 742 1.1.1.1.4.2 matt a <b>recipient</b> concurrency limit of 1 are special: these are grouped 743 1.1.1.1.4.2 matt by the actual recipient address rather than the nexthop, yielding 744 1.1.1.1.4.2 matt per-recipient concurrency limits rather than per-domain 745 1.1.1.1.4.2 matt concurrency limits. Per-recipient limits are appropriate when 746 1.1.1.1.4.2 matt performing final delivery to mailboxes rather than when relaying 747 1.1.1.1.4.2 matt to a remote server. </p> 748 1.1.1.1.4.2 matt 749 1.1.1.1.4.2 matt <p> Congestion occurs in the <a href="QSHAPE_README.html#active_queue">active queue</a> when one or more destinations 750 1.1.1.1.4.2 matt drain slower than the corresponding message input rate. </p> 751 1.1.1.1.4.2 matt 752 1.1.1.1.4.2 matt <p> Input into the <a href="QSHAPE_README.html#active_queue">active queue</a> comes both from new mail in the "incoming" 753 1.1.1.1.4.2 matt queue, and retries of mail in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. Should the "deferred" 754 1.1.1.1.4.2 matt queue get really large, retries of old mail can dominate the arrival 755 1.1.1.1.4.2 matt rate of new mail. Systems with more CPU, faster disks and more network 756 1.1.1.1.4.2 matt bandwidth can deal with larger <a href="QSHAPE_README.html#deferred_queue">deferred queues</a>, but as a rule of thumb 757 1.1.1.1.4.2 matt the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> scales to somewhere between 100,000 and 1,000,000 758 1.1.1.1.4.2 matt messages with good performance unlikely above that "limit". Systems with 759 1.1.1.1.4.2 matt queues this large should typically stop accepting new mail, or put the 760 1.1.1.1.4.2 matt backlog "on hold" until the underlying issue is fixed (provided that 761 1.1.1.1.4.2 matt there is enough capacity to handle just the new mail). </p> 762 1.1.1.1.4.2 matt 763 1.1.1.1.4.2 matt <p> When a destination is down for some time, the queue manager will 764 1.1.1.1.4.2 matt mark it dead, and immediately defer all mail for the destination without 765 1.1.1.1.4.2 matt trying to assign it to a delivery agent. In this case the messages 766 1.1.1.1.4.2 matt will quickly leave the <a href="QSHAPE_README.html#active_queue">active queue</a> and end up in the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> 767 1.1.1.1.4.2 matt (with Postfix < 2.4, this is done directly by the queue manager, 768 1.1.1.1.4.2 matt with Postfix ≥ 2.4 this is done via the "retry" delivery agent). </p> 769 1.1.1.1.4.2 matt 770 1.1.1.1.4.2 matt <p> When the destination is instead simply slow, or there is a problem 771 1.1.1.1.4.2 matt causing an excessive arrival rate the <a href="QSHAPE_README.html#active_queue">active queue</a> will grow and will 772 1.1.1.1.4.2 matt become dominated by mail to the congested destination. </p> 773 1.1.1.1.4.2 matt 774 1.1.1.1.4.2 matt <p> The only way to reduce congestion is to either reduce the input 775 1.1.1.1.4.2 matt rate or increase the throughput. Increasing the throughput requires 776 1.1.1.1.4.2 matt either increasing the concurrency or reducing the latency of 777 1.1.1.1.4.2 matt deliveries. </p> 778 1.1.1.1.4.2 matt 779 1.1.1.1.4.2 matt <p> For high volume sites a key tuning parameter is the number of 780 1.1.1.1.4.2 matt "smtp" delivery agents allocated to the "smtp" and "relay" transports. 781 1.1.1.1.4.2 matt High volume sites tend to send to many different destinations, many 782 1.1.1.1.4.2 matt of which may be down or slow, so a good fraction of the available 783 1.1.1.1.4.2 matt delivery agents will be blocked waiting for slow sites. Also mail 784 1.1.1.1.4.2 matt destined across the globe will incur large SMTP command-response 785 1.1.1.1.4.2 matt latencies, so high message throughput can only be achieved with 786 1.1.1.1.4.2 matt more concurrent delivery agents. </p> 787 1.1.1.1.4.2 matt 788 1.1.1.1.4.2 matt <p> The default "smtp" process limit of 100 is good enough for most 789 1.1.1.1.4.2 matt sites, and may even need to be lowered for sites with low bandwidth 790 1.1.1.1.4.2 matt connections (no use increasing concurrency once the network pipe 791 1.1.1.1.4.2 matt is full). When one finds that the queue is growing on an "idle" 792 1.1.1.1.4.2 matt system (CPU, disk I/O and network not exhausted) the remaining 793 1.1.1.1.4.2 matt reason for congestion is insufficient concurrency in the face of 794 1.1.1.1.4.2 matt a high average latency. If the number of outbound SMTP connections 795 1.1.1.1.4.2 matt (either ESTABLISHED or SYN_SENT) reaches the process limit, mail 796 1.1.1.1.4.2 matt is draining slowly and the system and network are not loaded, raise 797 1.1.1.1.4.2 matt the "smtp" and/or "relay" process limits! </p> 798 1.1.1.1.4.2 matt 799 1.1.1.1.4.2 matt <p> When a high volume destination is served by multiple MX hosts with 800 1.1.1.1.4.2 matt typically low delivery latency, performance can suffer dramatically when 801 1.1.1.1.4.2 matt one of the MX hosts is unresponsive and SMTP connections to that host 802 1.1.1.1.4.2 matt timeout. For example, if there are 2 equal weight MX hosts, the SMTP 803 1.1.1.1.4.2 matt connection timeout is 30 seconds and one of the MX hosts is down, the 804 1.1.1.1.4.2 matt average SMTP connection will take approximately 15 seconds to complete. 805 1.1.1.1.4.2 matt With a default per-destination concurrency limit of 20 connections, 806 1.1.1.1.4.2 matt throughput falls to just over 1 message per second. </p> 807 1.1.1.1.4.2 matt 808 1.1.1.1.4.2 matt <p> The best way to avoid bottlenecks when one or more MX hosts is 809 1.1.1.1.4.2 matt non-responsive is to use connection caching. Connection caching was 810 1.1.1.1.4.2 matt introduced with Postfix 2.2 and is by default enabled on demand for 811 1.1.1.1.4.2 matt destinations with a backlog of mail in the <a href="QSHAPE_README.html#active_queue">active queue</a>. When connection 812 1.1.1.1.4.2 matt caching is in effect for a particular destination, established connections 813 1.1.1.1.4.2 matt are re-used to send additional messages, this reduces the number of 814 1.1.1.1.4.2 matt connections made per message delivery and maintains good throughput even 815 1.1.1.1.4.2 matt in the face of partial unavailability of the destination's MX hosts. </p> 816 1.1.1.1.4.2 matt 817 1.1.1.1.4.2 matt <p> If connection caching is not available (Postfix < 2.2) or does 818 1.1.1.1.4.2 matt not provide a sufficient latency reduction, especially for the "relay" 819 1.1.1.1.4.2 matt transport used to forward mail to "your own" domains, consider setting 820 1.1.1.1.4.2 matt lower than default SMTP connection timeouts (1-5 seconds) and higher 821 1.1.1.1.4.2 matt than default destination concurrency limits. This will further reduce 822 1.1.1.1.4.2 matt latency and provide more concurrency to maintain throughput should 823 1.1.1.1.4.2 matt latency rise. </p> 824 1.1.1.1.4.2 matt 825 1.1.1.1.4.2 matt <p> Setting high concurrency limits to domains that are not your own may 826 1.1.1.1.4.2 matt be viewed as hostile by the receiving system, and steps may be taken 827 1.1.1.1.4.2 matt to prevent you from monopolizing the destination system's resources. 828 1.1.1.1.4.2 matt The defensive measures may substantially reduce your throughput or block 829 1.1.1.1.4.2 matt access entirely. Do not set aggressive concurrency limits to remote 830 1.1.1.1.4.2 matt domains without coordinating with the administrators of the target 831 1.1.1.1.4.2 matt domain. </p> 832 1.1.1.1.4.2 matt 833 1.1.1.1.4.2 matt <p> If necessary, dedicate and tune custom transports for selected high 834 1.1.1.1.4.2 matt volume destinations. The "relay" transport is provided for forwarding mail 835 1.1.1.1.4.2 matt to domains for which your server is a primary or backup MX host. These can 836 1.1.1.1.4.2 matt make up a substantial fraction of your email traffic. Use the "relay" and 837 1.1.1.1.4.2 matt not the "smtp" transport to send email to these domains. Using the "relay" 838 1.1.1.1.4.2 matt transport allocates a separate delivery agent pool to these destinations 839 1.1.1.1.4.2 matt and allows separate tuning of timeouts and concurrency limits. </p> 840 1.1.1.1.4.2 matt 841 1.1.1.1.4.2 matt <p> Another common cause of congestion is unwarranted flushing of the 842 1.1.1.1.4.2 matt entire <a href="QSHAPE_README.html#deferred_queue">deferred queue</a>. The deferred queue holds messages that are likely 843 1.1.1.1.4.2 matt to fail to be delivered and are also likely to be slow to fail delivery 844 1.1.1.1.4.2 matt (time out). As a result the most common reaction to a large <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> 845 1.1.1.1.4.2 matt (flush it!) is more than likely counter-productive, and typically makes 846 1.1.1.1.4.2 matt the congestion worse. Do not flush the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> unless you expect 847 1.1.1.1.4.2 matt that most of its content has recently become deliverable (e.g. <a href="postconf.5.html#relayhost">relayhost</a> 848 1.1.1.1.4.2 matt back up after an outage)! </p> 849 1.1.1.1.4.2 matt 850 1.1.1.1.4.2 matt <p> Note that whenever the queue manager is restarted, there may 851 1.1.1.1.4.2 matt already be messages in the <a href="QSHAPE_README.html#active_queue">active queue</a> directory, but the "real" 852 1.1.1.1.4.2 matt <a href="QSHAPE_README.html#active_queue">active queue</a> in memory is empty. In order to recover the in-memory 853 1.1.1.1.4.2 matt state, the queue manager moves all the <a href="QSHAPE_README.html#active_queue">active queue</a> messages 854 1.1.1.1.4.2 matt back into the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a>, and then uses its normal incoming 855 1.1.1.1.4.2 matt queue scan to refill the <a href="QSHAPE_README.html#active_queue">active queue</a>. The process of moving all 856 1.1.1.1.4.2 matt the messages back and forth, redoing transport table (<a href="trivial-rewrite.8.html">trivial-rewrite(8)</a> 857 1.1.1.1.4.2 matt resolve service) lookups, and re-importing the messages back into 858 1.1.1.1.4.2 matt memory is expensive. At all costs, avoid frequent restarts of the 859 1.1.1.1.4.2 matt queue manager (e.g. via frequent execution of "postfix reload"). </p> 860 1.1.1.1.4.2 matt 861 1.1.1.1.4.2 matt <h3> <a name="deferred_queue"> The "deferred" queue </a> </h3> 862 1.1.1.1.4.2 matt 863 1.1.1.1.4.2 matt <p> When all the deliverable recipients for a message are delivered, 864 1.1.1.1.4.2 matt and for some recipients delivery failed for a transient reason (it 865 1.1.1.1.4.2 matt might succeed later), the message is placed in the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a>. 866 1.1.1.1.4.2 matt </p> 867 1.1.1.1.4.2 matt 868 1.1.1.1.4.2 matt <p> The queue manager scans the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> periodically. The scan 869 1.1.1.1.4.2 matt interval is controlled by the <a href="postconf.5.html#queue_run_delay">queue_run_delay</a> parameter. While a deferred 870 1.1.1.1.4.2 matt queue scan is in progress, if an <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> scan is also in progress 871 1.1.1.1.4.2 matt (ideally these are brief since the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> should be short), the 872 1.1.1.1.4.2 matt queue manager alternates between looking for messages in the "incoming" 873 1.1.1.1.4.2 matt queue and in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. This "round-robin" strategy prevents 874 1.1.1.1.4.2 matt starvation of either the <a href="QSHAPE_README.html#incoming_queue">incoming</a> or the <a href="QSHAPE_README.html#deferred_queue">deferred queues</a>. </p> 875 1.1.1.1.4.2 matt 876 1.1.1.1.4.2 matt <p> Each <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> scan only brings a fraction of the deferred 877 1.1.1.1.4.2 matt queue back into the <a href="QSHAPE_README.html#active_queue">active queue</a> for a retry. This is because each 878 1.1.1.1.4.2 matt message in the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> is assigned a "cool-off" time when 879 1.1.1.1.4.2 matt it is deferred. This is done by time-warping the modification 880 1.1.1.1.4.2 matt time of the queue file into the future. The queue file is not 881 1.1.1.1.4.2 matt eligible for a retry if its modification time is not yet reached. 882 1.1.1.1.4.2 matt </p> 883 1.1.1.1.4.2 matt 884 1.1.1.1.4.2 matt <p> The "cool-off" time is at least $<a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> and at 885 1.1.1.1.4.2 matt most $<a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a>. The next retry time is set by doubling 886 1.1.1.1.4.2 matt the message's age in the queue, and adjusting up or down to lie 887 1.1.1.1.4.2 matt within the limits. This means that young messages are initially 888 1.1.1.1.4.2 matt retried more often than old messages. </p> 889 1.1.1.1.4.2 matt 890 1.1.1.1.4.2 matt <p> If a high volume site routinely has large <a href="QSHAPE_README.html#deferred_queue">deferred queues</a>, it 891 1.1.1.1.4.2 matt may be useful to adjust the <a href="postconf.5.html#queue_run_delay">queue_run_delay</a>, <a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> and 892 1.1.1.1.4.2 matt <a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a> to provide short enough delays on first failure 893 1.1.1.1.4.2 matt (Postfix ≥ 2.4 has a sensibly low minimal backoff time by default), 894 1.1.1.1.4.2 matt with perhaps longer delays after multiple failures, to reduce the 895 1.1.1.1.4.2 matt retransmission rate of old messages and thereby reduce the quantity 896 1.1.1.1.4.2 matt of previously deferred mail in the <a href="QSHAPE_README.html#active_queue">active queue</a>. If you want a really 897 1.1.1.1.4.2 matt low <a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a>, you may also want to lower <a href="postconf.5.html#queue_run_delay">queue_run_delay</a>, 898 1.1.1.1.4.2 matt but understand that more frequent scans will increase the demand for 899 1.1.1.1.4.2 matt disk I/O. </p> 900 1.1.1.1.4.2 matt 901 1.1.1.1.4.2 matt <p> One common cause of large <a href="QSHAPE_README.html#deferred_queue">deferred queues</a> is failure to validate 902 1.1.1.1.4.2 matt recipients at the SMTP input stage. Since spammers routinely launch 903 1.1.1.1.4.2 matt dictionary attacks from unrepliable sender addresses, the bounces 904 1.1.1.1.4.2 matt for invalid recipient addresses clog the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> (and at high 905 1.1.1.1.4.2 matt volumes proportionally clog the <a href="QSHAPE_README.html#active_queue">active queue</a>). Recipient validation 906 1.1.1.1.4.2 matt is strongly recommended through use of the <a href="postconf.5.html#local_recipient_maps">local_recipient_maps</a> and 907 1.1.1.1.4.2 matt <a href="postconf.5.html#relay_recipient_maps">relay_recipient_maps</a> parameters. Even when bounces drain quickly they 908 1.1.1.1.4.2 matt inundate innocent victims of forgery with unwanted email. To avoid 909 1.1.1.1.4.2 matt this, do not accept mail for invalid recipients. </p> 910 1.1.1.1.4.2 matt 911 1.1.1.1.4.2 matt <p> When a host with lots of deferred mail is down for some time, 912 1.1.1.1.4.2 matt it is possible for the entire <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> to reach its retry 913 1.1.1.1.4.2 matt time simultaneously. This can lead to a very full <a href="QSHAPE_README.html#active_queue">active queue</a> once 914 1.1.1.1.4.2 matt the host comes back up. The phenomenon can repeat approximately 915 1.1.1.1.4.2 matt every <a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a> seconds if the messages are again deferred 916 1.1.1.1.4.2 matt after a brief burst of congestion. Perhaps, a future Postfix release 917 1.1.1.1.4.2 matt will add a random offset to the retry time (or use a combination 918 1.1.1.1.4.2 matt of strategies) to reduce the odds of repeated complete deferred 919 1.1.1.1.4.2 matt queue flushes. </p> 920 1.1.1.1.4.2 matt 921 1.1.1.1.4.2 matt <h2><a name="credits">Credits</a></h2> 922 1.1.1.1.4.2 matt 923 1.1.1.1.4.2 matt <p> The <a href="qshape.1.html">qshape(1)</a> program was developed by Victor Duchovni of Morgan 924 1.1.1.1.4.2 matt Stanley, who also wrote the initial version of this document. </p> 925 1.1.1.1.4.2 matt 926 1.1.1.1.4.2 matt </body> 927 1.1.1.1.4.2 matt 928 1.1.1.1.4.2 matt </html> 929
Indexes created Sun Jun 07 00:24:08 UTC 2026