QSHAPE_README.html revision 1.1.1.6
1 1.1 tron <!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN" 2 1.1.1.6 christos "https://www.w3.org/TR/html4/loose.dtd"> 3 1.1 tron 4 1.1 tron <html> 5 1.1 tron 6 1.1 tron <head> 7 1.1 tron 8 1.1 tron <title>Postfix Bottleneck Analysis</title> 9 1.1 tron 10 1.1.1.4 christos <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> 11 1.1.1.5 christos <link rel='stylesheet' type='text/css' href='postfix-doc.css'> 12 1.1 tron 13 1.1 tron </head> 14 1.1 tron 15 1.1 tron <body> 16 1.1 tron 17 1.1 tron <h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix Bottleneck Analysis</h1> 18 1.1 tron 19 1.1 tron <hr> 20 1.1 tron 21 1.1 tron <h2>Purpose of this document </h2> 22 1.1 tron 23 1.1 tron <p> This document is an introduction to Postfix queue congestion analysis. 24 1.1 tron It explains how the <a href="qshape.1.html">qshape(1)</a> program can help to track down the 25 1.1 tron reason for queue congestion. <a href="qshape.1.html">qshape(1)</a> is bundled with Postfix 26 1.1 tron 2.1 and later source code, under the "auxiliary" directory. This 27 1.1 tron document describes <a href="qshape.1.html">qshape(1)</a> as bundled with Postfix 2.4. </p> 28 1.1 tron 29 1.1 tron <p> This document covers the following topics: </p> 30 1.1 tron 31 1.1 tron <ul> 32 1.1 tron 33 1.1 tron <li><a href="#qshape">Introducing the qshape tool</a> 34 1.1 tron 35 1.1 tron <li><a href="#trouble_shooting">Trouble shooting with qshape</a> 36 1.1 tron 37 1.1 tron <li><a href="#healthy">Example 1: Healthy queue</a> 38 1.1 tron 39 1.1 tron <li><a href="#dictionary_bounce">Example 2: Deferred queue full of 40 1.1 tron dictionary attack bounces</a></li> 41 1.1 tron 42 1.1 tron <li><a href="#active_congestion">Example 3: Congestion in the active 43 1.1 tron queue</a></li> 44 1.1 tron 45 1.1 tron <li><a href="#backlog">Example 4: High volume destination backlog</a> 46 1.1 tron 47 1.1 tron <li><a href="#queues">Postfix queue directories</a> 48 1.1 tron 49 1.1 tron <ul> 50 1.1 tron 51 1.1 tron <li> <a href="#maildrop_queue"> The "maildrop" queue </a> 52 1.1 tron 53 1.1 tron <li> <a href="#hold_queue"> The "hold" queue </a> 54 1.1 tron 55 1.1 tron <li> <a href="#incoming_queue"> The "incoming" queue </a> 56 1.1 tron 57 1.1 tron <li> <a href="#active_queue"> The "active" queue </a> 58 1.1 tron 59 1.1 tron <li> <a href="#deferred_queue"> The "deferred" queue </a> 60 1.1 tron 61 1.1 tron </ul> 62 1.1 tron 63 1.1 tron <li><a href="#credits">Credits</a> 64 1.1 tron 65 1.1 tron </ul> 66 1.1 tron 67 1.1 tron <h2><a name="qshape">Introducing the qshape tool</a></h2> 68 1.1 tron 69 1.1 tron <p> When mail is draining slowly or the queue is unexpectedly large, 70 1.1 tron run <a href="qshape.1.html">qshape(1)</a> as the super-user (root) to help zero in on the problem. 71 1.1 tron The <a href="qshape.1.html">qshape(1)</a> program displays a tabular view of the Postfix queue 72 1.1 tron contents. </p> 73 1.1 tron 74 1.1 tron <ul> 75 1.1 tron 76 1.1 tron <li> <p> On the horizontal axis, it displays the queue age with 77 1.1 tron fine granularity for recent messages and (geometrically) less fine 78 1.1 tron granularity for older messages. </p> 79 1.1 tron 80 1.1 tron <li> <p> The vertical axis displays the destination (or with the 81 1.1 tron "-s" switch the sender) domain. Domains with the most messages are 82 1.1 tron listed first. </p> 83 1.1 tron 84 1.1 tron </ul> 85 1.1 tron 86 1.1 tron <p> For example, in the output below we see the top 10 lines of 87 1.1 tron the (mostly forged) sender domain distribution for captured spam 88 1.1 tron in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>: </p> 89 1.1 tron 90 1.1 tron <blockquote> 91 1.1 tron <pre> 92 1.1 tron $ qshape -s hold | head 93 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 94 1.1 tron TOTAL 486 0 0 1 0 0 2 4 20 40 419 95 1.1 tron yahoo.com 14 0 0 1 0 0 0 0 1 0 12 96 1.1 tron extremepricecuts.net 13 0 0 0 0 0 0 0 2 0 11 97 1.1 tron ms35.hinet.net 12 0 0 0 0 0 0 0 0 1 11 98 1.1 tron winnersdaily.net 12 0 0 0 0 0 0 0 2 0 10 99 1.1 tron hotmail.com 11 0 0 0 0 0 0 0 0 1 10 100 1.1 tron worldnet.fr 6 0 0 0 0 0 0 0 0 0 6 101 1.1 tron ms41.hinet.net 6 0 0 0 0 0 0 0 0 0 6 102 1.1 tron osn.de 5 0 0 0 0 0 1 0 0 0 4 103 1.1 tron </pre> 104 1.1 tron </blockquote> 105 1.1 tron 106 1.1 tron <ul> 107 1.1 tron 108 1.1 tron <li> <p> The "T" column shows the total (in this case sender) count 109 1.1 tron for each domain. The columns with numbers above them, show counts 110 1.1 tron for messages aged fewer than that many minutes, but not younger 111 1.1 tron than the age limit for the previous column. The row labeled "TOTAL" 112 1.1 tron shows the total count for all domains. </p> 113 1.1 tron 114 1.1 tron <li> <p> In this example, there are 14 messages allegedly from 115 1.1 tron yahoo.com, 1 between 10 and 20 minutes old, 1 between 320 and 640 116 1.1 tron minutes old and 12 older than 1280 minutes (1440 minutes in a day). 117 1.1 tron </p> 118 1.1 tron 119 1.1 tron </ul> 120 1.1 tron 121 1.1 tron <p> When the output is a terminal intermediate results showing the top 20 122 1.1 tron domains (-n option) are displayed after every 1000 messages (-N option) 123 1.1 tron and the final output also shows only the top 20 domains. This makes 124 1.1.1.4 christos qshape useful even when the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> is very large and it may 125 1.1.1.4 christos otherwise take prohibitively long to read the entire "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. </p> 126 1.1 tron 127 1.1 tron <p> By default, qshape shows statistics for the union of both the 128 1.1.1.4 christos "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queues</a> which are the most relevant queues to 129 1.1 tron look at when analyzing performance. </p> 130 1.1 tron 131 1.1 tron <p> One can request an alternate list of queues: </p> 132 1.1 tron 133 1.1 tron <blockquote> 134 1.1 tron <pre> 135 1.1 tron $ qshape deferred 136 1.1 tron $ qshape incoming active deferred 137 1.1 tron </pre> 138 1.1 tron </blockquote> 139 1.1 tron 140 1.1.1.4 christos <p> this will show the age distribution of the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> or 141 1.1.1.4 christos the union of the "<a href="QSHAPE_README.html#incoming_queue">incoming"</a>, "<a href="QSHAPE_README.html#active_queue">active"</a> and "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a>. </p> 142 1.1 tron 143 1.1 tron <p> Command line options control the number of display "buckets", 144 1.1 tron the age limit for the smallest bucket, display of parent domain 145 1.1 tron counts and so on. The "-h" option outputs a summary of the available 146 1.1 tron switches. </p> 147 1.1 tron 148 1.1 tron <h2><a name="trouble_shooting">Trouble shooting with qshape</a> 149 1.1 tron </h2> 150 1.1 tron 151 1.1 tron <p> Large numbers in the qshape output represent a large number of 152 1.1 tron messages that are destined to (or alleged to come from) a particular 153 1.1 tron domain. It should be possible to tell at a glance which domains 154 1.1 tron dominate the queue sender or recipient counts, approximately when 155 1.1 tron a burst of mail started, and when it stopped. </p> 156 1.1 tron 157 1.1 tron <p> The problem destinations or sender domains appear near the top 158 1.1.1.4 christos left corner of the output table. Remember that the "<a href="QSHAPE_README.html#active_queue">active" queue</a> 159 1.1 tron can accommodate up to 20000 ($<a href="postconf.5.html#qmgr_message_active_limit">qmgr_message_active_limit</a>) messages. 160 1.1 tron To check whether this limit has been reached, use: </p> 161 1.1 tron 162 1.1 tron <blockquote> 163 1.1 tron <pre> 164 1.1 tron $ qshape -s active <i>(show sender statistics)</i> 165 1.1 tron </pre> 166 1.1 tron </blockquote> 167 1.1 tron 168 1.1.1.4 christos <p> If the total sender count is below 20000 the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is 169 1.1 tron not yet saturated, any high volume sender domains show near the 170 1.1 tron top of the output. 171 1.1 tron 172 1.1.1.4 christos <p> With <a href="qmgr.8.html">oqmgr(8)</a> the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is also limited to at most 20000 173 1.1 tron recipient addresses ($<a href="postconf.5.html#qmgr_message_recipient_limit">qmgr_message_recipient_limit</a>). To check for 174 1.1 tron exhaustion of this limit use: </p> 175 1.1 tron 176 1.1 tron <blockquote> 177 1.1 tron <pre> 178 1.1 tron $ qshape active <i>(show recipient statistics)</i> 179 1.1 tron </pre> 180 1.1 tron </blockquote> 181 1.1 tron 182 1.1 tron <p> Having found the high volume domains, it is often useful to 183 1.1 tron search the logs for recent messages pertaining to the domains in 184 1.1 tron question. </p> 185 1.1 tron 186 1.1 tron <blockquote> 187 1.1 tron <pre> 188 1.1 tron # Find deliveries to example.com 189 1.1 tron # 190 1.1 tron $ tail -10000 /var/log/maillog | 191 1.1.1.5 christos grep -E -i ': to=<.*@example\.com>,' | 192 1.1 tron less 193 1.1 tron 194 1.1 tron # Find messages from example.com 195 1.1 tron # 196 1.1 tron $ tail -10000 /var/log/maillog | 197 1.1.1.5 christos grep -E -i ': from=<.*@example\.com>,' | 198 1.1 tron less 199 1.1 tron </pre> 200 1.1 tron </blockquote> 201 1.1 tron 202 1.1 tron <p> You may want to drill in on some specific queue ids: </p> 203 1.1 tron 204 1.1 tron <blockquote> 205 1.1 tron <pre> 206 1.1 tron # Find all messages for a specific queue id. 207 1.1 tron # 208 1.1.1.5 christos $ tail -10000 /var/log/maillog | grep -E ': 2B2173FF68: ' 209 1.1 tron </pre> 210 1.1 tron </blockquote> 211 1.1 tron 212 1.1 tron <p> Also look for queue manager warning messages in the log. These 213 1.1 tron warnings can suggest strategies to reduce congestion. </p> 214 1.1 tron 215 1.1 tron <blockquote> 216 1.1 tron <pre> 217 1.1.1.5 christos $ grep -E 'qmgr.*(panic|fatal|error|warning):' /var/log/maillog 218 1.1 tron </pre> 219 1.1 tron </blockquote> 220 1.1 tron 221 1.1 tron <p> When all else fails try the Postfix mailing list for help, but 222 1.1 tron please don't forget to include the top 10 or 20 lines of <a href="qshape.1.html">qshape(1)</a> 223 1.1 tron output. </p> 224 1.1 tron 225 1.1 tron <h2><a name="healthy">Example 1: Healthy queue</a></h2> 226 1.1 tron 227 1.1.1.4 christos <p> When looking at just the "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queues</a>, under 228 1.1.1.4 christos normal conditions (no congestion) the "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queues</a> 229 1.1 tron are nearly empty. Mail leaves the system almost as quickly as it 230 1.1.1.4 christos comes in or is deferred without congestion in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. 231 1.1 tron </p> 232 1.1 tron 233 1.1 tron <blockquote> 234 1.1 tron <pre> 235 1.1.1.4 christos $ qshape <i>(show "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queue</a> status)</i> 236 1.1 tron 237 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 238 1.1 tron TOTAL 5 0 0 0 1 0 0 0 1 1 2 239 1.1 tron meri.uwasa.fi 5 0 0 0 1 0 0 0 1 1 2 240 1.1 tron </pre> 241 1.1 tron </blockquote> 242 1.1 tron 243 1.1.1.4 christos <p> If one looks at the two queues separately, the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> 244 1.1 tron is empty or perhaps briefly has one or two messages, while the 245 1.1.1.4 christos "<a href="QSHAPE_README.html#active_queue">active" queue</a> holds more messages and for a somewhat longer time: 246 1.1 tron </p> 247 1.1 tron 248 1.1 tron <blockquote> 249 1.1 tron <pre> 250 1.1 tron $ qshape incoming 251 1.1 tron 252 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 253 1.1 tron TOTAL 0 0 0 0 0 0 0 0 0 0 0 254 1.1 tron 255 1.1 tron $ qshape active 256 1.1 tron 257 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 258 1.1 tron TOTAL 5 0 0 0 1 0 0 0 1 1 2 259 1.1 tron meri.uwasa.fi 5 0 0 0 1 0 0 0 1 1 2 260 1.1 tron </pre> 261 1.1 tron </blockquote> 262 1.1 tron 263 1.1 tron <h2><a name="dictionary_bounce">Example 2: Deferred queue full of 264 1.1 tron dictionary attack bounces</a></h2> 265 1.1 tron 266 1.1 tron <p> This is from a server where recipient validation is not yet 267 1.1 tron available for some of the <a href="VIRTUAL_README.html#canonical">hosted domains</a>. Dictionary attacks on 268 1.1 tron the unvalidated domains result in bounce backscatter. The bounces 269 1.1 tron dominate the queue, but with proper tuning they do not saturate the 270 1.1.1.4 christos "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> or "<a href="QSHAPE_README.html#active_queue">active" queues</a>. The high volume of deferred mail is not 271 1.1 tron a direct cause for alarm. </p> 272 1.1 tron 273 1.1 tron <blockquote> 274 1.1 tron <pre> 275 1.1 tron $ qshape deferred | head 276 1.1 tron 277 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 278 1.1 tron TOTAL 2234 4 2 5 9 31 57 108 201 464 1353 279 1.1 tron heyhihellothere.com 207 0 0 1 1 6 6 8 25 68 92 280 1.1 tron pleazerzoneprod.com 105 0 0 0 0 0 0 0 5 44 56 281 1.1 tron groups.msn.com 63 2 1 2 4 4 14 14 14 8 0 282 1.1 tron orion.toppoint.de 49 0 0 0 1 0 2 4 3 16 23 283 1.1 tron kali.com.cn 46 0 0 0 0 1 0 2 6 12 25 284 1.1 tron meri.uwasa.fi 44 0 0 0 0 1 0 2 8 11 22 285 1.1 tron gjr.paknet.com.pk 43 1 0 0 1 1 3 3 6 12 16 286 1.1 tron aristotle.algonet.se 41 0 0 0 0 0 1 2 11 12 15 287 1.1 tron </pre> 288 1.1 tron </blockquote> 289 1.1 tron 290 1.1 tron <p> The domains shown are mostly bulk-mailers and all the volume 291 1.1 tron is the tail end of the time distribution, showing that short term 292 1.1 tron arrival rates are moderate. Larger numbers and lower message ages 293 1.1 tron are more indicative of current trouble. Old mail still going nowhere 294 1.1.1.4 christos is largely harmless so long as the "<a href="QSHAPE_README.html#active_queue">active"</a> and "<a href="QSHAPE_README.html#incoming_queue">incoming" queues</a> are 295 1.1 tron short. We can also see that the groups.msn.com undeliverables are 296 1.1 tron low rate steady stream rather than a concentrated dictionary attack 297 1.1 tron that is now over. </p> 298 1.1 tron 299 1.1 tron <blockquote> 300 1.1 tron <pre> 301 1.1 tron $ qshape -s deferred | head 302 1.1 tron 303 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 304 1.1 tron TOTAL 2193 4 4 5 8 33 56 104 205 465 1309 305 1.1 tron MAILER-DAEMON 1709 4 4 5 8 33 55 101 198 452 849 306 1.1 tron example.com 263 0 0 0 0 0 0 0 0 2 261 307 1.1 tron example.org 209 0 0 0 0 0 1 3 6 11 188 308 1.1 tron example.net 6 0 0 0 0 0 0 0 0 0 6 309 1.1 tron example.edu 3 0 0 0 0 0 0 0 0 0 3 310 1.1 tron example.gov 2 0 0 0 0 0 0 0 1 0 1 311 1.1 tron example.mil 1 0 0 0 0 0 0 0 0 0 1 312 1.1 tron </pre> 313 1.1 tron </blockquote> 314 1.1 tron 315 1.1 tron <p> Looking at the sender distribution, we see that as expected 316 1.1 tron most of the messages are bounces. </p> 317 1.1 tron 318 1.1 tron <h2><a name="active_congestion">Example 3: Congestion in the active 319 1.1 tron queue</a></h2> 320 1.1 tron 321 1.1 tron <p> This example is taken from a Feb 2004 discussion on the Postfix 322 1.1.1.4 christos Users list. Congestion was reported with the 323 1.1.1.4 christos "<a href="QSHAPE_README.html#active_queue">active"</a> and "<a href="QSHAPE_README.html#incoming_queue">incoming" queues</a> 324 1.1.1.4 christos large and not shrinking despite very large delivery agent 325 1.1 tron process limits. The thread is archived at: 326 1.1.1.6 christos <a href="https://web.archive.org/web/20120227170207/http://archives.neohapsis.com/archives/postfix/2004-02/thread.html#1371">https://web.archive.org/web/20120227170207/http://archives.neohapsis.com/archives/postfix/2004-02/thread.html#1371</a> 327 1.1.1.2 tron </p> 328 1.1 tron 329 1.1 tron <p> Using an older version of <a href="qshape.1.html">qshape(1)</a> it was quickly determined 330 1.1 tron that all the messages were for just a few destinations: </p> 331 1.1 tron 332 1.1 tron <blockquote> 333 1.1 tron <pre> 334 1.1.1.4 christos $ qshape <i>(show "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queue</a> status)</i> 335 1.1 tron 336 1.1 tron T A 5 10 20 40 80 160 320 320+ 337 1.1 tron TOTAL 11775 9996 0 0 1 1 42 94 221 1420 338 1.1 tron user.sourceforge.net 7678 7678 0 0 0 0 0 0 0 0 339 1.1 tron lists.sourceforge.net 2313 2313 0 0 0 0 0 0 0 0 340 1.1 tron gzd.gotdns.com 102 0 0 0 0 0 0 0 2 100 341 1.1 tron </pre> 342 1.1 tron </blockquote> 343 1.1 tron 344 1.1.1.4 christos <p> The "A" column showed the count of messages in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>, 345 1.1.1.4 christos and the numbered columns showed totals for the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. At 346 1.1.1.4 christos 10000 messages (Postfix 1.x "<a href="QSHAPE_README.html#active_queue">active" queue</a> size limit) the "<a href="QSHAPE_README.html#active_queue">active" queue</a> 347 1.1.1.4 christos is full. The "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> was growing rapidly. </p> 348 1.1 tron 349 1.1 tron <p> With the trouble destinations clearly identified, the administrator 350 1.1 tron quickly found and fixed the problem. It is substantially harder to 351 1.1 tron glean the same information from the logs. While a careful reading 352 1.1 tron of <a href="mailq.1.html">mailq(1)</a> output should yield similar results, it is much harder 353 1.1 tron to gauge the magnitude of the problem by looking at the queue 354 1.1 tron one message at a time. </p> 355 1.1 tron 356 1.1 tron <h2><a name="backlog">Example 4: High volume destination backlog</a></h2> 357 1.1 tron 358 1.1 tron <p> When a site you send a lot of email to is down or slow, mail 359 1.1.1.4 christos messages will rapidly build up in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>, or worse, in 360 1.1.1.4 christos the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. The qshape output will show large numbers for 361 1.1 tron the destination domain in all age buckets that overlap the starting 362 1.1 tron time of the problem: </p> 363 1.1 tron 364 1.1 tron <blockquote> 365 1.1 tron <pre> 366 1.1 tron $ qshape deferred | head 367 1.1 tron 368 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 369 1.1 tron TOTAL 5000 200 200 400 800 1600 1000 200 200 200 200 370 1.1 tron highvolume.com 4000 160 160 320 640 1280 1440 0 0 0 0 371 1.1 tron ... 372 1.1 tron </pre> 373 1.1 tron </blockquote> 374 1.1 tron 375 1.1 tron <p> Here the "highvolume.com" destination is continuing to accumulate 376 1.1.1.4 christos deferred mail. The "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queues</a> are fine, but the 377 1.1.1.4 christos "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> started growing some time between 1 and 2 hours ago 378 1.1 tron and continues to grow. </p> 379 1.1 tron 380 1.1 tron <p> If the high volume destination is not down, but is instead 381 1.1.1.4 christos slow, one might see similar congestion in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. 382 1.1.1.4 christos "<a href="QSHAPE_README.html#active_queue">Active" queue</a> congestion is a greater cause for alarm; one might need to 383 1.1 tron take measures to ensure that the mail is deferred instead or even 384 1.1 tron add an <a href="access.5.html">access(5)</a> rule asking the sender to try again later. </p> 385 1.1 tron 386 1.1 tron <p> If a high volume destination exhibits frequent bursts of consecutive 387 1.1 tron connections refused by all MX hosts or "421 Server busy errors", it 388 1.1 tron is possible for the queue manager to mark the destination as "dead" 389 1.1 tron despite the transient nature of the errors. The destination will be 390 1.1 tron retried again after the expiration of a $<a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> timer. 391 1.1 tron If the error bursts are frequent enough it may be that only a small 392 1.1 tron quantity of email is delivered before the destination is again marked 393 1.1 tron "dead". In some cases enabling static (not on demand) connection 394 1.1 tron caching by listing the appropriate nexthop domain in a table included in 395 1.1 tron "<a href="postconf.5.html#smtp_connection_cache_destinations">smtp_connection_cache_destinations</a>" may help to reduce the error rate, 396 1.1 tron because most messages will re-use existing connections. </p> 397 1.1 tron 398 1.1 tron <p> The MTA that has been observed most frequently to exhibit such 399 1.1 tron bursts of errors is Microsoft Exchange, which refuses connections 400 1.1 tron under load. Some proxy virus scanners in front of the Exchange 401 1.1 tron server propagate the refused connection to the client as a "421" 402 1.1 tron error. </p> 403 1.1 tron 404 1.1 tron <p> Note that it is now possible to configure Postfix to exhibit similarly 405 1.1 tron erratic behavior by misconfiguring the <a href="anvil.8.html">anvil(8)</a> service. Do not use 406 1.1 tron <a href="anvil.8.html">anvil(8)</a> for steady-state rate limiting, its purpose is (unintentional) 407 1.1 tron DoS prevention and the rate limits set should be very generous! </p> 408 1.1 tron 409 1.1 tron <p> If one finds oneself needing to deliver a high volume of mail to a 410 1.1 tron destination that exhibits frequent brief bursts of errors and connection 411 1.1 tron caching does not solve the problem, there is a subtle workaround. </p> 412 1.1 tron 413 1.1 tron <ul> 414 1.1 tron 415 1.1 tron <li> <p> Postfix version 2.5 and later: </p> 416 1.1 tron 417 1.1 tron <ul> 418 1.1 tron 419 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" transport 420 1.1 tron for the destination in question. In the example below we will call 421 1.1 tron it "fragile". </p> 422 1.1 tron 423 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> configure a reasonable process limit for the 424 1.1 tron cloned smtp transport (a number in the 10-20 range is typical). </p> 425 1.1 tron 426 1.1 tron <li> <p> IMPORTANT!!! In <a href="postconf.5.html">main.cf</a> configure a large per-destination 427 1.1 tron pseudo-cohort failure limit for the cloned smtp transport. </p> 428 1.1 tron 429 1.1 tron <pre> 430 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 431 1.1.1.3 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = <a href="DATABASE_README.html#types">hash</a>:/etc/postfix/transport 432 1.1 tron fragile_destination_concurrency_failed_cohort_limit = 100 433 1.1 tron fragile_destination_concurrency_limit = 20 434 1.1 tron 435 1.1 tron /etc/postfix/transport: 436 1.1 tron example.com fragile: 437 1.1 tron 438 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 439 1.1 tron # service type private unpriv chroot wakeup maxproc command 440 1.1 tron fragile unix - - n - 20 smtp 441 1.1 tron </pre> 442 1.1 tron 443 1.1 tron <p> See also the documentation for 444 1.1 tron <a href="postconf.5.html#default_destination_concurrency_failed_cohort_limit">default_destination_concurrency_failed_cohort_limit</a> and 445 1.1 tron <a href="postconf.5.html#default_destination_concurrency_limit">default_destination_concurrency_limit</a>. </p> 446 1.1 tron 447 1.1 tron </ul> 448 1.1 tron 449 1.1 tron <li> <p> Earlier Postfix versions: </p> 450 1.1 tron 451 1.1 tron <ul> 452 1.1 tron 453 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" 454 1.1 tron transport for the destination in question. In the example below 455 1.1 tron we will call it "fragile". </p> 456 1.1 tron 457 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> configure a reasonable process limit for the 458 1.1 tron transport (a number in the 10-20 range is typical). </p> 459 1.1 tron 460 1.1 tron <li> <p> IMPORTANT!!! In <a href="postconf.5.html">main.cf</a> configure a very large initial 461 1.1 tron and destination concurrency limit for this transport (say 2000). </p> 462 1.1 tron 463 1.1 tron <pre> 464 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 465 1.1.1.3 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = <a href="DATABASE_README.html#types">hash</a>:/etc/postfix/transport 466 1.1 tron <a href="postconf.5.html#initial_destination_concurrency">initial_destination_concurrency</a> = 2000 467 1.1 tron fragile_destination_concurrency_limit = 2000 468 1.1 tron 469 1.1 tron /etc/postfix/transport: 470 1.1 tron example.com fragile: 471 1.1 tron 472 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 473 1.1 tron # service type private unpriv chroot wakeup maxproc command 474 1.1 tron fragile unix - - n - 20 smtp 475 1.1 tron </pre> 476 1.1 tron 477 1.1 tron <p> See also the documentation for <a href="postconf.5.html#default_destination_concurrency_limit">default_destination_concurrency_limit</a>. 478 1.1 tron </p> 479 1.1 tron 480 1.1 tron </ul> 481 1.1 tron 482 1.1 tron </ul> 483 1.1 tron 484 1.1 tron <p> The effect of this configuration is that up to 2000 485 1.1 tron consecutive errors are tolerated without marking the destination 486 1.1 tron dead, while the total concurrency remains reasonable (10-20 487 1.1 tron processes). This trick is only for a very specialized situation: 488 1.1 tron high volume delivery into a channel with multi-error bursts 489 1.1 tron that is capable of high throughput, but is repeatedly throttled by 490 1.1 tron the bursts of errors. </p> 491 1.1 tron 492 1.1 tron <p> When a destination is unable to handle the load even after the 493 1.1 tron Postfix process limit is reduced to 1, a desperate measure is to 494 1.1 tron insert brief delays between delivery attempts. </p> 495 1.1 tron 496 1.1 tron <ul> 497 1.1 tron 498 1.1 tron <li> <p> Postfix version 2.5 and later: </p> 499 1.1 tron 500 1.1 tron <ul> 501 1.1 tron 502 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" transport 503 1.1 tron for the problem destination. In the example below we call it "slow". 504 1.1 tron </p> 505 1.1 tron 506 1.1 tron <li> <p> In <a href="postconf.5.html">main.cf</a> configure a short delay between deliveries to 507 1.1 tron the same destination. </p> 508 1.1 tron 509 1.1 tron <pre> 510 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 511 1.1.1.3 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = <a href="DATABASE_README.html#types">hash</a>:/etc/postfix/transport 512 1.1 tron slow_destination_rate_delay = 1 513 1.1.1.2 tron slow_destination_concurrency_failed_cohort_limit = 100 514 1.1 tron 515 1.1 tron /etc/postfix/transport: 516 1.1 tron example.com slow: 517 1.1 tron 518 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 519 1.1 tron # service type private unpriv chroot wakeup maxproc command 520 1.1 tron slow unix - - n - - smtp 521 1.1 tron </pre> 522 1.1 tron 523 1.1 tron </ul> 524 1.1 tron 525 1.1 tron <p> See also the documentation for <a href="postconf.5.html#default_destination_rate_delay">default_destination_rate_delay</a>. </p> 526 1.1 tron 527 1.1 tron <p> This solution forces the Postfix <a href="smtp.8.html">smtp(8)</a> client to wait for 528 1.1 tron $slow_destination_rate_delay seconds between deliveries to the same 529 1.1 tron destination. </p> 530 1.1 tron 531 1.1.1.2 tron <p> IMPORTANT!! The large slow_destination_concurrency_failed_cohort_limit 532 1.1.1.2 tron value is needed. This prevents Postfix from deferring all mail for 533 1.1.1.2 tron the same destination after only one connection or handshake error 534 1.1.1.2 tron (the reason for this is that non-zero slow_destination_rate_delay 535 1.1.1.2 tron forces a per-destination concurrency of 1). </p> 536 1.1.1.2 tron 537 1.1 tron <li> <p> Earlier Postfix versions: </p> 538 1.1 tron 539 1.1 tron <ul> 540 1.1 tron 541 1.1 tron <li> <p> In the transport map entry for the problem destination, 542 1.1 tron specify a dead host as the primary nexthop. </p> 543 1.1 tron 544 1.1 tron <li> <p> In the <a href="master.5.html">master.cf</a> entry for the transport specify the 545 1.1 tron problem destination as the <a href="postconf.5.html#fallback_relay">fallback_relay</a> and specify a small 546 1.1 tron <a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a> value. </p> 547 1.1 tron 548 1.1 tron <pre> 549 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 550 1.1.1.3 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = <a href="DATABASE_README.html#types">hash</a>:/etc/postfix/transport 551 1.1 tron 552 1.1 tron /etc/postfix/transport: 553 1.1 tron example.com slow:[dead.host] 554 1.1 tron 555 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 556 1.1 tron # service type private unpriv chroot wakeup maxproc command 557 1.1 tron slow unix - - n - 1 smtp 558 1.1 tron -o <a href="postconf.5.html#fallback_relay">fallback_relay</a>=problem.example.com 559 1.1 tron -o <a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a>=1 560 1.1 tron -o <a href="postconf.5.html#smtp_connection_cache_on_demand">smtp_connection_cache_on_demand</a>=no 561 1.1 tron </pre> 562 1.1 tron 563 1.1 tron </ul> 564 1.1 tron 565 1.1 tron <p> This solution forces the Postfix <a href="smtp.8.html">smtp(8)</a> client to wait for 566 1.1 tron $<a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a> seconds between deliveries. The connection 567 1.1 tron caching feature is disabled to prevent the client from skipping 568 1.1 tron over the dead host. </p> 569 1.1 tron 570 1.1 tron </ul> 571 1.1 tron 572 1.1 tron <h2><a name="queues">Postfix queue directories</a></h2> 573 1.1 tron 574 1.1 tron <p> The following sections describe Postfix queues: their purpose, 575 1.1 tron what normal behavior looks like, and how to diagnose abnormal 576 1.1 tron behavior. </p> 577 1.1 tron 578 1.1 tron <h3> <a name="maildrop_queue"> The "maildrop" queue </a> </h3> 579 1.1 tron 580 1.1 tron <p> Messages that have been submitted via the Postfix <a href="sendmail.1.html">sendmail(1)</a> 581 1.1 tron command, but not yet brought into the main Postfix queue by the 582 1.1 tron <a href="pickup.8.html">pickup(8)</a> service, await processing in the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a>. Messages 583 1.1 tron can be added to the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> even when the Postfix system 584 1.1 tron is not running. They will begin to be processed once Postfix is 585 1.1 tron started. </p> 586 1.1 tron 587 1.1 tron <p> The "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> is drained by the single threaded <a href="pickup.8.html">pickup(8)</a> 588 1.1 tron service scanning the queue directory periodically or when notified 589 1.1 tron of new message arrival by the <a href="postdrop.1.html">postdrop(1)</a> program. The <a href="postdrop.1.html">postdrop(1)</a> 590 1.1 tron program is a setgid helper that allows the unprivileged Postfix 591 1.1 tron <a href="sendmail.1.html">sendmail(1)</a> program to inject mail into the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> and 592 1.1 tron to notify the <a href="pickup.8.html">pickup(8)</a> service of its arrival. </p> 593 1.1 tron 594 1.1 tron <p> All mail that enters the main Postfix queue does so via the 595 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> service. The cleanup service is responsible for envelope 596 1.1 tron and header rewriting, header and body regular expression checks, 597 1.1 tron automatic bcc recipient processing, milter content processing, and 598 1.1 tron reliable insertion of the message into the Postfix "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>. </p> 599 1.1 tron 600 1.1 tron <p> In the absence of excessive CPU consumption in <a href="cleanup.8.html">cleanup(8)</a> header 601 1.1 tron or body regular expression checks or other software consuming all 602 1.1 tron available CPU resources, Postfix performance is disk I/O bound. 603 1.1 tron The rate at which the <a href="pickup.8.html">pickup(8)</a> service can inject messages into 604 1.1 tron the queue is largely determined by disk access times, since the 605 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> service must commit the message to stable storage before 606 1.1 tron returning success. The same is true of the <a href="postdrop.1.html">postdrop(1)</a> program 607 1.1 tron writing the message to the "maildrop" directory. </p> 608 1.1 tron 609 1.1 tron <p> As the pickup service is single threaded, it can only deliver 610 1.1 tron one message at a time at a rate that does not exceed the reciprocal 611 1.1 tron disk I/O latency (+ CPU if not negligible) of the cleanup service. 612 1.1 tron </p> 613 1.1 tron 614 1.1 tron <p> Congestion in this queue is indicative of an excessive local message 615 1.1 tron submission rate or perhaps excessive CPU consumption in the <a href="cleanup.8.html">cleanup(8)</a> 616 1.1 tron service due to excessive <a href="postconf.5.html#body_checks">body_checks</a>, or (Postfix ≥ 2.3) high latency 617 1.1 tron milters. </p> 618 1.1 tron 619 1.1.1.4 christos <p> Note, that once the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is full, the cleanup service 620 1.1 tron will attempt to slow down message injection by pausing $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> 621 1.1 tron for each message. In this case "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> congestion may be 622 1.1 tron a consequence of congestion downstream, rather than a problem in 623 1.1 tron its own right. </p> 624 1.1 tron 625 1.1 tron <p> Note, you should not attempt to deliver large volumes of mail via 626 1.1 tron the <a href="pickup.8.html">pickup(8)</a> service. High volume sites should avoid using "simple" 627 1.1 tron content filters that re-inject scanned mail via Postfix <a href="sendmail.1.html">sendmail(1)</a> 628 1.1 tron and <a href="postdrop.1.html">postdrop(1)</a>. </p> 629 1.1 tron 630 1.1 tron <p> A high arrival rate of locally submitted mail may be an indication 631 1.1 tron of an uncaught forwarding loop, or a run-away notification program. 632 1.1 tron Try to keep the volume of local mail injection to a moderate level. 633 1.1 tron </p> 634 1.1 tron 635 1.1 tron <p> The "postsuper -r" command can place selected messages into 636 1.1 tron the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> for reprocessing. This is most useful for 637 1.1 tron resetting any stale <a href="postconf.5.html#content_filter">content_filter</a> settings. Requeuing a large number 638 1.1 tron of messages using "postsuper -r" can clearly cause a spike in the 639 1.1 tron size of the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a>. </p> 640 1.1 tron 641 1.1 tron <h3> <a name="hold_queue"> The "hold" queue </a> </h3> 642 1.1 tron 643 1.1 tron <p> The administrator can define "smtpd" <a href="access.5.html">access(5)</a> policies, or 644 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> header/body checks that cause messages to be automatically 645 1.1 tron diverted from normal processing and placed indefinitely in the 646 1.1.1.4 christos "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>. Messages placed in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> stay there until 647 1.1 tron the administrator intervenes. No periodic delivery attempts are 648 1.1 tron made for messages in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>. The <a href="postsuper.1.html">postsuper(1)</a> command 649 1.1 tron can be used to manually release messages into the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. 650 1.1 tron </p> 651 1.1 tron 652 1.1 tron <p> Messages can potentially stay in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> longer than 653 1.1 tron $<a href="postconf.5.html#maximal_queue_lifetime">maximal_queue_lifetime</a>. If such "old" messages need to be released from 654 1.1.1.4 christos the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>, they should typically be moved into the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> 655 1.1.1.4 christos using "postsuper -r", so that the message gets a new timestamp and 656 1.1 tron is given more than one opportunity to be delivered. Messages that are 657 1.1 tron "young" can be moved directly into the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> using 658 1.1 tron "postsuper -H". </p> 659 1.1 tron 660 1.1 tron <p> The "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> plays little role in Postfix performance, and 661 1.1 tron monitoring of the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> is typically more closely motivated 662 1.1 tron by tracking spam and malware, than by performance issues. </p> 663 1.1 tron 664 1.1 tron <h3> <a name="incoming_queue"> The "incoming" queue </a> </h3> 665 1.1 tron 666 1.1 tron <p> All new mail entering the Postfix queue is written by the 667 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> service into the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>. New queue files are 668 1.1 tron created owned by the "postfix" user with an access bitmask (or 669 1.1 tron mode) of 0600. Once a queue file is ready for further processing 670 1.1 tron the <a href="cleanup.8.html">cleanup(8)</a> service changes the queue file mode to 0700 and 671 1.1 tron notifies the queue manager of new mail arrival. The queue manager 672 1.1 tron ignores incomplete queue files whose mode is 0600, as these are 673 1.1 tron still being written by cleanup. </p> 674 1.1 tron 675 1.1.1.4 christos <p> The queue manager scans the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> bringing any new 676 1.1.1.4 christos mail into the "<a href="QSHAPE_README.html#active_queue">active" queue</a> if the "<a href="QSHAPE_README.html#active_queue">active" queue</a> resource limits 677 1.1.1.4 christos have not been exceeded. By default, the "<a href="QSHAPE_README.html#active_queue">active" queue</a> accommodates 678 1.1.1.4 christos at most 20000 messages. Once the "<a href="QSHAPE_README.html#active_queue">active" queue</a> message limit is 679 1.1.1.4 christos reached, the queue manager stops scanning the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> 680 1.1.1.4 christos (and the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>, see below). </p> 681 1.1 tron 682 1.1.1.4 christos <p> Under normal conditions the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> is nearly empty (has 683 1.1 tron only mode 0600 files), with the queue manager able to import new 684 1.1.1.4 christos messages into the "<a href="QSHAPE_README.html#active_queue">active" queue</a> as soon as they become available. 685 1.1 tron </p> 686 1.1 tron 687 1.1.1.4 christos <p> The "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> grows when the message input rate spikes 688 1.1 tron above the rate at which the queue manager can import messages into 689 1.1.1.4 christos the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. The main factors slowing down the queue manager 690 1.1 tron are disk I/O and lookup queries to the trivial-rewrite service. If the queue 691 1.1 tron manager is routinely not keeping up, consider not using "slow" 692 1.1 tron lookup services (MySQL, LDAP, ...) for transport lookups or speeding 693 1.1 tron up the hosts that provide the lookup service. If the problem is I/O 694 1.1 tron starvation, consider striping the queue over more disks, faster controllers 695 1.1 tron with a battery write cache, or other hardware improvements. At the very 696 1.1 tron least, make sure that the queue directory is mounted with the "noatime" 697 1.1 tron option if applicable to the underlying filesystem. </p> 698 1.1 tron 699 1.1 tron <p> The <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> parameter is used to clamp the input rate 700 1.1 tron when the queue manager starts to fall behind. The <a href="cleanup.8.html">cleanup(8)</a> service 701 1.1 tron will pause for $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> seconds before creating a new queue 702 1.1 tron file if it cannot obtain a "token" from the queue manager. </p> 703 1.1 tron 704 1.1 tron <p> Since the number of <a href="cleanup.8.html">cleanup(8)</a> processes is limited in most 705 1.1 tron cases by the SMTP server concurrency, the input rate can exceed 706 1.1 tron the output rate by at most "SMTP connection count" / $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> 707 1.1 tron messages per second. </p> 708 1.1 tron 709 1.1 tron <p> With a default process limit of 100, and an <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> of 710 1.1 tron 1s, the coupling is strong enough to limit a single run-away injector 711 1.1 tron to 1 message per second, but is not strong enough to deflect an 712 1.1 tron excessive input rate from many sources at the same time. </p> 713 1.1 tron 714 1.1 tron <p> If a server is being hammered from multiple directions, consider 715 1.1.1.4 christos raising the <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> to 10 seconds, but only if the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> 716 1.1.1.4 christos is growing even while the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is not full and the 717 1.1 tron trivial-rewrite service is using a fast transport lookup mechanism. 718 1.1 tron </p> 719 1.1 tron 720 1.1 tron <h3> <a name="active_queue"> The "active" queue </a> </h3> 721 1.1 tron 722 1.1 tron <p> The queue manager is a delivery agent scheduler; it works to 723 1.1 tron ensure fast and fair delivery of mail to all destinations within 724 1.1 tron designated resource limits. </p> 725 1.1 tron 726 1.1.1.4 christos <p> The "<a href="QSHAPE_README.html#active_queue">active" queue</a> is somewhat analogous to an operating system's 727 1.1.1.4 christos process run queue. Messages in the "<a href="QSHAPE_README.html#active_queue">active" queue</a> are ready to be 728 1.1 tron sent (runnable), but are not necessarily in the process of being 729 1.1 tron sent (running). </p> 730 1.1 tron 731 1.1 tron <p> While most Postfix administrators think of the "<a href="QSHAPE_README.html#active_queue">active" queue</a> 732 1.1 tron as a directory on disk, the real "<a href="QSHAPE_README.html#active_queue">active" queue</a> is a set of data 733 1.1 tron structures in the memory of the queue manager process. </p> 734 1.1 tron 735 1.1.1.4 christos <p> Messages in the "<a href="QSHAPE_README.html#maildrop_queue">maildrop"</a>, "<a href="QSHAPE_README.html#hold_queue">hold"</a>, "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a> 736 1.1.1.4 christos (see below) do not occupy memory; they are safely stored on 737 1.1 tron disk waiting for their turn to be processed. The envelope information 738 1.1 tron for messages in the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is managed in memory, allowing 739 1.1 tron the queue manager to do global scheduling, allocating available 740 1.1.1.4 christos delivery agent processes to an appropriate message in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. </p> 741 1.1 tron 742 1.1.1.4 christos <p> Within the "<a href="QSHAPE_README.html#active_queue">active" queue</a>, (multi-recipient) messages are broken 743 1.1 tron up into groups of recipients that share the same transport/nexthop 744 1.1 tron combination; the group size is capped by the transport's recipient 745 1.1 tron concurrency limit. </p> 746 1.1 tron 747 1.1 tron <p> Multiple recipient groups (from one or more messages) are queued 748 1.1 tron for delivery grouped by transport/nexthop combination. The 749 1.1 tron <b>destination</b> concurrency limit for the transports caps the number 750 1.1 tron of simultaneous delivery attempts for each nexthop. Transports with 751 1.1 tron a <b>recipient</b> concurrency limit of 1 are special: these are grouped 752 1.1 tron by the actual recipient address rather than the nexthop, yielding 753 1.1 tron per-recipient concurrency limits rather than per-domain 754 1.1 tron concurrency limits. Per-recipient limits are appropriate when 755 1.1 tron performing final delivery to mailboxes rather than when relaying 756 1.1 tron to a remote server. </p> 757 1.1 tron 758 1.1.1.4 christos <p> Congestion occurs in the "<a href="QSHAPE_README.html#active_queue">active" queue</a> when one or more destinations 759 1.1 tron drain slower than the corresponding message input rate. </p> 760 1.1 tron 761 1.1.1.4 christos <p> Input into the "<a href="QSHAPE_README.html#active_queue">active" queue</a> comes both from new mail in the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>, 762 1.1.1.4 christos and retries of mail in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. Should the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 763 1.1.1.4 christos get really large, retries of old mail can dominate the arrival 764 1.1 tron rate of new mail. Systems with more CPU, faster disks and more network 765 1.1.1.4 christos bandwidth can deal with larger "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a>, but as a rule of thumb 766 1.1.1.4 christos the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> scales to somewhere between 100,000 and 1,000,000 767 1.1 tron messages with good performance unlikely above that "limit". Systems with 768 1.1 tron queues this large should typically stop accepting new mail, or put the 769 1.1 tron backlog "on hold" until the underlying issue is fixed (provided that 770 1.1 tron there is enough capacity to handle just the new mail). </p> 771 1.1 tron 772 1.1 tron <p> When a destination is down for some time, the queue manager will 773 1.1 tron mark it dead, and immediately defer all mail for the destination without 774 1.1 tron trying to assign it to a delivery agent. In this case the messages 775 1.1.1.4 christos will quickly leave the "<a href="QSHAPE_README.html#active_queue">active" queue</a> and end up in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 776 1.1 tron (with Postfix < 2.4, this is done directly by the queue manager, 777 1.1 tron with Postfix ≥ 2.4 this is done via the "retry" delivery agent). </p> 778 1.1 tron 779 1.1 tron <p> When the destination is instead simply slow, or there is a problem 780 1.1.1.4 christos causing an excessive arrival rate the "<a href="QSHAPE_README.html#active_queue">active" queue</a> will grow and will 781 1.1 tron become dominated by mail to the congested destination. </p> 782 1.1 tron 783 1.1 tron <p> The only way to reduce congestion is to either reduce the input 784 1.1 tron rate or increase the throughput. Increasing the throughput requires 785 1.1 tron either increasing the concurrency or reducing the latency of 786 1.1 tron deliveries. </p> 787 1.1 tron 788 1.1 tron <p> For high volume sites a key tuning parameter is the number of 789 1.1 tron "smtp" delivery agents allocated to the "smtp" and "relay" transports. 790 1.1 tron High volume sites tend to send to many different destinations, many 791 1.1 tron of which may be down or slow, so a good fraction of the available 792 1.1 tron delivery agents will be blocked waiting for slow sites. Also mail 793 1.1 tron destined across the globe will incur large SMTP command-response 794 1.1 tron latencies, so high message throughput can only be achieved with 795 1.1 tron more concurrent delivery agents. </p> 796 1.1 tron 797 1.1 tron <p> The default "smtp" process limit of 100 is good enough for most 798 1.1 tron sites, and may even need to be lowered for sites with low bandwidth 799 1.1 tron connections (no use increasing concurrency once the network pipe 800 1.1 tron is full). When one finds that the queue is growing on an "idle" 801 1.1 tron system (CPU, disk I/O and network not exhausted) the remaining 802 1.1 tron reason for congestion is insufficient concurrency in the face of 803 1.1 tron a high average latency. If the number of outbound SMTP connections 804 1.1 tron (either ESTABLISHED or SYN_SENT) reaches the process limit, mail 805 1.1 tron is draining slowly and the system and network are not loaded, raise 806 1.1 tron the "smtp" and/or "relay" process limits! </p> 807 1.1 tron 808 1.1 tron <p> When a high volume destination is served by multiple MX hosts with 809 1.1 tron typically low delivery latency, performance can suffer dramatically when 810 1.1 tron one of the MX hosts is unresponsive and SMTP connections to that host 811 1.1 tron timeout. For example, if there are 2 equal weight MX hosts, the SMTP 812 1.1 tron connection timeout is 30 seconds and one of the MX hosts is down, the 813 1.1 tron average SMTP connection will take approximately 15 seconds to complete. 814 1.1 tron With a default per-destination concurrency limit of 20 connections, 815 1.1 tron throughput falls to just over 1 message per second. </p> 816 1.1 tron 817 1.1 tron <p> The best way to avoid bottlenecks when one or more MX hosts is 818 1.1 tron non-responsive is to use connection caching. Connection caching was 819 1.1 tron introduced with Postfix 2.2 and is by default enabled on demand for 820 1.1.1.4 christos destinations with a backlog of mail in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. When connection 821 1.1 tron caching is in effect for a particular destination, established connections 822 1.1 tron are re-used to send additional messages, this reduces the number of 823 1.1 tron connections made per message delivery and maintains good throughput even 824 1.1 tron in the face of partial unavailability of the destination's MX hosts. </p> 825 1.1 tron 826 1.1 tron <p> If connection caching is not available (Postfix < 2.2) or does 827 1.1 tron not provide a sufficient latency reduction, especially for the "relay" 828 1.1 tron transport used to forward mail to "your own" domains, consider setting 829 1.1 tron lower than default SMTP connection timeouts (1-5 seconds) and higher 830 1.1 tron than default destination concurrency limits. This will further reduce 831 1.1 tron latency and provide more concurrency to maintain throughput should 832 1.1 tron latency rise. </p> 833 1.1 tron 834 1.1 tron <p> Setting high concurrency limits to domains that are not your own may 835 1.1 tron be viewed as hostile by the receiving system, and steps may be taken 836 1.1 tron to prevent you from monopolizing the destination system's resources. 837 1.1 tron The defensive measures may substantially reduce your throughput or block 838 1.1 tron access entirely. Do not set aggressive concurrency limits to remote 839 1.1 tron domains without coordinating with the administrators of the target 840 1.1 tron domain. </p> 841 1.1 tron 842 1.1 tron <p> If necessary, dedicate and tune custom transports for selected high 843 1.1 tron volume destinations. The "relay" transport is provided for forwarding mail 844 1.1 tron to domains for which your server is a primary or backup MX host. These can 845 1.1 tron make up a substantial fraction of your email traffic. Use the "relay" and 846 1.1 tron not the "smtp" transport to send email to these domains. Using the "relay" 847 1.1 tron transport allocates a separate delivery agent pool to these destinations 848 1.1 tron and allows separate tuning of timeouts and concurrency limits. </p> 849 1.1 tron 850 1.1 tron <p> Another common cause of congestion is unwarranted flushing of the 851 1.1.1.4 christos entire "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. The "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> holds messages that are likely 852 1.1 tron to fail to be delivered and are also likely to be slow to fail delivery 853 1.1.1.4 christos (time out). As a result the most common reaction to a large "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 854 1.1 tron (flush it!) is more than likely counter-productive, and typically makes 855 1.1.1.4 christos the congestion worse. Do not flush the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> unless you expect 856 1.1 tron that most of its content has recently become deliverable (e.g. <a href="postconf.5.html#relayhost">relayhost</a> 857 1.1 tron back up after an outage)! </p> 858 1.1 tron 859 1.1 tron <p> Note that whenever the queue manager is restarted, there may 860 1.1.1.4 christos already be messages in the "<a href="QSHAPE_README.html#active_queue">active" queue</a> directory, but the "real" 861 1.1.1.4 christos "<a href="QSHAPE_README.html#active_queue">active" queue</a> in memory is empty. In order to recover the in-memory 862 1.1.1.4 christos state, the queue manager moves all the "<a href="QSHAPE_README.html#active_queue">active" queue</a> messages 863 1.1.1.4 christos back into the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>, and then uses its normal "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> 864 1.1.1.4 christos scan to refill the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. The process of moving all 865 1.1 tron the messages back and forth, redoing transport table (<a href="trivial-rewrite.8.html">trivial-rewrite(8)</a> 866 1.1 tron resolve service) lookups, and re-importing the messages back into 867 1.1 tron memory is expensive. At all costs, avoid frequent restarts of the 868 1.1 tron queue manager (e.g. via frequent execution of "postfix reload"). </p> 869 1.1 tron 870 1.1 tron <h3> <a name="deferred_queue"> The "deferred" queue </a> </h3> 871 1.1 tron 872 1.1 tron <p> When all the deliverable recipients for a message are delivered, 873 1.1 tron and for some recipients delivery failed for a transient reason (it 874 1.1.1.4 christos might succeed later), the message is placed in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. 875 1.1 tron </p> 876 1.1 tron 877 1.1.1.4 christos <p> The queue manager scans the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> periodically. The scan 878 1.1.1.4 christos interval is controlled by the <a href="postconf.5.html#queue_run_delay">queue_run_delay</a> parameter. While a "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 879 1.1.1.4 christos scan is in progress, if an "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> scan is also in progress 880 1.1.1.4 christos (ideally these are brief since the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> should be short), the 881 1.1.1.4 christos queue manager alternates between looking for messages in the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> 882 1.1.1.4 christos and in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. This "round-robin" strategy prevents 883 1.1.1.4 christos starvation of either the "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> or the "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a>. </p> 884 1.1.1.4 christos 885 1.1.1.4 christos <p> Each "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> scan only brings a fraction of the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 886 1.1.1.4 christos back into the "<a href="QSHAPE_README.html#active_queue">active" queue</a> for a retry. This is because each 887 1.1.1.4 christos message in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> is assigned a "cool-off" time when 888 1.1 tron it is deferred. This is done by time-warping the modification 889 1.1 tron time of the queue file into the future. The queue file is not 890 1.1 tron eligible for a retry if its modification time is not yet reached. 891 1.1 tron </p> 892 1.1 tron 893 1.1 tron <p> The "cool-off" time is at least $<a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> and at 894 1.1 tron most $<a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a>. The next retry time is set by doubling 895 1.1 tron the message's age in the queue, and adjusting up or down to lie 896 1.1 tron within the limits. This means that young messages are initially 897 1.1 tron retried more often than old messages. </p> 898 1.1 tron 899 1.1.1.4 christos <p> If a high volume site routinely has large "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a>, it 900 1.1 tron may be useful to adjust the <a href="postconf.5.html#queue_run_delay">queue_run_delay</a>, <a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> and 901 1.1 tron <a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a> to provide short enough delays on first failure 902 1.1 tron (Postfix ≥ 2.4 has a sensibly low minimal backoff time by default), 903 1.1 tron with perhaps longer delays after multiple failures, to reduce the 904 1.1 tron retransmission rate of old messages and thereby reduce the quantity 905 1.1.1.4 christos of previously deferred mail in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. If you want a really 906 1.1 tron low <a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a>, you may also want to lower <a href="postconf.5.html#queue_run_delay">queue_run_delay</a>, 907 1.1 tron but understand that more frequent scans will increase the demand for 908 1.1 tron disk I/O. </p> 909 1.1 tron 910 1.1.1.4 christos <p> One common cause of large "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a> is failure to validate 911 1.1 tron recipients at the SMTP input stage. Since spammers routinely launch 912 1.1 tron dictionary attacks from unrepliable sender addresses, the bounces 913 1.1.1.4 christos for invalid recipient addresses clog the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> (and at high 914 1.1.1.4 christos volumes proportionally clog the "<a href="QSHAPE_README.html#active_queue">active" queue</a>). Recipient validation 915 1.1 tron is strongly recommended through use of the <a href="postconf.5.html#local_recipient_maps">local_recipient_maps</a> and 916 1.1 tron <a href="postconf.5.html#relay_recipient_maps">relay_recipient_maps</a> parameters. Even when bounces drain quickly they 917 1.1 tron inundate innocent victims of forgery with unwanted email. To avoid 918 1.1 tron this, do not accept mail for invalid recipients. </p> 919 1.1 tron 920 1.1 tron <p> When a host with lots of deferred mail is down for some time, 921 1.1.1.4 christos it is possible for the entire "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> to reach its retry 922 1.1.1.4 christos time simultaneously. This can lead to a very full "<a href="QSHAPE_README.html#active_queue">active" queue</a> once 923 1.1 tron the host comes back up. The phenomenon can repeat approximately 924 1.1 tron every <a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a> seconds if the messages are again deferred 925 1.1 tron after a brief burst of congestion. Perhaps, a future Postfix release 926 1.1 tron will add a random offset to the retry time (or use a combination 927 1.1.1.4 christos of strategies) to reduce the odds of repeated complete "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 928 1.1.1.4 christos flushes. </p> 929 1.1 tron 930 1.1 tron <h2><a name="credits">Credits</a></h2> 931 1.1 tron 932 1.1 tron <p> The <a href="qshape.1.html">qshape(1)</a> program was developed by Victor Duchovni of Morgan 933 1.1 tron Stanley, who also wrote the initial version of this document. </p> 934 1.1 tron 935 1.1 tron </body> 936 1.1 tron 937 1.1 tron </html> 938
Indexes created Tue Jun 09 00:24:00 UTC 2026