QSHAPE_README.html revision 1.1.1.5
1 1.1 tron <!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN" 2 1.1 tron "http://www.w3.org/TR/html4/loose.dtd"> 3 1.1 tron 4 1.1 tron <html> 5 1.1 tron 6 1.1 tron <head> 7 1.1 tron 8 1.1 tron <title>Postfix Bottleneck Analysis</title> 9 1.1 tron 10 1.1.1.4 christos <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> 11 1.1.1.5 christos <link rel='stylesheet' type='text/css' href='postfix-doc.css'> 12 1.1 tron 13 1.1 tron </head> 14 1.1 tron 15 1.1 tron <body> 16 1.1 tron 17 1.1 tron <h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix Bottleneck Analysis</h1> 18 1.1 tron 19 1.1 tron <hr> 20 1.1 tron 21 1.1 tron <h2>Purpose of this document </h2> 22 1.1 tron 23 1.1 tron <p> This document is an introduction to Postfix queue congestion analysis. 24 1.1 tron It explains how the <a href="qshape.1.html">qshape(1)</a> program can help to track down the 25 1.1 tron reason for queue congestion. <a href="qshape.1.html">qshape(1)</a> is bundled with Postfix 26 1.1 tron 2.1 and later source code, under the "auxiliary" directory. This 27 1.1 tron document describes <a href="qshape.1.html">qshape(1)</a> as bundled with Postfix 2.4. </p> 28 1.1 tron 29 1.1 tron <p> This document covers the following topics: </p> 30 1.1 tron 31 1.1 tron <ul> 32 1.1 tron 33 1.1 tron <li><a href="#qshape">Introducing the qshape tool</a> 34 1.1 tron 35 1.1 tron <li><a href="#trouble_shooting">Trouble shooting with qshape</a> 36 1.1 tron 37 1.1 tron <li><a href="#healthy">Example 1: Healthy queue</a> 38 1.1 tron 39 1.1 tron <li><a href="#dictionary_bounce">Example 2: Deferred queue full of 40 1.1 tron dictionary attack bounces</a></li> 41 1.1 tron 42 1.1 tron <li><a href="#active_congestion">Example 3: Congestion in the active 43 1.1 tron queue</a></li> 44 1.1 tron 45 1.1 tron <li><a href="#backlog">Example 4: High volume destination backlog</a> 46 1.1 tron 47 1.1 tron <li><a href="#queues">Postfix queue directories</a> 48 1.1 tron 49 1.1 tron <ul> 50 1.1 tron 51 1.1 tron <li> <a href="#maildrop_queue"> The "maildrop" queue </a> 52 1.1 tron 53 1.1 tron <li> <a href="#hold_queue"> The "hold" queue </a> 54 1.1 tron 55 1.1 tron <li> <a href="#incoming_queue"> The "incoming" queue </a> 56 1.1 tron 57 1.1 tron <li> <a href="#active_queue"> The "active" queue </a> 58 1.1 tron 59 1.1 tron <li> <a href="#deferred_queue"> The "deferred" queue </a> 60 1.1 tron 61 1.1 tron </ul> 62 1.1 tron 63 1.1 tron <li><a href="#credits">Credits</a> 64 1.1 tron 65 1.1 tron </ul> 66 1.1 tron 67 1.1 tron <h2><a name="qshape">Introducing the qshape tool</a></h2> 68 1.1 tron 69 1.1 tron <p> When mail is draining slowly or the queue is unexpectedly large, 70 1.1 tron run <a href="qshape.1.html">qshape(1)</a> as the super-user (root) to help zero in on the problem. 71 1.1 tron The <a href="qshape.1.html">qshape(1)</a> program displays a tabular view of the Postfix queue 72 1.1 tron contents. </p> 73 1.1 tron 74 1.1 tron <ul> 75 1.1 tron 76 1.1 tron <li> <p> On the horizontal axis, it displays the queue age with 77 1.1 tron fine granularity for recent messages and (geometrically) less fine 78 1.1 tron granularity for older messages. </p> 79 1.1 tron 80 1.1 tron <li> <p> The vertical axis displays the destination (or with the 81 1.1 tron "-s" switch the sender) domain. Domains with the most messages are 82 1.1 tron listed first. </p> 83 1.1 tron 84 1.1 tron </ul> 85 1.1 tron 86 1.1 tron <p> For example, in the output below we see the top 10 lines of 87 1.1 tron the (mostly forged) sender domain distribution for captured spam 88 1.1 tron in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>: </p> 89 1.1 tron 90 1.1 tron <blockquote> 91 1.1 tron <pre> 92 1.1 tron $ qshape -s hold | head 93 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 94 1.1 tron TOTAL 486 0 0 1 0 0 2 4 20 40 419 95 1.1 tron yahoo.com 14 0 0 1 0 0 0 0 1 0 12 96 1.1 tron extremepricecuts.net 13 0 0 0 0 0 0 0 2 0 11 97 1.1 tron ms35.hinet.net 12 0 0 0 0 0 0 0 0 1 11 98 1.1 tron winnersdaily.net 12 0 0 0 0 0 0 0 2 0 10 99 1.1 tron hotmail.com 11 0 0 0 0 0 0 0 0 1 10 100 1.1 tron worldnet.fr 6 0 0 0 0 0 0 0 0 0 6 101 1.1 tron ms41.hinet.net 6 0 0 0 0 0 0 0 0 0 6 102 1.1 tron osn.de 5 0 0 0 0 0 1 0 0 0 4 103 1.1 tron </pre> 104 1.1 tron </blockquote> 105 1.1 tron 106 1.1 tron <ul> 107 1.1 tron 108 1.1 tron <li> <p> The "T" column shows the total (in this case sender) count 109 1.1 tron for each domain. The columns with numbers above them, show counts 110 1.1 tron for messages aged fewer than that many minutes, but not younger 111 1.1 tron than the age limit for the previous column. The row labeled "TOTAL" 112 1.1 tron shows the total count for all domains. </p> 113 1.1 tron 114 1.1 tron <li> <p> In this example, there are 14 messages allegedly from 115 1.1 tron yahoo.com, 1 between 10 and 20 minutes old, 1 between 320 and 640 116 1.1 tron minutes old and 12 older than 1280 minutes (1440 minutes in a day). 117 1.1 tron </p> 118 1.1 tron 119 1.1 tron </ul> 120 1.1 tron 121 1.1 tron <p> When the output is a terminal intermediate results showing the top 20 122 1.1 tron domains (-n option) are displayed after every 1000 messages (-N option) 123 1.1 tron and the final output also shows only the top 20 domains. This makes 124 1.1.1.4 christos qshape useful even when the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> is very large and it may 125 1.1.1.4 christos otherwise take prohibitively long to read the entire "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. </p> 126 1.1 tron 127 1.1 tron <p> By default, qshape shows statistics for the union of both the 128 1.1.1.4 christos "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queues</a> which are the most relevant queues to 129 1.1 tron look at when analyzing performance. </p> 130 1.1 tron 131 1.1 tron <p> One can request an alternate list of queues: </p> 132 1.1 tron 133 1.1 tron <blockquote> 134 1.1 tron <pre> 135 1.1 tron $ qshape deferred 136 1.1 tron $ qshape incoming active deferred 137 1.1 tron </pre> 138 1.1 tron </blockquote> 139 1.1 tron 140 1.1.1.4 christos <p> this will show the age distribution of the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> or 141 1.1.1.4 christos the union of the "<a href="QSHAPE_README.html#incoming_queue">incoming"</a>, "<a href="QSHAPE_README.html#active_queue">active"</a> and "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a>. </p> 142 1.1 tron 143 1.1 tron <p> Command line options control the number of display "buckets", 144 1.1 tron the age limit for the smallest bucket, display of parent domain 145 1.1 tron counts and so on. The "-h" option outputs a summary of the available 146 1.1 tron switches. </p> 147 1.1 tron 148 1.1 tron <h2><a name="trouble_shooting">Trouble shooting with qshape</a> 149 1.1 tron </h2> 150 1.1 tron 151 1.1 tron <p> Large numbers in the qshape output represent a large number of 152 1.1 tron messages that are destined to (or alleged to come from) a particular 153 1.1 tron domain. It should be possible to tell at a glance which domains 154 1.1 tron dominate the queue sender or recipient counts, approximately when 155 1.1 tron a burst of mail started, and when it stopped. </p> 156 1.1 tron 157 1.1 tron <p> The problem destinations or sender domains appear near the top 158 1.1.1.4 christos left corner of the output table. Remember that the "<a href="QSHAPE_README.html#active_queue">active" queue</a> 159 1.1 tron can accommodate up to 20000 ($<a href="postconf.5.html#qmgr_message_active_limit">qmgr_message_active_limit</a>) messages. 160 1.1 tron To check whether this limit has been reached, use: </p> 161 1.1 tron 162 1.1 tron <blockquote> 163 1.1 tron <pre> 164 1.1 tron $ qshape -s active <i>(show sender statistics)</i> 165 1.1 tron </pre> 166 1.1 tron </blockquote> 167 1.1 tron 168 1.1.1.4 christos <p> If the total sender count is below 20000 the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is 169 1.1 tron not yet saturated, any high volume sender domains show near the 170 1.1 tron top of the output. 171 1.1 tron 172 1.1.1.4 christos <p> With <a href="qmgr.8.html">oqmgr(8)</a> the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is also limited to at most 20000 173 1.1 tron recipient addresses ($<a href="postconf.5.html#qmgr_message_recipient_limit">qmgr_message_recipient_limit</a>). To check for 174 1.1 tron exhaustion of this limit use: </p> 175 1.1 tron 176 1.1 tron <blockquote> 177 1.1 tron <pre> 178 1.1 tron $ qshape active <i>(show recipient statistics)</i> 179 1.1 tron </pre> 180 1.1 tron </blockquote> 181 1.1 tron 182 1.1 tron <p> Having found the high volume domains, it is often useful to 183 1.1 tron search the logs for recent messages pertaining to the domains in 184 1.1 tron question. </p> 185 1.1 tron 186 1.1 tron <blockquote> 187 1.1 tron <pre> 188 1.1 tron # Find deliveries to example.com 189 1.1 tron # 190 1.1 tron $ tail -10000 /var/log/maillog | 191 1.1.1.5 christos grep -E -i ': to=<.*@example\.com>,' | 192 1.1 tron less 193 1.1 tron 194 1.1 tron # Find messages from example.com 195 1.1 tron # 196 1.1 tron $ tail -10000 /var/log/maillog | 197 1.1.1.5 christos grep -E -i ': from=<.*@example\.com>,' | 198 1.1 tron less 199 1.1 tron </pre> 200 1.1 tron </blockquote> 201 1.1 tron 202 1.1 tron <p> You may want to drill in on some specific queue ids: </p> 203 1.1 tron 204 1.1 tron <blockquote> 205 1.1 tron <pre> 206 1.1 tron # Find all messages for a specific queue id. 207 1.1 tron # 208 1.1.1.5 christos $ tail -10000 /var/log/maillog | grep -E ': 2B2173FF68: ' 209 1.1 tron </pre> 210 1.1 tron </blockquote> 211 1.1 tron 212 1.1 tron <p> Also look for queue manager warning messages in the log. These 213 1.1 tron warnings can suggest strategies to reduce congestion. </p> 214 1.1 tron 215 1.1 tron <blockquote> 216 1.1 tron <pre> 217 1.1.1.5 christos $ grep -E 'qmgr.*(panic|fatal|error|warning):' /var/log/maillog 218 1.1 tron </pre> 219 1.1 tron </blockquote> 220 1.1 tron 221 1.1 tron <p> When all else fails try the Postfix mailing list for help, but 222 1.1 tron please don't forget to include the top 10 or 20 lines of <a href="qshape.1.html">qshape(1)</a> 223 1.1 tron output. </p> 224 1.1 tron 225 1.1 tron <h2><a name="healthy">Example 1: Healthy queue</a></h2> 226 1.1 tron 227 1.1.1.4 christos <p> When looking at just the "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queues</a>, under 228 1.1.1.4 christos normal conditions (no congestion) the "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queues</a> 229 1.1 tron are nearly empty. Mail leaves the system almost as quickly as it 230 1.1.1.4 christos comes in or is deferred without congestion in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. 231 1.1 tron </p> 232 1.1 tron 233 1.1 tron <blockquote> 234 1.1 tron <pre> 235 1.1.1.4 christos $ qshape <i>(show "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queue</a> status)</i> 236 1.1 tron 237 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 238 1.1 tron TOTAL 5 0 0 0 1 0 0 0 1 1 2 239 1.1 tron meri.uwasa.fi 5 0 0 0 1 0 0 0 1 1 2 240 1.1 tron </pre> 241 1.1 tron </blockquote> 242 1.1 tron 243 1.1.1.4 christos <p> If one looks at the two queues separately, the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> 244 1.1 tron is empty or perhaps briefly has one or two messages, while the 245 1.1.1.4 christos "<a href="QSHAPE_README.html#active_queue">active" queue</a> holds more messages and for a somewhat longer time: 246 1.1 tron </p> 247 1.1 tron 248 1.1 tron <blockquote> 249 1.1 tron <pre> 250 1.1 tron $ qshape incoming 251 1.1 tron 252 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 253 1.1 tron TOTAL 0 0 0 0 0 0 0 0 0 0 0 254 1.1 tron 255 1.1 tron $ qshape active 256 1.1 tron 257 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 258 1.1 tron TOTAL 5 0 0 0 1 0 0 0 1 1 2 259 1.1 tron meri.uwasa.fi 5 0 0 0 1 0 0 0 1 1 2 260 1.1 tron </pre> 261 1.1 tron </blockquote> 262 1.1 tron 263 1.1 tron <h2><a name="dictionary_bounce">Example 2: Deferred queue full of 264 1.1 tron dictionary attack bounces</a></h2> 265 1.1 tron 266 1.1 tron <p> This is from a server where recipient validation is not yet 267 1.1 tron available for some of the <a href="VIRTUAL_README.html#canonical">hosted domains</a>. Dictionary attacks on 268 1.1 tron the unvalidated domains result in bounce backscatter. The bounces 269 1.1 tron dominate the queue, but with proper tuning they do not saturate the 270 1.1.1.4 christos "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> or "<a href="QSHAPE_README.html#active_queue">active" queues</a>. The high volume of deferred mail is not 271 1.1 tron a direct cause for alarm. </p> 272 1.1 tron 273 1.1 tron <blockquote> 274 1.1 tron <pre> 275 1.1 tron $ qshape deferred | head 276 1.1 tron 277 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 278 1.1 tron TOTAL 2234 4 2 5 9 31 57 108 201 464 1353 279 1.1 tron heyhihellothere.com 207 0 0 1 1 6 6 8 25 68 92 280 1.1 tron pleazerzoneprod.com 105 0 0 0 0 0 0 0 5 44 56 281 1.1 tron groups.msn.com 63 2 1 2 4 4 14 14 14 8 0 282 1.1 tron orion.toppoint.de 49 0 0 0 1 0 2 4 3 16 23 283 1.1 tron kali.com.cn 46 0 0 0 0 1 0 2 6 12 25 284 1.1 tron meri.uwasa.fi 44 0 0 0 0 1 0 2 8 11 22 285 1.1 tron gjr.paknet.com.pk 43 1 0 0 1 1 3 3 6 12 16 286 1.1 tron aristotle.algonet.se 41 0 0 0 0 0 1 2 11 12 15 287 1.1 tron </pre> 288 1.1 tron </blockquote> 289 1.1 tron 290 1.1 tron <p> The domains shown are mostly bulk-mailers and all the volume 291 1.1 tron is the tail end of the time distribution, showing that short term 292 1.1 tron arrival rates are moderate. Larger numbers and lower message ages 293 1.1 tron are more indicative of current trouble. Old mail still going nowhere 294 1.1.1.4 christos is largely harmless so long as the "<a href="QSHAPE_README.html#active_queue">active"</a> and "<a href="QSHAPE_README.html#incoming_queue">incoming" queues</a> are 295 1.1 tron short. We can also see that the groups.msn.com undeliverables are 296 1.1 tron low rate steady stream rather than a concentrated dictionary attack 297 1.1 tron that is now over. </p> 298 1.1 tron 299 1.1 tron <blockquote> 300 1.1 tron <pre> 301 1.1 tron $ qshape -s deferred | head 302 1.1 tron 303 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 304 1.1 tron TOTAL 2193 4 4 5 8 33 56 104 205 465 1309 305 1.1 tron MAILER-DAEMON 1709 4 4 5 8 33 55 101 198 452 849 306 1.1 tron example.com 263 0 0 0 0 0 0 0 0 2 261 307 1.1 tron example.org 209 0 0 0 0 0 1 3 6 11 188 308 1.1 tron example.net 6 0 0 0 0 0 0 0 0 0 6 309 1.1 tron example.edu 3 0 0 0 0 0 0 0 0 0 3 310 1.1 tron example.gov 2 0 0 0 0 0 0 0 1 0 1 311 1.1 tron example.mil 1 0 0 0 0 0 0 0 0 0 1 312 1.1 tron </pre> 313 1.1 tron </blockquote> 314 1.1 tron 315 1.1 tron <p> Looking at the sender distribution, we see that as expected 316 1.1 tron most of the messages are bounces. </p> 317 1.1 tron 318 1.1 tron <h2><a name="active_congestion">Example 3: Congestion in the active 319 1.1 tron queue</a></h2> 320 1.1 tron 321 1.1 tron <p> This example is taken from a Feb 2004 discussion on the Postfix 322 1.1.1.4 christos Users list. Congestion was reported with the 323 1.1.1.4 christos "<a href="QSHAPE_README.html#active_queue">active"</a> and "<a href="QSHAPE_README.html#incoming_queue">incoming" queues</a> 324 1.1.1.4 christos large and not shrinking despite very large delivery agent 325 1.1 tron process limits. The thread is archived at: 326 1.1.1.2 tron <a href="http://groups.google.com/groups?threadm=c0b7js$2r65$1@FreeBSD.csie.NCTU.edu.tw">http://groups.google.com/groups?threadm=c0b7js$2r65$1@FreeBSD.csie.NCTU.edu.tw</a> 327 1.1.1.2 tron and 328 1.1.1.2 tron <a href="http://archives.neohapsis.com/archives/postfix/2004-02/thread.html#1371">http://archives.neohapsis.com/archives/postfix/2004-02/thread.html#1371</a> 329 1.1.1.2 tron </p> 330 1.1 tron 331 1.1 tron <p> Using an older version of <a href="qshape.1.html">qshape(1)</a> it was quickly determined 332 1.1 tron that all the messages were for just a few destinations: </p> 333 1.1 tron 334 1.1 tron <blockquote> 335 1.1 tron <pre> 336 1.1.1.4 christos $ qshape <i>(show "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queue</a> status)</i> 337 1.1 tron 338 1.1 tron T A 5 10 20 40 80 160 320 320+ 339 1.1 tron TOTAL 11775 9996 0 0 1 1 42 94 221 1420 340 1.1 tron user.sourceforge.net 7678 7678 0 0 0 0 0 0 0 0 341 1.1 tron lists.sourceforge.net 2313 2313 0 0 0 0 0 0 0 0 342 1.1 tron gzd.gotdns.com 102 0 0 0 0 0 0 0 2 100 343 1.1 tron </pre> 344 1.1 tron </blockquote> 345 1.1 tron 346 1.1.1.4 christos <p> The "A" column showed the count of messages in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>, 347 1.1.1.4 christos and the numbered columns showed totals for the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. At 348 1.1.1.4 christos 10000 messages (Postfix 1.x "<a href="QSHAPE_README.html#active_queue">active" queue</a> size limit) the "<a href="QSHAPE_README.html#active_queue">active" queue</a> 349 1.1.1.4 christos is full. The "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> was growing rapidly. </p> 350 1.1 tron 351 1.1 tron <p> With the trouble destinations clearly identified, the administrator 352 1.1 tron quickly found and fixed the problem. It is substantially harder to 353 1.1 tron glean the same information from the logs. While a careful reading 354 1.1 tron of <a href="mailq.1.html">mailq(1)</a> output should yield similar results, it is much harder 355 1.1 tron to gauge the magnitude of the problem by looking at the queue 356 1.1 tron one message at a time. </p> 357 1.1 tron 358 1.1 tron <h2><a name="backlog">Example 4: High volume destination backlog</a></h2> 359 1.1 tron 360 1.1 tron <p> When a site you send a lot of email to is down or slow, mail 361 1.1.1.4 christos messages will rapidly build up in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>, or worse, in 362 1.1.1.4 christos the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. The qshape output will show large numbers for 363 1.1 tron the destination domain in all age buckets that overlap the starting 364 1.1 tron time of the problem: </p> 365 1.1 tron 366 1.1 tron <blockquote> 367 1.1 tron <pre> 368 1.1 tron $ qshape deferred | head 369 1.1 tron 370 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 371 1.1 tron TOTAL 5000 200 200 400 800 1600 1000 200 200 200 200 372 1.1 tron highvolume.com 4000 160 160 320 640 1280 1440 0 0 0 0 373 1.1 tron ... 374 1.1 tron </pre> 375 1.1 tron </blockquote> 376 1.1 tron 377 1.1 tron <p> Here the "highvolume.com" destination is continuing to accumulate 378 1.1.1.4 christos deferred mail. The "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#active_queue">active" queues</a> are fine, but the 379 1.1.1.4 christos "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> started growing some time between 1 and 2 hours ago 380 1.1 tron and continues to grow. </p> 381 1.1 tron 382 1.1 tron <p> If the high volume destination is not down, but is instead 383 1.1.1.4 christos slow, one might see similar congestion in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. 384 1.1.1.4 christos "<a href="QSHAPE_README.html#active_queue">Active" queue</a> congestion is a greater cause for alarm; one might need to 385 1.1 tron take measures to ensure that the mail is deferred instead or even 386 1.1 tron add an <a href="access.5.html">access(5)</a> rule asking the sender to try again later. </p> 387 1.1 tron 388 1.1 tron <p> If a high volume destination exhibits frequent bursts of consecutive 389 1.1 tron connections refused by all MX hosts or "421 Server busy errors", it 390 1.1 tron is possible for the queue manager to mark the destination as "dead" 391 1.1 tron despite the transient nature of the errors. The destination will be 392 1.1 tron retried again after the expiration of a $<a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> timer. 393 1.1 tron If the error bursts are frequent enough it may be that only a small 394 1.1 tron quantity of email is delivered before the destination is again marked 395 1.1 tron "dead". In some cases enabling static (not on demand) connection 396 1.1 tron caching by listing the appropriate nexthop domain in a table included in 397 1.1 tron "<a href="postconf.5.html#smtp_connection_cache_destinations">smtp_connection_cache_destinations</a>" may help to reduce the error rate, 398 1.1 tron because most messages will re-use existing connections. </p> 399 1.1 tron 400 1.1 tron <p> The MTA that has been observed most frequently to exhibit such 401 1.1 tron bursts of errors is Microsoft Exchange, which refuses connections 402 1.1 tron under load. Some proxy virus scanners in front of the Exchange 403 1.1 tron server propagate the refused connection to the client as a "421" 404 1.1 tron error. </p> 405 1.1 tron 406 1.1 tron <p> Note that it is now possible to configure Postfix to exhibit similarly 407 1.1 tron erratic behavior by misconfiguring the <a href="anvil.8.html">anvil(8)</a> service. Do not use 408 1.1 tron <a href="anvil.8.html">anvil(8)</a> for steady-state rate limiting, its purpose is (unintentional) 409 1.1 tron DoS prevention and the rate limits set should be very generous! </p> 410 1.1 tron 411 1.1 tron <p> If one finds oneself needing to deliver a high volume of mail to a 412 1.1 tron destination that exhibits frequent brief bursts of errors and connection 413 1.1 tron caching does not solve the problem, there is a subtle workaround. </p> 414 1.1 tron 415 1.1 tron <ul> 416 1.1 tron 417 1.1 tron <li> <p> Postfix version 2.5 and later: </p> 418 1.1 tron 419 1.1 tron <ul> 420 1.1 tron 421 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" transport 422 1.1 tron for the destination in question. In the example below we will call 423 1.1 tron it "fragile". </p> 424 1.1 tron 425 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> configure a reasonable process limit for the 426 1.1 tron cloned smtp transport (a number in the 10-20 range is typical). </p> 427 1.1 tron 428 1.1 tron <li> <p> IMPORTANT!!! In <a href="postconf.5.html">main.cf</a> configure a large per-destination 429 1.1 tron pseudo-cohort failure limit for the cloned smtp transport. </p> 430 1.1 tron 431 1.1 tron <pre> 432 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 433 1.1.1.3 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = <a href="DATABASE_README.html#types">hash</a>:/etc/postfix/transport 434 1.1 tron fragile_destination_concurrency_failed_cohort_limit = 100 435 1.1 tron fragile_destination_concurrency_limit = 20 436 1.1 tron 437 1.1 tron /etc/postfix/transport: 438 1.1 tron example.com fragile: 439 1.1 tron 440 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 441 1.1 tron # service type private unpriv chroot wakeup maxproc command 442 1.1 tron fragile unix - - n - 20 smtp 443 1.1 tron </pre> 444 1.1 tron 445 1.1 tron <p> See also the documentation for 446 1.1 tron <a href="postconf.5.html#default_destination_concurrency_failed_cohort_limit">default_destination_concurrency_failed_cohort_limit</a> and 447 1.1 tron <a href="postconf.5.html#default_destination_concurrency_limit">default_destination_concurrency_limit</a>. </p> 448 1.1 tron 449 1.1 tron </ul> 450 1.1 tron 451 1.1 tron <li> <p> Earlier Postfix versions: </p> 452 1.1 tron 453 1.1 tron <ul> 454 1.1 tron 455 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" 456 1.1 tron transport for the destination in question. In the example below 457 1.1 tron we will call it "fragile". </p> 458 1.1 tron 459 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> configure a reasonable process limit for the 460 1.1 tron transport (a number in the 10-20 range is typical). </p> 461 1.1 tron 462 1.1 tron <li> <p> IMPORTANT!!! In <a href="postconf.5.html">main.cf</a> configure a very large initial 463 1.1 tron and destination concurrency limit for this transport (say 2000). </p> 464 1.1 tron 465 1.1 tron <pre> 466 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 467 1.1.1.3 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = <a href="DATABASE_README.html#types">hash</a>:/etc/postfix/transport 468 1.1 tron <a href="postconf.5.html#initial_destination_concurrency">initial_destination_concurrency</a> = 2000 469 1.1 tron fragile_destination_concurrency_limit = 2000 470 1.1 tron 471 1.1 tron /etc/postfix/transport: 472 1.1 tron example.com fragile: 473 1.1 tron 474 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 475 1.1 tron # service type private unpriv chroot wakeup maxproc command 476 1.1 tron fragile unix - - n - 20 smtp 477 1.1 tron </pre> 478 1.1 tron 479 1.1 tron <p> See also the documentation for <a href="postconf.5.html#default_destination_concurrency_limit">default_destination_concurrency_limit</a>. 480 1.1 tron </p> 481 1.1 tron 482 1.1 tron </ul> 483 1.1 tron 484 1.1 tron </ul> 485 1.1 tron 486 1.1 tron <p> The effect of this configuration is that up to 2000 487 1.1 tron consecutive errors are tolerated without marking the destination 488 1.1 tron dead, while the total concurrency remains reasonable (10-20 489 1.1 tron processes). This trick is only for a very specialized situation: 490 1.1 tron high volume delivery into a channel with multi-error bursts 491 1.1 tron that is capable of high throughput, but is repeatedly throttled by 492 1.1 tron the bursts of errors. </p> 493 1.1 tron 494 1.1 tron <p> When a destination is unable to handle the load even after the 495 1.1 tron Postfix process limit is reduced to 1, a desperate measure is to 496 1.1 tron insert brief delays between delivery attempts. </p> 497 1.1 tron 498 1.1 tron <ul> 499 1.1 tron 500 1.1 tron <li> <p> Postfix version 2.5 and later: </p> 501 1.1 tron 502 1.1 tron <ul> 503 1.1 tron 504 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" transport 505 1.1 tron for the problem destination. In the example below we call it "slow". 506 1.1 tron </p> 507 1.1 tron 508 1.1 tron <li> <p> In <a href="postconf.5.html">main.cf</a> configure a short delay between deliveries to 509 1.1 tron the same destination. </p> 510 1.1 tron 511 1.1 tron <pre> 512 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 513 1.1.1.3 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = <a href="DATABASE_README.html#types">hash</a>:/etc/postfix/transport 514 1.1 tron slow_destination_rate_delay = 1 515 1.1.1.2 tron slow_destination_concurrency_failed_cohort_limit = 100 516 1.1 tron 517 1.1 tron /etc/postfix/transport: 518 1.1 tron example.com slow: 519 1.1 tron 520 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 521 1.1 tron # service type private unpriv chroot wakeup maxproc command 522 1.1 tron slow unix - - n - - smtp 523 1.1 tron </pre> 524 1.1 tron 525 1.1 tron </ul> 526 1.1 tron 527 1.1 tron <p> See also the documentation for <a href="postconf.5.html#default_destination_rate_delay">default_destination_rate_delay</a>. </p> 528 1.1 tron 529 1.1 tron <p> This solution forces the Postfix <a href="smtp.8.html">smtp(8)</a> client to wait for 530 1.1 tron $slow_destination_rate_delay seconds between deliveries to the same 531 1.1 tron destination. </p> 532 1.1 tron 533 1.1.1.2 tron <p> IMPORTANT!! The large slow_destination_concurrency_failed_cohort_limit 534 1.1.1.2 tron value is needed. This prevents Postfix from deferring all mail for 535 1.1.1.2 tron the same destination after only one connection or handshake error 536 1.1.1.2 tron (the reason for this is that non-zero slow_destination_rate_delay 537 1.1.1.2 tron forces a per-destination concurrency of 1). </p> 538 1.1.1.2 tron 539 1.1 tron <li> <p> Earlier Postfix versions: </p> 540 1.1 tron 541 1.1 tron <ul> 542 1.1 tron 543 1.1 tron <li> <p> In the transport map entry for the problem destination, 544 1.1 tron specify a dead host as the primary nexthop. </p> 545 1.1 tron 546 1.1 tron <li> <p> In the <a href="master.5.html">master.cf</a> entry for the transport specify the 547 1.1 tron problem destination as the <a href="postconf.5.html#fallback_relay">fallback_relay</a> and specify a small 548 1.1 tron <a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a> value. </p> 549 1.1 tron 550 1.1 tron <pre> 551 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 552 1.1.1.3 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = <a href="DATABASE_README.html#types">hash</a>:/etc/postfix/transport 553 1.1 tron 554 1.1 tron /etc/postfix/transport: 555 1.1 tron example.com slow:[dead.host] 556 1.1 tron 557 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 558 1.1 tron # service type private unpriv chroot wakeup maxproc command 559 1.1 tron slow unix - - n - 1 smtp 560 1.1 tron -o <a href="postconf.5.html#fallback_relay">fallback_relay</a>=problem.example.com 561 1.1 tron -o <a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a>=1 562 1.1 tron -o <a href="postconf.5.html#smtp_connection_cache_on_demand">smtp_connection_cache_on_demand</a>=no 563 1.1 tron </pre> 564 1.1 tron 565 1.1 tron </ul> 566 1.1 tron 567 1.1 tron <p> This solution forces the Postfix <a href="smtp.8.html">smtp(8)</a> client to wait for 568 1.1 tron $<a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a> seconds between deliveries. The connection 569 1.1 tron caching feature is disabled to prevent the client from skipping 570 1.1 tron over the dead host. </p> 571 1.1 tron 572 1.1 tron </ul> 573 1.1 tron 574 1.1 tron <h2><a name="queues">Postfix queue directories</a></h2> 575 1.1 tron 576 1.1 tron <p> The following sections describe Postfix queues: their purpose, 577 1.1 tron what normal behavior looks like, and how to diagnose abnormal 578 1.1 tron behavior. </p> 579 1.1 tron 580 1.1 tron <h3> <a name="maildrop_queue"> The "maildrop" queue </a> </h3> 581 1.1 tron 582 1.1 tron <p> Messages that have been submitted via the Postfix <a href="sendmail.1.html">sendmail(1)</a> 583 1.1 tron command, but not yet brought into the main Postfix queue by the 584 1.1 tron <a href="pickup.8.html">pickup(8)</a> service, await processing in the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a>. Messages 585 1.1 tron can be added to the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> even when the Postfix system 586 1.1 tron is not running. They will begin to be processed once Postfix is 587 1.1 tron started. </p> 588 1.1 tron 589 1.1 tron <p> The "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> is drained by the single threaded <a href="pickup.8.html">pickup(8)</a> 590 1.1 tron service scanning the queue directory periodically or when notified 591 1.1 tron of new message arrival by the <a href="postdrop.1.html">postdrop(1)</a> program. The <a href="postdrop.1.html">postdrop(1)</a> 592 1.1 tron program is a setgid helper that allows the unprivileged Postfix 593 1.1 tron <a href="sendmail.1.html">sendmail(1)</a> program to inject mail into the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> and 594 1.1 tron to notify the <a href="pickup.8.html">pickup(8)</a> service of its arrival. </p> 595 1.1 tron 596 1.1 tron <p> All mail that enters the main Postfix queue does so via the 597 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> service. The cleanup service is responsible for envelope 598 1.1 tron and header rewriting, header and body regular expression checks, 599 1.1 tron automatic bcc recipient processing, milter content processing, and 600 1.1 tron reliable insertion of the message into the Postfix "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>. </p> 601 1.1 tron 602 1.1 tron <p> In the absence of excessive CPU consumption in <a href="cleanup.8.html">cleanup(8)</a> header 603 1.1 tron or body regular expression checks or other software consuming all 604 1.1 tron available CPU resources, Postfix performance is disk I/O bound. 605 1.1 tron The rate at which the <a href="pickup.8.html">pickup(8)</a> service can inject messages into 606 1.1 tron the queue is largely determined by disk access times, since the 607 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> service must commit the message to stable storage before 608 1.1 tron returning success. The same is true of the <a href="postdrop.1.html">postdrop(1)</a> program 609 1.1 tron writing the message to the "maildrop" directory. </p> 610 1.1 tron 611 1.1 tron <p> As the pickup service is single threaded, it can only deliver 612 1.1 tron one message at a time at a rate that does not exceed the reciprocal 613 1.1 tron disk I/O latency (+ CPU if not negligible) of the cleanup service. 614 1.1 tron </p> 615 1.1 tron 616 1.1 tron <p> Congestion in this queue is indicative of an excessive local message 617 1.1 tron submission rate or perhaps excessive CPU consumption in the <a href="cleanup.8.html">cleanup(8)</a> 618 1.1 tron service due to excessive <a href="postconf.5.html#body_checks">body_checks</a>, or (Postfix ≥ 2.3) high latency 619 1.1 tron milters. </p> 620 1.1 tron 621 1.1.1.4 christos <p> Note, that once the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is full, the cleanup service 622 1.1 tron will attempt to slow down message injection by pausing $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> 623 1.1 tron for each message. In this case "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> congestion may be 624 1.1 tron a consequence of congestion downstream, rather than a problem in 625 1.1 tron its own right. </p> 626 1.1 tron 627 1.1 tron <p> Note, you should not attempt to deliver large volumes of mail via 628 1.1 tron the <a href="pickup.8.html">pickup(8)</a> service. High volume sites should avoid using "simple" 629 1.1 tron content filters that re-inject scanned mail via Postfix <a href="sendmail.1.html">sendmail(1)</a> 630 1.1 tron and <a href="postdrop.1.html">postdrop(1)</a>. </p> 631 1.1 tron 632 1.1 tron <p> A high arrival rate of locally submitted mail may be an indication 633 1.1 tron of an uncaught forwarding loop, or a run-away notification program. 634 1.1 tron Try to keep the volume of local mail injection to a moderate level. 635 1.1 tron </p> 636 1.1 tron 637 1.1 tron <p> The "postsuper -r" command can place selected messages into 638 1.1 tron the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> for reprocessing. This is most useful for 639 1.1 tron resetting any stale <a href="postconf.5.html#content_filter">content_filter</a> settings. Requeuing a large number 640 1.1 tron of messages using "postsuper -r" can clearly cause a spike in the 641 1.1 tron size of the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a>. </p> 642 1.1 tron 643 1.1 tron <h3> <a name="hold_queue"> The "hold" queue </a> </h3> 644 1.1 tron 645 1.1 tron <p> The administrator can define "smtpd" <a href="access.5.html">access(5)</a> policies, or 646 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> header/body checks that cause messages to be automatically 647 1.1 tron diverted from normal processing and placed indefinitely in the 648 1.1.1.4 christos "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>. Messages placed in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> stay there until 649 1.1 tron the administrator intervenes. No periodic delivery attempts are 650 1.1 tron made for messages in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>. The <a href="postsuper.1.html">postsuper(1)</a> command 651 1.1 tron can be used to manually release messages into the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. 652 1.1 tron </p> 653 1.1 tron 654 1.1 tron <p> Messages can potentially stay in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> longer than 655 1.1 tron $<a href="postconf.5.html#maximal_queue_lifetime">maximal_queue_lifetime</a>. If such "old" messages need to be released from 656 1.1.1.4 christos the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>, they should typically be moved into the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> 657 1.1.1.4 christos using "postsuper -r", so that the message gets a new timestamp and 658 1.1 tron is given more than one opportunity to be delivered. Messages that are 659 1.1 tron "young" can be moved directly into the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> using 660 1.1 tron "postsuper -H". </p> 661 1.1 tron 662 1.1 tron <p> The "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> plays little role in Postfix performance, and 663 1.1 tron monitoring of the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> is typically more closely motivated 664 1.1 tron by tracking spam and malware, than by performance issues. </p> 665 1.1 tron 666 1.1 tron <h3> <a name="incoming_queue"> The "incoming" queue </a> </h3> 667 1.1 tron 668 1.1 tron <p> All new mail entering the Postfix queue is written by the 669 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> service into the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>. New queue files are 670 1.1 tron created owned by the "postfix" user with an access bitmask (or 671 1.1 tron mode) of 0600. Once a queue file is ready for further processing 672 1.1 tron the <a href="cleanup.8.html">cleanup(8)</a> service changes the queue file mode to 0700 and 673 1.1 tron notifies the queue manager of new mail arrival. The queue manager 674 1.1 tron ignores incomplete queue files whose mode is 0600, as these are 675 1.1 tron still being written by cleanup. </p> 676 1.1 tron 677 1.1.1.4 christos <p> The queue manager scans the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> bringing any new 678 1.1.1.4 christos mail into the "<a href="QSHAPE_README.html#active_queue">active" queue</a> if the "<a href="QSHAPE_README.html#active_queue">active" queue</a> resource limits 679 1.1.1.4 christos have not been exceeded. By default, the "<a href="QSHAPE_README.html#active_queue">active" queue</a> accommodates 680 1.1.1.4 christos at most 20000 messages. Once the "<a href="QSHAPE_README.html#active_queue">active" queue</a> message limit is 681 1.1.1.4 christos reached, the queue manager stops scanning the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> 682 1.1.1.4 christos (and the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>, see below). </p> 683 1.1 tron 684 1.1.1.4 christos <p> Under normal conditions the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> is nearly empty (has 685 1.1 tron only mode 0600 files), with the queue manager able to import new 686 1.1.1.4 christos messages into the "<a href="QSHAPE_README.html#active_queue">active" queue</a> as soon as they become available. 687 1.1 tron </p> 688 1.1 tron 689 1.1.1.4 christos <p> The "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> grows when the message input rate spikes 690 1.1 tron above the rate at which the queue manager can import messages into 691 1.1.1.4 christos the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. The main factors slowing down the queue manager 692 1.1 tron are disk I/O and lookup queries to the trivial-rewrite service. If the queue 693 1.1 tron manager is routinely not keeping up, consider not using "slow" 694 1.1 tron lookup services (MySQL, LDAP, ...) for transport lookups or speeding 695 1.1 tron up the hosts that provide the lookup service. If the problem is I/O 696 1.1 tron starvation, consider striping the queue over more disks, faster controllers 697 1.1 tron with a battery write cache, or other hardware improvements. At the very 698 1.1 tron least, make sure that the queue directory is mounted with the "noatime" 699 1.1 tron option if applicable to the underlying filesystem. </p> 700 1.1 tron 701 1.1 tron <p> The <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> parameter is used to clamp the input rate 702 1.1 tron when the queue manager starts to fall behind. The <a href="cleanup.8.html">cleanup(8)</a> service 703 1.1 tron will pause for $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> seconds before creating a new queue 704 1.1 tron file if it cannot obtain a "token" from the queue manager. </p> 705 1.1 tron 706 1.1 tron <p> Since the number of <a href="cleanup.8.html">cleanup(8)</a> processes is limited in most 707 1.1 tron cases by the SMTP server concurrency, the input rate can exceed 708 1.1 tron the output rate by at most "SMTP connection count" / $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> 709 1.1 tron messages per second. </p> 710 1.1 tron 711 1.1 tron <p> With a default process limit of 100, and an <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> of 712 1.1 tron 1s, the coupling is strong enough to limit a single run-away injector 713 1.1 tron to 1 message per second, but is not strong enough to deflect an 714 1.1 tron excessive input rate from many sources at the same time. </p> 715 1.1 tron 716 1.1 tron <p> If a server is being hammered from multiple directions, consider 717 1.1.1.4 christos raising the <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> to 10 seconds, but only if the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> 718 1.1.1.4 christos is growing even while the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is not full and the 719 1.1 tron trivial-rewrite service is using a fast transport lookup mechanism. 720 1.1 tron </p> 721 1.1 tron 722 1.1 tron <h3> <a name="active_queue"> The "active" queue </a> </h3> 723 1.1 tron 724 1.1 tron <p> The queue manager is a delivery agent scheduler; it works to 725 1.1 tron ensure fast and fair delivery of mail to all destinations within 726 1.1 tron designated resource limits. </p> 727 1.1 tron 728 1.1.1.4 christos <p> The "<a href="QSHAPE_README.html#active_queue">active" queue</a> is somewhat analogous to an operating system's 729 1.1.1.4 christos process run queue. Messages in the "<a href="QSHAPE_README.html#active_queue">active" queue</a> are ready to be 730 1.1 tron sent (runnable), but are not necessarily in the process of being 731 1.1 tron sent (running). </p> 732 1.1 tron 733 1.1 tron <p> While most Postfix administrators think of the "<a href="QSHAPE_README.html#active_queue">active" queue</a> 734 1.1 tron as a directory on disk, the real "<a href="QSHAPE_README.html#active_queue">active" queue</a> is a set of data 735 1.1 tron structures in the memory of the queue manager process. </p> 736 1.1 tron 737 1.1.1.4 christos <p> Messages in the "<a href="QSHAPE_README.html#maildrop_queue">maildrop"</a>, "<a href="QSHAPE_README.html#hold_queue">hold"</a>, "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a> 738 1.1.1.4 christos (see below) do not occupy memory; they are safely stored on 739 1.1 tron disk waiting for their turn to be processed. The envelope information 740 1.1 tron for messages in the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is managed in memory, allowing 741 1.1 tron the queue manager to do global scheduling, allocating available 742 1.1.1.4 christos delivery agent processes to an appropriate message in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. </p> 743 1.1 tron 744 1.1.1.4 christos <p> Within the "<a href="QSHAPE_README.html#active_queue">active" queue</a>, (multi-recipient) messages are broken 745 1.1 tron up into groups of recipients that share the same transport/nexthop 746 1.1 tron combination; the group size is capped by the transport's recipient 747 1.1 tron concurrency limit. </p> 748 1.1 tron 749 1.1 tron <p> Multiple recipient groups (from one or more messages) are queued 750 1.1 tron for delivery grouped by transport/nexthop combination. The 751 1.1 tron <b>destination</b> concurrency limit for the transports caps the number 752 1.1 tron of simultaneous delivery attempts for each nexthop. Transports with 753 1.1 tron a <b>recipient</b> concurrency limit of 1 are special: these are grouped 754 1.1 tron by the actual recipient address rather than the nexthop, yielding 755 1.1 tron per-recipient concurrency limits rather than per-domain 756 1.1 tron concurrency limits. Per-recipient limits are appropriate when 757 1.1 tron performing final delivery to mailboxes rather than when relaying 758 1.1 tron to a remote server. </p> 759 1.1 tron 760 1.1.1.4 christos <p> Congestion occurs in the "<a href="QSHAPE_README.html#active_queue">active" queue</a> when one or more destinations 761 1.1 tron drain slower than the corresponding message input rate. </p> 762 1.1 tron 763 1.1.1.4 christos <p> Input into the "<a href="QSHAPE_README.html#active_queue">active" queue</a> comes both from new mail in the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>, 764 1.1.1.4 christos and retries of mail in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. Should the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 765 1.1.1.4 christos get really large, retries of old mail can dominate the arrival 766 1.1 tron rate of new mail. Systems with more CPU, faster disks and more network 767 1.1.1.4 christos bandwidth can deal with larger "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a>, but as a rule of thumb 768 1.1.1.4 christos the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> scales to somewhere between 100,000 and 1,000,000 769 1.1 tron messages with good performance unlikely above that "limit". Systems with 770 1.1 tron queues this large should typically stop accepting new mail, or put the 771 1.1 tron backlog "on hold" until the underlying issue is fixed (provided that 772 1.1 tron there is enough capacity to handle just the new mail). </p> 773 1.1 tron 774 1.1 tron <p> When a destination is down for some time, the queue manager will 775 1.1 tron mark it dead, and immediately defer all mail for the destination without 776 1.1 tron trying to assign it to a delivery agent. In this case the messages 777 1.1.1.4 christos will quickly leave the "<a href="QSHAPE_README.html#active_queue">active" queue</a> and end up in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 778 1.1 tron (with Postfix < 2.4, this is done directly by the queue manager, 779 1.1 tron with Postfix ≥ 2.4 this is done via the "retry" delivery agent). </p> 780 1.1 tron 781 1.1 tron <p> When the destination is instead simply slow, or there is a problem 782 1.1.1.4 christos causing an excessive arrival rate the "<a href="QSHAPE_README.html#active_queue">active" queue</a> will grow and will 783 1.1 tron become dominated by mail to the congested destination. </p> 784 1.1 tron 785 1.1 tron <p> The only way to reduce congestion is to either reduce the input 786 1.1 tron rate or increase the throughput. Increasing the throughput requires 787 1.1 tron either increasing the concurrency or reducing the latency of 788 1.1 tron deliveries. </p> 789 1.1 tron 790 1.1 tron <p> For high volume sites a key tuning parameter is the number of 791 1.1 tron "smtp" delivery agents allocated to the "smtp" and "relay" transports. 792 1.1 tron High volume sites tend to send to many different destinations, many 793 1.1 tron of which may be down or slow, so a good fraction of the available 794 1.1 tron delivery agents will be blocked waiting for slow sites. Also mail 795 1.1 tron destined across the globe will incur large SMTP command-response 796 1.1 tron latencies, so high message throughput can only be achieved with 797 1.1 tron more concurrent delivery agents. </p> 798 1.1 tron 799 1.1 tron <p> The default "smtp" process limit of 100 is good enough for most 800 1.1 tron sites, and may even need to be lowered for sites with low bandwidth 801 1.1 tron connections (no use increasing concurrency once the network pipe 802 1.1 tron is full). When one finds that the queue is growing on an "idle" 803 1.1 tron system (CPU, disk I/O and network not exhausted) the remaining 804 1.1 tron reason for congestion is insufficient concurrency in the face of 805 1.1 tron a high average latency. If the number of outbound SMTP connections 806 1.1 tron (either ESTABLISHED or SYN_SENT) reaches the process limit, mail 807 1.1 tron is draining slowly and the system and network are not loaded, raise 808 1.1 tron the "smtp" and/or "relay" process limits! </p> 809 1.1 tron 810 1.1 tron <p> When a high volume destination is served by multiple MX hosts with 811 1.1 tron typically low delivery latency, performance can suffer dramatically when 812 1.1 tron one of the MX hosts is unresponsive and SMTP connections to that host 813 1.1 tron timeout. For example, if there are 2 equal weight MX hosts, the SMTP 814 1.1 tron connection timeout is 30 seconds and one of the MX hosts is down, the 815 1.1 tron average SMTP connection will take approximately 15 seconds to complete. 816 1.1 tron With a default per-destination concurrency limit of 20 connections, 817 1.1 tron throughput falls to just over 1 message per second. </p> 818 1.1 tron 819 1.1 tron <p> The best way to avoid bottlenecks when one or more MX hosts is 820 1.1 tron non-responsive is to use connection caching. Connection caching was 821 1.1 tron introduced with Postfix 2.2 and is by default enabled on demand for 822 1.1.1.4 christos destinations with a backlog of mail in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. When connection 823 1.1 tron caching is in effect for a particular destination, established connections 824 1.1 tron are re-used to send additional messages, this reduces the number of 825 1.1 tron connections made per message delivery and maintains good throughput even 826 1.1 tron in the face of partial unavailability of the destination's MX hosts. </p> 827 1.1 tron 828 1.1 tron <p> If connection caching is not available (Postfix < 2.2) or does 829 1.1 tron not provide a sufficient latency reduction, especially for the "relay" 830 1.1 tron transport used to forward mail to "your own" domains, consider setting 831 1.1 tron lower than default SMTP connection timeouts (1-5 seconds) and higher 832 1.1 tron than default destination concurrency limits. This will further reduce 833 1.1 tron latency and provide more concurrency to maintain throughput should 834 1.1 tron latency rise. </p> 835 1.1 tron 836 1.1 tron <p> Setting high concurrency limits to domains that are not your own may 837 1.1 tron be viewed as hostile by the receiving system, and steps may be taken 838 1.1 tron to prevent you from monopolizing the destination system's resources. 839 1.1 tron The defensive measures may substantially reduce your throughput or block 840 1.1 tron access entirely. Do not set aggressive concurrency limits to remote 841 1.1 tron domains without coordinating with the administrators of the target 842 1.1 tron domain. </p> 843 1.1 tron 844 1.1 tron <p> If necessary, dedicate and tune custom transports for selected high 845 1.1 tron volume destinations. The "relay" transport is provided for forwarding mail 846 1.1 tron to domains for which your server is a primary or backup MX host. These can 847 1.1 tron make up a substantial fraction of your email traffic. Use the "relay" and 848 1.1 tron not the "smtp" transport to send email to these domains. Using the "relay" 849 1.1 tron transport allocates a separate delivery agent pool to these destinations 850 1.1 tron and allows separate tuning of timeouts and concurrency limits. </p> 851 1.1 tron 852 1.1 tron <p> Another common cause of congestion is unwarranted flushing of the 853 1.1.1.4 christos entire "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. The "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> holds messages that are likely 854 1.1 tron to fail to be delivered and are also likely to be slow to fail delivery 855 1.1.1.4 christos (time out). As a result the most common reaction to a large "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 856 1.1 tron (flush it!) is more than likely counter-productive, and typically makes 857 1.1.1.4 christos the congestion worse. Do not flush the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> unless you expect 858 1.1 tron that most of its content has recently become deliverable (e.g. <a href="postconf.5.html#relayhost">relayhost</a> 859 1.1 tron back up after an outage)! </p> 860 1.1 tron 861 1.1 tron <p> Note that whenever the queue manager is restarted, there may 862 1.1.1.4 christos already be messages in the "<a href="QSHAPE_README.html#active_queue">active" queue</a> directory, but the "real" 863 1.1.1.4 christos "<a href="QSHAPE_README.html#active_queue">active" queue</a> in memory is empty. In order to recover the in-memory 864 1.1.1.4 christos state, the queue manager moves all the "<a href="QSHAPE_README.html#active_queue">active" queue</a> messages 865 1.1.1.4 christos back into the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>, and then uses its normal "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> 866 1.1.1.4 christos scan to refill the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. The process of moving all 867 1.1 tron the messages back and forth, redoing transport table (<a href="trivial-rewrite.8.html">trivial-rewrite(8)</a> 868 1.1 tron resolve service) lookups, and re-importing the messages back into 869 1.1 tron memory is expensive. At all costs, avoid frequent restarts of the 870 1.1 tron queue manager (e.g. via frequent execution of "postfix reload"). </p> 871 1.1 tron 872 1.1 tron <h3> <a name="deferred_queue"> The "deferred" queue </a> </h3> 873 1.1 tron 874 1.1 tron <p> When all the deliverable recipients for a message are delivered, 875 1.1 tron and for some recipients delivery failed for a transient reason (it 876 1.1.1.4 christos might succeed later), the message is placed in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. 877 1.1 tron </p> 878 1.1 tron 879 1.1.1.4 christos <p> The queue manager scans the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> periodically. The scan 880 1.1.1.4 christos interval is controlled by the <a href="postconf.5.html#queue_run_delay">queue_run_delay</a> parameter. While a "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 881 1.1.1.4 christos scan is in progress, if an "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> scan is also in progress 882 1.1.1.4 christos (ideally these are brief since the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> should be short), the 883 1.1.1.4 christos queue manager alternates between looking for messages in the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a> 884 1.1.1.4 christos and in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. This "round-robin" strategy prevents 885 1.1.1.4 christos starvation of either the "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> or the "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a>. </p> 886 1.1.1.4 christos 887 1.1.1.4 christos <p> Each "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> scan only brings a fraction of the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 888 1.1.1.4 christos back into the "<a href="QSHAPE_README.html#active_queue">active" queue</a> for a retry. This is because each 889 1.1.1.4 christos message in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> is assigned a "cool-off" time when 890 1.1 tron it is deferred. This is done by time-warping the modification 891 1.1 tron time of the queue file into the future. The queue file is not 892 1.1 tron eligible for a retry if its modification time is not yet reached. 893 1.1 tron </p> 894 1.1 tron 895 1.1 tron <p> The "cool-off" time is at least $<a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> and at 896 1.1 tron most $<a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a>. The next retry time is set by doubling 897 1.1 tron the message's age in the queue, and adjusting up or down to lie 898 1.1 tron within the limits. This means that young messages are initially 899 1.1 tron retried more often than old messages. </p> 900 1.1 tron 901 1.1.1.4 christos <p> If a high volume site routinely has large "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a>, it 902 1.1 tron may be useful to adjust the <a href="postconf.5.html#queue_run_delay">queue_run_delay</a>, <a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> and 903 1.1 tron <a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a> to provide short enough delays on first failure 904 1.1 tron (Postfix ≥ 2.4 has a sensibly low minimal backoff time by default), 905 1.1 tron with perhaps longer delays after multiple failures, to reduce the 906 1.1 tron retransmission rate of old messages and thereby reduce the quantity 907 1.1.1.4 christos of previously deferred mail in the "<a href="QSHAPE_README.html#active_queue">active" queue</a>. If you want a really 908 1.1 tron low <a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a>, you may also want to lower <a href="postconf.5.html#queue_run_delay">queue_run_delay</a>, 909 1.1 tron but understand that more frequent scans will increase the demand for 910 1.1 tron disk I/O. </p> 911 1.1 tron 912 1.1.1.4 christos <p> One common cause of large "<a href="QSHAPE_README.html#deferred_queue">deferred" queues</a> is failure to validate 913 1.1 tron recipients at the SMTP input stage. Since spammers routinely launch 914 1.1 tron dictionary attacks from unrepliable sender addresses, the bounces 915 1.1.1.4 christos for invalid recipient addresses clog the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> (and at high 916 1.1.1.4 christos volumes proportionally clog the "<a href="QSHAPE_README.html#active_queue">active" queue</a>). Recipient validation 917 1.1 tron is strongly recommended through use of the <a href="postconf.5.html#local_recipient_maps">local_recipient_maps</a> and 918 1.1 tron <a href="postconf.5.html#relay_recipient_maps">relay_recipient_maps</a> parameters. Even when bounces drain quickly they 919 1.1 tron inundate innocent victims of forgery with unwanted email. To avoid 920 1.1 tron this, do not accept mail for invalid recipients. </p> 921 1.1 tron 922 1.1 tron <p> When a host with lots of deferred mail is down for some time, 923 1.1.1.4 christos it is possible for the entire "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> to reach its retry 924 1.1.1.4 christos time simultaneously. This can lead to a very full "<a href="QSHAPE_README.html#active_queue">active" queue</a> once 925 1.1 tron the host comes back up. The phenomenon can repeat approximately 926 1.1 tron every <a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a> seconds if the messages are again deferred 927 1.1 tron after a brief burst of congestion. Perhaps, a future Postfix release 928 1.1 tron will add a random offset to the retry time (or use a combination 929 1.1.1.4 christos of strategies) to reduce the odds of repeated complete "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> 930 1.1.1.4 christos flushes. </p> 931 1.1 tron 932 1.1 tron <h2><a name="credits">Credits</a></h2> 933 1.1 tron 934 1.1 tron <p> The <a href="qshape.1.html">qshape(1)</a> program was developed by Victor Duchovni of Morgan 935 1.1 tron Stanley, who also wrote the initial version of this document. </p> 936 1.1 tron 937 1.1 tron </body> 938 1.1 tron 939 1.1 tron </html> 940
Indexes created Sun Jun 07 00:24:08 UTC 2026