QSHAPE_README.html revision 1.1.1.2
1 1.1 tron <!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN" 2 1.1 tron "http://www.w3.org/TR/html4/loose.dtd"> 3 1.1 tron 4 1.1 tron <html> 5 1.1 tron 6 1.1 tron <head> 7 1.1 tron 8 1.1 tron <title>Postfix Bottleneck Analysis</title> 9 1.1 tron 10 1.1 tron <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> 11 1.1 tron 12 1.1 tron </head> 13 1.1 tron 14 1.1 tron <body> 15 1.1 tron 16 1.1 tron <h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix Bottleneck Analysis</h1> 17 1.1 tron 18 1.1 tron <hr> 19 1.1 tron 20 1.1 tron <h2>Purpose of this document </h2> 21 1.1 tron 22 1.1 tron <p> This document is an introduction to Postfix queue congestion analysis. 23 1.1 tron It explains how the <a href="qshape.1.html">qshape(1)</a> program can help to track down the 24 1.1 tron reason for queue congestion. <a href="qshape.1.html">qshape(1)</a> is bundled with Postfix 25 1.1 tron 2.1 and later source code, under the "auxiliary" directory. This 26 1.1 tron document describes <a href="qshape.1.html">qshape(1)</a> as bundled with Postfix 2.4. </p> 27 1.1 tron 28 1.1 tron <p> This document covers the following topics: </p> 29 1.1 tron 30 1.1 tron <ul> 31 1.1 tron 32 1.1 tron <li><a href="#qshape">Introducing the qshape tool</a> 33 1.1 tron 34 1.1 tron <li><a href="#trouble_shooting">Trouble shooting with qshape</a> 35 1.1 tron 36 1.1 tron <li><a href="#healthy">Example 1: Healthy queue</a> 37 1.1 tron 38 1.1 tron <li><a href="#dictionary_bounce">Example 2: Deferred queue full of 39 1.1 tron dictionary attack bounces</a></li> 40 1.1 tron 41 1.1 tron <li><a href="#active_congestion">Example 3: Congestion in the active 42 1.1 tron queue</a></li> 43 1.1 tron 44 1.1 tron <li><a href="#backlog">Example 4: High volume destination backlog</a> 45 1.1 tron 46 1.1 tron <li><a href="#queues">Postfix queue directories</a> 47 1.1 tron 48 1.1 tron <ul> 49 1.1 tron 50 1.1 tron <li> <a href="#maildrop_queue"> The "maildrop" queue </a> 51 1.1 tron 52 1.1 tron <li> <a href="#hold_queue"> The "hold" queue </a> 53 1.1 tron 54 1.1 tron <li> <a href="#incoming_queue"> The "incoming" queue </a> 55 1.1 tron 56 1.1 tron <li> <a href="#active_queue"> The "active" queue </a> 57 1.1 tron 58 1.1 tron <li> <a href="#deferred_queue"> The "deferred" queue </a> 59 1.1 tron 60 1.1 tron </ul> 61 1.1 tron 62 1.1 tron <li><a href="#credits">Credits</a> 63 1.1 tron 64 1.1 tron </ul> 65 1.1 tron 66 1.1 tron <h2><a name="qshape">Introducing the qshape tool</a></h2> 67 1.1 tron 68 1.1 tron <p> When mail is draining slowly or the queue is unexpectedly large, 69 1.1 tron run <a href="qshape.1.html">qshape(1)</a> as the super-user (root) to help zero in on the problem. 70 1.1 tron The <a href="qshape.1.html">qshape(1)</a> program displays a tabular view of the Postfix queue 71 1.1 tron contents. </p> 72 1.1 tron 73 1.1 tron <ul> 74 1.1 tron 75 1.1 tron <li> <p> On the horizontal axis, it displays the queue age with 76 1.1 tron fine granularity for recent messages and (geometrically) less fine 77 1.1 tron granularity for older messages. </p> 78 1.1 tron 79 1.1 tron <li> <p> The vertical axis displays the destination (or with the 80 1.1 tron "-s" switch the sender) domain. Domains with the most messages are 81 1.1 tron listed first. </p> 82 1.1 tron 83 1.1 tron </ul> 84 1.1 tron 85 1.1 tron <p> For example, in the output below we see the top 10 lines of 86 1.1 tron the (mostly forged) sender domain distribution for captured spam 87 1.1 tron in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>: </p> 88 1.1 tron 89 1.1 tron <blockquote> 90 1.1 tron <pre> 91 1.1 tron $ qshape -s hold | head 92 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 93 1.1 tron TOTAL 486 0 0 1 0 0 2 4 20 40 419 94 1.1 tron yahoo.com 14 0 0 1 0 0 0 0 1 0 12 95 1.1 tron extremepricecuts.net 13 0 0 0 0 0 0 0 2 0 11 96 1.1 tron ms35.hinet.net 12 0 0 0 0 0 0 0 0 1 11 97 1.1 tron winnersdaily.net 12 0 0 0 0 0 0 0 2 0 10 98 1.1 tron hotmail.com 11 0 0 0 0 0 0 0 0 1 10 99 1.1 tron worldnet.fr 6 0 0 0 0 0 0 0 0 0 6 100 1.1 tron ms41.hinet.net 6 0 0 0 0 0 0 0 0 0 6 101 1.1 tron osn.de 5 0 0 0 0 0 1 0 0 0 4 102 1.1 tron </pre> 103 1.1 tron </blockquote> 104 1.1 tron 105 1.1 tron <ul> 106 1.1 tron 107 1.1 tron <li> <p> The "T" column shows the total (in this case sender) count 108 1.1 tron for each domain. The columns with numbers above them, show counts 109 1.1 tron for messages aged fewer than that many minutes, but not younger 110 1.1 tron than the age limit for the previous column. The row labeled "TOTAL" 111 1.1 tron shows the total count for all domains. </p> 112 1.1 tron 113 1.1 tron <li> <p> In this example, there are 14 messages allegedly from 114 1.1 tron yahoo.com, 1 between 10 and 20 minutes old, 1 between 320 and 640 115 1.1 tron minutes old and 12 older than 1280 minutes (1440 minutes in a day). 116 1.1 tron </p> 117 1.1 tron 118 1.1 tron </ul> 119 1.1 tron 120 1.1 tron <p> When the output is a terminal intermediate results showing the top 20 121 1.1 tron domains (-n option) are displayed after every 1000 messages (-N option) 122 1.1 tron and the final output also shows only the top 20 domains. This makes 123 1.1 tron qshape useful even when the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> is very large and it may 124 1.1 tron otherwise take prohibitively long to read the entire <a href="QSHAPE_README.html#deferred_queue">deferred queue</a>. </p> 125 1.1 tron 126 1.1 tron <p> By default, qshape shows statistics for the union of both the 127 1.1 tron <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queues</a> which are the most relevant queues to 128 1.1 tron look at when analyzing performance. </p> 129 1.1 tron 130 1.1 tron <p> One can request an alternate list of queues: </p> 131 1.1 tron 132 1.1 tron <blockquote> 133 1.1 tron <pre> 134 1.1 tron $ qshape deferred 135 1.1 tron $ qshape incoming active deferred 136 1.1 tron </pre> 137 1.1 tron </blockquote> 138 1.1 tron 139 1.1 tron <p> this will show the age distribution of the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> or 140 1.1 tron the union of the incoming active and <a href="QSHAPE_README.html#deferred_queue">deferred queues</a>. </p> 141 1.1 tron 142 1.1 tron <p> Command line options control the number of display "buckets", 143 1.1 tron the age limit for the smallest bucket, display of parent domain 144 1.1 tron counts and so on. The "-h" option outputs a summary of the available 145 1.1 tron switches. </p> 146 1.1 tron 147 1.1 tron <h2><a name="trouble_shooting">Trouble shooting with qshape</a> 148 1.1 tron </h2> 149 1.1 tron 150 1.1 tron <p> Large numbers in the qshape output represent a large number of 151 1.1 tron messages that are destined to (or alleged to come from) a particular 152 1.1 tron domain. It should be possible to tell at a glance which domains 153 1.1 tron dominate the queue sender or recipient counts, approximately when 154 1.1 tron a burst of mail started, and when it stopped. </p> 155 1.1 tron 156 1.1 tron <p> The problem destinations or sender domains appear near the top 157 1.1 tron left corner of the output table. Remember that the <a href="QSHAPE_README.html#active_queue">active queue</a> 158 1.1 tron can accommodate up to 20000 ($<a href="postconf.5.html#qmgr_message_active_limit">qmgr_message_active_limit</a>) messages. 159 1.1 tron To check whether this limit has been reached, use: </p> 160 1.1 tron 161 1.1 tron <blockquote> 162 1.1 tron <pre> 163 1.1 tron $ qshape -s active <i>(show sender statistics)</i> 164 1.1 tron </pre> 165 1.1 tron </blockquote> 166 1.1 tron 167 1.1 tron <p> If the total sender count is below 20000 the <a href="QSHAPE_README.html#active_queue">active queue</a> is 168 1.1 tron not yet saturated, any high volume sender domains show near the 169 1.1 tron top of the output. 170 1.1 tron 171 1.1 tron <p> With <a href="qmgr.8.html">oqmgr(8)</a> the <a href="QSHAPE_README.html#active_queue">active queue</a> is also limited to at most 20000 172 1.1 tron recipient addresses ($<a href="postconf.5.html#qmgr_message_recipient_limit">qmgr_message_recipient_limit</a>). To check for 173 1.1 tron exhaustion of this limit use: </p> 174 1.1 tron 175 1.1 tron <blockquote> 176 1.1 tron <pre> 177 1.1 tron $ qshape active <i>(show recipient statistics)</i> 178 1.1 tron </pre> 179 1.1 tron </blockquote> 180 1.1 tron 181 1.1 tron <p> Having found the high volume domains, it is often useful to 182 1.1 tron search the logs for recent messages pertaining to the domains in 183 1.1 tron question. </p> 184 1.1 tron 185 1.1 tron <blockquote> 186 1.1 tron <pre> 187 1.1 tron # Find deliveries to example.com 188 1.1 tron # 189 1.1 tron $ tail -10000 /var/log/maillog | 190 1.1 tron egrep -i ': to=<.*@example\.com>,' | 191 1.1 tron less 192 1.1 tron 193 1.1 tron # Find messages from example.com 194 1.1 tron # 195 1.1 tron $ tail -10000 /var/log/maillog | 196 1.1 tron egrep -i ': from=<.*@example\.com>,' | 197 1.1 tron less 198 1.1 tron </pre> 199 1.1 tron </blockquote> 200 1.1 tron 201 1.1 tron <p> You may want to drill in on some specific queue ids: </p> 202 1.1 tron 203 1.1 tron <blockquote> 204 1.1 tron <pre> 205 1.1 tron # Find all messages for a specific queue id. 206 1.1 tron # 207 1.1 tron $ tail -10000 /var/log/maillog | egrep ': 2B2173FF68: ' 208 1.1 tron </pre> 209 1.1 tron </blockquote> 210 1.1 tron 211 1.1 tron <p> Also look for queue manager warning messages in the log. These 212 1.1 tron warnings can suggest strategies to reduce congestion. </p> 213 1.1 tron 214 1.1 tron <blockquote> 215 1.1 tron <pre> 216 1.1 tron $ egrep 'qmgr.*(panic|fatal|error|warning):' /var/log/maillog 217 1.1 tron </pre> 218 1.1 tron </blockquote> 219 1.1 tron 220 1.1 tron <p> When all else fails try the Postfix mailing list for help, but 221 1.1 tron please don't forget to include the top 10 or 20 lines of <a href="qshape.1.html">qshape(1)</a> 222 1.1 tron output. </p> 223 1.1 tron 224 1.1 tron <h2><a name="healthy">Example 1: Healthy queue</a></h2> 225 1.1 tron 226 1.1 tron <p> When looking at just the <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queues</a>, under 227 1.1 tron normal conditions (no congestion) the <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queues</a> 228 1.1 tron are nearly empty. Mail leaves the system almost as quickly as it 229 1.1 tron comes in or is deferred without congestion in the <a href="QSHAPE_README.html#active_queue">active queue</a>. 230 1.1 tron </p> 231 1.1 tron 232 1.1 tron <blockquote> 233 1.1 tron <pre> 234 1.1 tron $ qshape <i>(show <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queue</a> status)</i> 235 1.1 tron 236 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 237 1.1 tron TOTAL 5 0 0 0 1 0 0 0 1 1 2 238 1.1 tron meri.uwasa.fi 5 0 0 0 1 0 0 0 1 1 2 239 1.1 tron </pre> 240 1.1 tron </blockquote> 241 1.1 tron 242 1.1 tron <p> If one looks at the two queues separately, the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> 243 1.1 tron is empty or perhaps briefly has one or two messages, while the 244 1.1 tron <a href="QSHAPE_README.html#active_queue">active queue</a> holds more messages and for a somewhat longer time: 245 1.1 tron </p> 246 1.1 tron 247 1.1 tron <blockquote> 248 1.1 tron <pre> 249 1.1 tron $ qshape incoming 250 1.1 tron 251 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 252 1.1 tron TOTAL 0 0 0 0 0 0 0 0 0 0 0 253 1.1 tron 254 1.1 tron $ qshape active 255 1.1 tron 256 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 257 1.1 tron TOTAL 5 0 0 0 1 0 0 0 1 1 2 258 1.1 tron meri.uwasa.fi 5 0 0 0 1 0 0 0 1 1 2 259 1.1 tron </pre> 260 1.1 tron </blockquote> 261 1.1 tron 262 1.1 tron <h2><a name="dictionary_bounce">Example 2: Deferred queue full of 263 1.1 tron dictionary attack bounces</a></h2> 264 1.1 tron 265 1.1 tron <p> This is from a server where recipient validation is not yet 266 1.1 tron available for some of the <a href="VIRTUAL_README.html#canonical">hosted domains</a>. Dictionary attacks on 267 1.1 tron the unvalidated domains result in bounce backscatter. The bounces 268 1.1 tron dominate the queue, but with proper tuning they do not saturate the 269 1.1 tron <a href="QSHAPE_README.html#incoming_queue">incoming</a> or <a href="QSHAPE_README.html#active_queue">active queues</a>. The high volume of deferred mail is not 270 1.1 tron a direct cause for alarm. </p> 271 1.1 tron 272 1.1 tron <blockquote> 273 1.1 tron <pre> 274 1.1 tron $ qshape deferred | head 275 1.1 tron 276 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 277 1.1 tron TOTAL 2234 4 2 5 9 31 57 108 201 464 1353 278 1.1 tron heyhihellothere.com 207 0 0 1 1 6 6 8 25 68 92 279 1.1 tron pleazerzoneprod.com 105 0 0 0 0 0 0 0 5 44 56 280 1.1 tron groups.msn.com 63 2 1 2 4 4 14 14 14 8 0 281 1.1 tron orion.toppoint.de 49 0 0 0 1 0 2 4 3 16 23 282 1.1 tron kali.com.cn 46 0 0 0 0 1 0 2 6 12 25 283 1.1 tron meri.uwasa.fi 44 0 0 0 0 1 0 2 8 11 22 284 1.1 tron gjr.paknet.com.pk 43 1 0 0 1 1 3 3 6 12 16 285 1.1 tron aristotle.algonet.se 41 0 0 0 0 0 1 2 11 12 15 286 1.1 tron </pre> 287 1.1 tron </blockquote> 288 1.1 tron 289 1.1 tron <p> The domains shown are mostly bulk-mailers and all the volume 290 1.1 tron is the tail end of the time distribution, showing that short term 291 1.1 tron arrival rates are moderate. Larger numbers and lower message ages 292 1.1 tron are more indicative of current trouble. Old mail still going nowhere 293 1.1 tron is largely harmless so long as the active and <a href="QSHAPE_README.html#incoming_queue">incoming queues</a> are 294 1.1 tron short. We can also see that the groups.msn.com undeliverables are 295 1.1 tron low rate steady stream rather than a concentrated dictionary attack 296 1.1 tron that is now over. </p> 297 1.1 tron 298 1.1 tron <blockquote> 299 1.1 tron <pre> 300 1.1 tron $ qshape -s deferred | head 301 1.1 tron 302 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 303 1.1 tron TOTAL 2193 4 4 5 8 33 56 104 205 465 1309 304 1.1 tron MAILER-DAEMON 1709 4 4 5 8 33 55 101 198 452 849 305 1.1 tron example.com 263 0 0 0 0 0 0 0 0 2 261 306 1.1 tron example.org 209 0 0 0 0 0 1 3 6 11 188 307 1.1 tron example.net 6 0 0 0 0 0 0 0 0 0 6 308 1.1 tron example.edu 3 0 0 0 0 0 0 0 0 0 3 309 1.1 tron example.gov 2 0 0 0 0 0 0 0 1 0 1 310 1.1 tron example.mil 1 0 0 0 0 0 0 0 0 0 1 311 1.1 tron </pre> 312 1.1 tron </blockquote> 313 1.1 tron 314 1.1 tron <p> Looking at the sender distribution, we see that as expected 315 1.1 tron most of the messages are bounces. </p> 316 1.1 tron 317 1.1 tron <h2><a name="active_congestion">Example 3: Congestion in the active 318 1.1 tron queue</a></h2> 319 1.1 tron 320 1.1 tron <p> This example is taken from a Feb 2004 discussion on the Postfix 321 1.1 tron Users list. Congestion was reported with the active and incoming 322 1.1 tron queues large and not shrinking despite very large delivery agent 323 1.1 tron process limits. The thread is archived at: 324 1.1.1.2 tron <a href="http://groups.google.com/groups?threadm=c0b7js$2r65$1@FreeBSD.csie.NCTU.edu.tw">http://groups.google.com/groups?threadm=c0b7js$2r65$1@FreeBSD.csie.NCTU.edu.tw</a> 325 1.1.1.2 tron and 326 1.1.1.2 tron <a href="http://archives.neohapsis.com/archives/postfix/2004-02/thread.html#1371">http://archives.neohapsis.com/archives/postfix/2004-02/thread.html#1371</a> 327 1.1.1.2 tron </p> 328 1.1 tron 329 1.1 tron <p> Using an older version of <a href="qshape.1.html">qshape(1)</a> it was quickly determined 330 1.1 tron that all the messages were for just a few destinations: </p> 331 1.1 tron 332 1.1 tron <blockquote> 333 1.1 tron <pre> 334 1.1 tron $ qshape <i>(show <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queue</a> status)</i> 335 1.1 tron 336 1.1 tron T A 5 10 20 40 80 160 320 320+ 337 1.1 tron TOTAL 11775 9996 0 0 1 1 42 94 221 1420 338 1.1 tron user.sourceforge.net 7678 7678 0 0 0 0 0 0 0 0 339 1.1 tron lists.sourceforge.net 2313 2313 0 0 0 0 0 0 0 0 340 1.1 tron gzd.gotdns.com 102 0 0 0 0 0 0 0 2 100 341 1.1 tron </pre> 342 1.1 tron </blockquote> 343 1.1 tron 344 1.1 tron <p> The "A" column showed the count of messages in the <a href="QSHAPE_README.html#active_queue">active queue</a>, 345 1.1 tron and the numbered columns showed totals for the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a>. At 346 1.1 tron 10000 messages (Postfix 1.x <a href="QSHAPE_README.html#active_queue">active queue</a> size limit) the active 347 1.1 tron queue is full. The incoming was growing rapidly. </p> 348 1.1 tron 349 1.1 tron <p> With the trouble destinations clearly identified, the administrator 350 1.1 tron quickly found and fixed the problem. It is substantially harder to 351 1.1 tron glean the same information from the logs. While a careful reading 352 1.1 tron of <a href="mailq.1.html">mailq(1)</a> output should yield similar results, it is much harder 353 1.1 tron to gauge the magnitude of the problem by looking at the queue 354 1.1 tron one message at a time. </p> 355 1.1 tron 356 1.1 tron <h2><a name="backlog">Example 4: High volume destination backlog</a></h2> 357 1.1 tron 358 1.1 tron <p> When a site you send a lot of email to is down or slow, mail 359 1.1 tron messages will rapidly build up in the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a>, or worse, in 360 1.1 tron the <a href="QSHAPE_README.html#active_queue">active queue</a>. The qshape output will show large numbers for 361 1.1 tron the destination domain in all age buckets that overlap the starting 362 1.1 tron time of the problem: </p> 363 1.1 tron 364 1.1 tron <blockquote> 365 1.1 tron <pre> 366 1.1 tron $ qshape deferred | head 367 1.1 tron 368 1.1 tron T 5 10 20 40 80 160 320 640 1280 1280+ 369 1.1 tron TOTAL 5000 200 200 400 800 1600 1000 200 200 200 200 370 1.1 tron highvolume.com 4000 160 160 320 640 1280 1440 0 0 0 0 371 1.1 tron ... 372 1.1 tron </pre> 373 1.1 tron </blockquote> 374 1.1 tron 375 1.1 tron <p> Here the "highvolume.com" destination is continuing to accumulate 376 1.1 tron deferred mail. The <a href="QSHAPE_README.html#incoming_queue">incoming</a> and <a href="QSHAPE_README.html#active_queue">active queues</a> are fine, but the 377 1.1 tron <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> started growing some time between 1 and 2 hours ago 378 1.1 tron and continues to grow. </p> 379 1.1 tron 380 1.1 tron <p> If the high volume destination is not down, but is instead 381 1.1 tron slow, one might see similar congestion in the <a href="QSHAPE_README.html#active_queue">active queue</a>. Active 382 1.1 tron queue congestion is a greater cause for alarm; one might need to 383 1.1 tron take measures to ensure that the mail is deferred instead or even 384 1.1 tron add an <a href="access.5.html">access(5)</a> rule asking the sender to try again later. </p> 385 1.1 tron 386 1.1 tron <p> If a high volume destination exhibits frequent bursts of consecutive 387 1.1 tron connections refused by all MX hosts or "421 Server busy errors", it 388 1.1 tron is possible for the queue manager to mark the destination as "dead" 389 1.1 tron despite the transient nature of the errors. The destination will be 390 1.1 tron retried again after the expiration of a $<a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> timer. 391 1.1 tron If the error bursts are frequent enough it may be that only a small 392 1.1 tron quantity of email is delivered before the destination is again marked 393 1.1 tron "dead". In some cases enabling static (not on demand) connection 394 1.1 tron caching by listing the appropriate nexthop domain in a table included in 395 1.1 tron "<a href="postconf.5.html#smtp_connection_cache_destinations">smtp_connection_cache_destinations</a>" may help to reduce the error rate, 396 1.1 tron because most messages will re-use existing connections. </p> 397 1.1 tron 398 1.1 tron <p> The MTA that has been observed most frequently to exhibit such 399 1.1 tron bursts of errors is Microsoft Exchange, which refuses connections 400 1.1 tron under load. Some proxy virus scanners in front of the Exchange 401 1.1 tron server propagate the refused connection to the client as a "421" 402 1.1 tron error. </p> 403 1.1 tron 404 1.1 tron <p> Note that it is now possible to configure Postfix to exhibit similarly 405 1.1 tron erratic behavior by misconfiguring the <a href="anvil.8.html">anvil(8)</a> service. Do not use 406 1.1 tron <a href="anvil.8.html">anvil(8)</a> for steady-state rate limiting, its purpose is (unintentional) 407 1.1 tron DoS prevention and the rate limits set should be very generous! </p> 408 1.1 tron 409 1.1 tron <p> If one finds oneself needing to deliver a high volume of mail to a 410 1.1 tron destination that exhibits frequent brief bursts of errors and connection 411 1.1 tron caching does not solve the problem, there is a subtle workaround. </p> 412 1.1 tron 413 1.1 tron <ul> 414 1.1 tron 415 1.1 tron <li> <p> Postfix version 2.5 and later: </p> 416 1.1 tron 417 1.1 tron <ul> 418 1.1 tron 419 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" transport 420 1.1 tron for the destination in question. In the example below we will call 421 1.1 tron it "fragile". </p> 422 1.1 tron 423 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> configure a reasonable process limit for the 424 1.1 tron cloned smtp transport (a number in the 10-20 range is typical). </p> 425 1.1 tron 426 1.1 tron <li> <p> IMPORTANT!!! In <a href="postconf.5.html">main.cf</a> configure a large per-destination 427 1.1 tron pseudo-cohort failure limit for the cloned smtp transport. </p> 428 1.1 tron 429 1.1 tron <pre> 430 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 431 1.1 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = hash:/etc/postfix/transport 432 1.1 tron fragile_destination_concurrency_failed_cohort_limit = 100 433 1.1 tron fragile_destination_concurrency_limit = 20 434 1.1 tron 435 1.1 tron /etc/postfix/transport: 436 1.1 tron example.com fragile: 437 1.1 tron 438 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 439 1.1 tron # service type private unpriv chroot wakeup maxproc command 440 1.1 tron fragile unix - - n - 20 smtp 441 1.1 tron </pre> 442 1.1 tron 443 1.1 tron <p> See also the documentation for 444 1.1 tron <a href="postconf.5.html#default_destination_concurrency_failed_cohort_limit">default_destination_concurrency_failed_cohort_limit</a> and 445 1.1 tron <a href="postconf.5.html#default_destination_concurrency_limit">default_destination_concurrency_limit</a>. </p> 446 1.1 tron 447 1.1 tron </ul> 448 1.1 tron 449 1.1 tron <li> <p> Earlier Postfix versions: </p> 450 1.1 tron 451 1.1 tron <ul> 452 1.1 tron 453 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" 454 1.1 tron transport for the destination in question. In the example below 455 1.1 tron we will call it "fragile". </p> 456 1.1 tron 457 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> configure a reasonable process limit for the 458 1.1 tron transport (a number in the 10-20 range is typical). </p> 459 1.1 tron 460 1.1 tron <li> <p> IMPORTANT!!! In <a href="postconf.5.html">main.cf</a> configure a very large initial 461 1.1 tron and destination concurrency limit for this transport (say 2000). </p> 462 1.1 tron 463 1.1 tron <pre> 464 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 465 1.1 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = hash:/etc/postfix/transport 466 1.1 tron <a href="postconf.5.html#initial_destination_concurrency">initial_destination_concurrency</a> = 2000 467 1.1 tron fragile_destination_concurrency_limit = 2000 468 1.1 tron 469 1.1 tron /etc/postfix/transport: 470 1.1 tron example.com fragile: 471 1.1 tron 472 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 473 1.1 tron # service type private unpriv chroot wakeup maxproc command 474 1.1 tron fragile unix - - n - 20 smtp 475 1.1 tron </pre> 476 1.1 tron 477 1.1 tron <p> See also the documentation for <a href="postconf.5.html#default_destination_concurrency_limit">default_destination_concurrency_limit</a>. 478 1.1 tron </p> 479 1.1 tron 480 1.1 tron </ul> 481 1.1 tron 482 1.1 tron </ul> 483 1.1 tron 484 1.1 tron <p> The effect of this configuration is that up to 2000 485 1.1 tron consecutive errors are tolerated without marking the destination 486 1.1 tron dead, while the total concurrency remains reasonable (10-20 487 1.1 tron processes). This trick is only for a very specialized situation: 488 1.1 tron high volume delivery into a channel with multi-error bursts 489 1.1 tron that is capable of high throughput, but is repeatedly throttled by 490 1.1 tron the bursts of errors. </p> 491 1.1 tron 492 1.1 tron <p> When a destination is unable to handle the load even after the 493 1.1 tron Postfix process limit is reduced to 1, a desperate measure is to 494 1.1 tron insert brief delays between delivery attempts. </p> 495 1.1 tron 496 1.1 tron <ul> 497 1.1 tron 498 1.1 tron <li> <p> Postfix version 2.5 and later: </p> 499 1.1 tron 500 1.1 tron <ul> 501 1.1 tron 502 1.1 tron <li> <p> In <a href="master.5.html">master.cf</a> set up a dedicated clone of the "smtp" transport 503 1.1 tron for the problem destination. In the example below we call it "slow". 504 1.1 tron </p> 505 1.1 tron 506 1.1 tron <li> <p> In <a href="postconf.5.html">main.cf</a> configure a short delay between deliveries to 507 1.1 tron the same destination. </p> 508 1.1 tron 509 1.1 tron <pre> 510 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 511 1.1 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = hash:/etc/postfix/transport 512 1.1 tron slow_destination_rate_delay = 1 513 1.1.1.2 tron slow_destination_concurrency_failed_cohort_limit = 100 514 1.1 tron 515 1.1 tron /etc/postfix/transport: 516 1.1 tron example.com slow: 517 1.1 tron 518 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 519 1.1 tron # service type private unpriv chroot wakeup maxproc command 520 1.1 tron slow unix - - n - - smtp 521 1.1 tron </pre> 522 1.1 tron 523 1.1 tron </ul> 524 1.1 tron 525 1.1 tron <p> See also the documentation for <a href="postconf.5.html#default_destination_rate_delay">default_destination_rate_delay</a>. </p> 526 1.1 tron 527 1.1 tron <p> This solution forces the Postfix <a href="smtp.8.html">smtp(8)</a> client to wait for 528 1.1 tron $slow_destination_rate_delay seconds between deliveries to the same 529 1.1 tron destination. </p> 530 1.1 tron 531 1.1.1.2 tron <p> IMPORTANT!! The large slow_destination_concurrency_failed_cohort_limit 532 1.1.1.2 tron value is needed. This prevents Postfix from deferring all mail for 533 1.1.1.2 tron the same destination after only one connection or handshake error 534 1.1.1.2 tron (the reason for this is that non-zero slow_destination_rate_delay 535 1.1.1.2 tron forces a per-destination concurrency of 1). </p> 536 1.1.1.2 tron 537 1.1 tron <li> <p> Earlier Postfix versions: </p> 538 1.1 tron 539 1.1 tron <ul> 540 1.1 tron 541 1.1 tron <li> <p> In the transport map entry for the problem destination, 542 1.1 tron specify a dead host as the primary nexthop. </p> 543 1.1 tron 544 1.1 tron <li> <p> In the <a href="master.5.html">master.cf</a> entry for the transport specify the 545 1.1 tron problem destination as the <a href="postconf.5.html#fallback_relay">fallback_relay</a> and specify a small 546 1.1 tron <a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a> value. </p> 547 1.1 tron 548 1.1 tron <pre> 549 1.1 tron /etc/postfix/<a href="postconf.5.html">main.cf</a>: 550 1.1 tron <a href="postconf.5.html#transport_maps">transport_maps</a> = hash:/etc/postfix/transport 551 1.1 tron 552 1.1 tron /etc/postfix/transport: 553 1.1 tron example.com slow:[dead.host] 554 1.1 tron 555 1.1 tron /etc/postfix/<a href="master.5.html">master.cf</a>: 556 1.1 tron # service type private unpriv chroot wakeup maxproc command 557 1.1 tron slow unix - - n - 1 smtp 558 1.1 tron -o <a href="postconf.5.html#fallback_relay">fallback_relay</a>=problem.example.com 559 1.1 tron -o <a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a>=1 560 1.1 tron -o <a href="postconf.5.html#smtp_connection_cache_on_demand">smtp_connection_cache_on_demand</a>=no 561 1.1 tron </pre> 562 1.1 tron 563 1.1 tron </ul> 564 1.1 tron 565 1.1 tron <p> This solution forces the Postfix <a href="smtp.8.html">smtp(8)</a> client to wait for 566 1.1 tron $<a href="postconf.5.html#smtp_connect_timeout">smtp_connect_timeout</a> seconds between deliveries. The connection 567 1.1 tron caching feature is disabled to prevent the client from skipping 568 1.1 tron over the dead host. </p> 569 1.1 tron 570 1.1 tron </ul> 571 1.1 tron 572 1.1 tron <h2><a name="queues">Postfix queue directories</a></h2> 573 1.1 tron 574 1.1 tron <p> The following sections describe Postfix queues: their purpose, 575 1.1 tron what normal behavior looks like, and how to diagnose abnormal 576 1.1 tron behavior. </p> 577 1.1 tron 578 1.1 tron <h3> <a name="maildrop_queue"> The "maildrop" queue </a> </h3> 579 1.1 tron 580 1.1 tron <p> Messages that have been submitted via the Postfix <a href="sendmail.1.html">sendmail(1)</a> 581 1.1 tron command, but not yet brought into the main Postfix queue by the 582 1.1 tron <a href="pickup.8.html">pickup(8)</a> service, await processing in the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a>. Messages 583 1.1 tron can be added to the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> even when the Postfix system 584 1.1 tron is not running. They will begin to be processed once Postfix is 585 1.1 tron started. </p> 586 1.1 tron 587 1.1 tron <p> The "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> is drained by the single threaded <a href="pickup.8.html">pickup(8)</a> 588 1.1 tron service scanning the queue directory periodically or when notified 589 1.1 tron of new message arrival by the <a href="postdrop.1.html">postdrop(1)</a> program. The <a href="postdrop.1.html">postdrop(1)</a> 590 1.1 tron program is a setgid helper that allows the unprivileged Postfix 591 1.1 tron <a href="sendmail.1.html">sendmail(1)</a> program to inject mail into the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> and 592 1.1 tron to notify the <a href="pickup.8.html">pickup(8)</a> service of its arrival. </p> 593 1.1 tron 594 1.1 tron <p> All mail that enters the main Postfix queue does so via the 595 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> service. The cleanup service is responsible for envelope 596 1.1 tron and header rewriting, header and body regular expression checks, 597 1.1 tron automatic bcc recipient processing, milter content processing, and 598 1.1 tron reliable insertion of the message into the Postfix "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>. </p> 599 1.1 tron 600 1.1 tron <p> In the absence of excessive CPU consumption in <a href="cleanup.8.html">cleanup(8)</a> header 601 1.1 tron or body regular expression checks or other software consuming all 602 1.1 tron available CPU resources, Postfix performance is disk I/O bound. 603 1.1 tron The rate at which the <a href="pickup.8.html">pickup(8)</a> service can inject messages into 604 1.1 tron the queue is largely determined by disk access times, since the 605 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> service must commit the message to stable storage before 606 1.1 tron returning success. The same is true of the <a href="postdrop.1.html">postdrop(1)</a> program 607 1.1 tron writing the message to the "maildrop" directory. </p> 608 1.1 tron 609 1.1 tron <p> As the pickup service is single threaded, it can only deliver 610 1.1 tron one message at a time at a rate that does not exceed the reciprocal 611 1.1 tron disk I/O latency (+ CPU if not negligible) of the cleanup service. 612 1.1 tron </p> 613 1.1 tron 614 1.1 tron <p> Congestion in this queue is indicative of an excessive local message 615 1.1 tron submission rate or perhaps excessive CPU consumption in the <a href="cleanup.8.html">cleanup(8)</a> 616 1.1 tron service due to excessive <a href="postconf.5.html#body_checks">body_checks</a>, or (Postfix ≥ 2.3) high latency 617 1.1 tron milters. </p> 618 1.1 tron 619 1.1 tron <p> Note, that once the <a href="QSHAPE_README.html#active_queue">active queue</a> is full, the cleanup service 620 1.1 tron will attempt to slow down message injection by pausing $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> 621 1.1 tron for each message. In this case "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> congestion may be 622 1.1 tron a consequence of congestion downstream, rather than a problem in 623 1.1 tron its own right. </p> 624 1.1 tron 625 1.1 tron <p> Note, you should not attempt to deliver large volumes of mail via 626 1.1 tron the <a href="pickup.8.html">pickup(8)</a> service. High volume sites should avoid using "simple" 627 1.1 tron content filters that re-inject scanned mail via Postfix <a href="sendmail.1.html">sendmail(1)</a> 628 1.1 tron and <a href="postdrop.1.html">postdrop(1)</a>. </p> 629 1.1 tron 630 1.1 tron <p> A high arrival rate of locally submitted mail may be an indication 631 1.1 tron of an uncaught forwarding loop, or a run-away notification program. 632 1.1 tron Try to keep the volume of local mail injection to a moderate level. 633 1.1 tron </p> 634 1.1 tron 635 1.1 tron <p> The "postsuper -r" command can place selected messages into 636 1.1 tron the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a> for reprocessing. This is most useful for 637 1.1 tron resetting any stale <a href="postconf.5.html#content_filter">content_filter</a> settings. Requeuing a large number 638 1.1 tron of messages using "postsuper -r" can clearly cause a spike in the 639 1.1 tron size of the "<a href="QSHAPE_README.html#maildrop_queue">maildrop" queue</a>. </p> 640 1.1 tron 641 1.1 tron <h3> <a name="hold_queue"> The "hold" queue </a> </h3> 642 1.1 tron 643 1.1 tron <p> The administrator can define "smtpd" <a href="access.5.html">access(5)</a> policies, or 644 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> header/body checks that cause messages to be automatically 645 1.1 tron diverted from normal processing and placed indefinitely in the 646 1.1 tron "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>. Messages placed in the "hold" queue stay there until 647 1.1 tron the administrator intervenes. No periodic delivery attempts are 648 1.1 tron made for messages in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>. The <a href="postsuper.1.html">postsuper(1)</a> command 649 1.1 tron can be used to manually release messages into the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. 650 1.1 tron </p> 651 1.1 tron 652 1.1 tron <p> Messages can potentially stay in the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> longer than 653 1.1 tron $<a href="postconf.5.html#maximal_queue_lifetime">maximal_queue_lifetime</a>. If such "old" messages need to be released from 654 1.1 tron the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a>, they should typically be moved into the "maildrop" 655 1.1 tron queue using "postsuper -r", so that the message gets a new timestamp and 656 1.1 tron is given more than one opportunity to be delivered. Messages that are 657 1.1 tron "young" can be moved directly into the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a> using 658 1.1 tron "postsuper -H". </p> 659 1.1 tron 660 1.1 tron <p> The "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> plays little role in Postfix performance, and 661 1.1 tron monitoring of the "<a href="QSHAPE_README.html#hold_queue">hold" queue</a> is typically more closely motivated 662 1.1 tron by tracking spam and malware, than by performance issues. </p> 663 1.1 tron 664 1.1 tron <h3> <a name="incoming_queue"> The "incoming" queue </a> </h3> 665 1.1 tron 666 1.1 tron <p> All new mail entering the Postfix queue is written by the 667 1.1 tron <a href="cleanup.8.html">cleanup(8)</a> service into the "<a href="QSHAPE_README.html#incoming_queue">incoming" queue</a>. New queue files are 668 1.1 tron created owned by the "postfix" user with an access bitmask (or 669 1.1 tron mode) of 0600. Once a queue file is ready for further processing 670 1.1 tron the <a href="cleanup.8.html">cleanup(8)</a> service changes the queue file mode to 0700 and 671 1.1 tron notifies the queue manager of new mail arrival. The queue manager 672 1.1 tron ignores incomplete queue files whose mode is 0600, as these are 673 1.1 tron still being written by cleanup. </p> 674 1.1 tron 675 1.1 tron <p> The queue manager scans the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> bringing any new 676 1.1 tron mail into the "<a href="QSHAPE_README.html#active_queue">active" queue</a> if the active queue resource limits 677 1.1 tron have not been exceeded. By default, the <a href="QSHAPE_README.html#active_queue">active queue</a> accommodates 678 1.1 tron at most 20000 messages. Once the <a href="QSHAPE_README.html#active_queue">active queue</a> message limit is 679 1.1 tron reached, the queue manager stops scanning the incoming (and deferred, 680 1.1 tron see below) queue. </p> 681 1.1 tron 682 1.1 tron <p> Under normal conditions the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> is nearly empty (has 683 1.1 tron only mode 0600 files), with the queue manager able to import new 684 1.1 tron messages into the <a href="QSHAPE_README.html#active_queue">active queue</a> as soon as they become available. 685 1.1 tron </p> 686 1.1 tron 687 1.1 tron <p> The <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> grows when the message input rate spikes 688 1.1 tron above the rate at which the queue manager can import messages into 689 1.1 tron the <a href="QSHAPE_README.html#active_queue">active queue</a>. The main factors slowing down the queue manager 690 1.1 tron are disk I/O and lookup queries to the trivial-rewrite service. If the queue 691 1.1 tron manager is routinely not keeping up, consider not using "slow" 692 1.1 tron lookup services (MySQL, LDAP, ...) for transport lookups or speeding 693 1.1 tron up the hosts that provide the lookup service. If the problem is I/O 694 1.1 tron starvation, consider striping the queue over more disks, faster controllers 695 1.1 tron with a battery write cache, or other hardware improvements. At the very 696 1.1 tron least, make sure that the queue directory is mounted with the "noatime" 697 1.1 tron option if applicable to the underlying filesystem. </p> 698 1.1 tron 699 1.1 tron <p> The <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> parameter is used to clamp the input rate 700 1.1 tron when the queue manager starts to fall behind. The <a href="cleanup.8.html">cleanup(8)</a> service 701 1.1 tron will pause for $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> seconds before creating a new queue 702 1.1 tron file if it cannot obtain a "token" from the queue manager. </p> 703 1.1 tron 704 1.1 tron <p> Since the number of <a href="cleanup.8.html">cleanup(8)</a> processes is limited in most 705 1.1 tron cases by the SMTP server concurrency, the input rate can exceed 706 1.1 tron the output rate by at most "SMTP connection count" / $<a href="postconf.5.html#in_flow_delay">in_flow_delay</a> 707 1.1 tron messages per second. </p> 708 1.1 tron 709 1.1 tron <p> With a default process limit of 100, and an <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> of 710 1.1 tron 1s, the coupling is strong enough to limit a single run-away injector 711 1.1 tron to 1 message per second, but is not strong enough to deflect an 712 1.1 tron excessive input rate from many sources at the same time. </p> 713 1.1 tron 714 1.1 tron <p> If a server is being hammered from multiple directions, consider 715 1.1 tron raising the <a href="postconf.5.html#in_flow_delay">in_flow_delay</a> to 10 seconds, but only if the incoming 716 1.1 tron queue is growing even while the <a href="QSHAPE_README.html#active_queue">active queue</a> is not full and the 717 1.1 tron trivial-rewrite service is using a fast transport lookup mechanism. 718 1.1 tron </p> 719 1.1 tron 720 1.1 tron <h3> <a name="active_queue"> The "active" queue </a> </h3> 721 1.1 tron 722 1.1 tron <p> The queue manager is a delivery agent scheduler; it works to 723 1.1 tron ensure fast and fair delivery of mail to all destinations within 724 1.1 tron designated resource limits. </p> 725 1.1 tron 726 1.1 tron <p> The <a href="QSHAPE_README.html#active_queue">active queue</a> is somewhat analogous to an operating system's 727 1.1 tron process run queue. Messages in the <a href="QSHAPE_README.html#active_queue">active queue</a> are ready to be 728 1.1 tron sent (runnable), but are not necessarily in the process of being 729 1.1 tron sent (running). </p> 730 1.1 tron 731 1.1 tron <p> While most Postfix administrators think of the "<a href="QSHAPE_README.html#active_queue">active" queue</a> 732 1.1 tron as a directory on disk, the real "<a href="QSHAPE_README.html#active_queue">active" queue</a> is a set of data 733 1.1 tron structures in the memory of the queue manager process. </p> 734 1.1 tron 735 1.1 tron <p> Messages in the "<a href="QSHAPE_README.html#maildrop_queue">maildrop"</a>, "<a href="QSHAPE_README.html#hold_queue">hold"</a>, "<a href="QSHAPE_README.html#incoming_queue">incoming"</a> and "deferred" 736 1.1 tron queues (see below) do not occupy memory; they are safely stored on 737 1.1 tron disk waiting for their turn to be processed. The envelope information 738 1.1 tron for messages in the "<a href="QSHAPE_README.html#active_queue">active" queue</a> is managed in memory, allowing 739 1.1 tron the queue manager to do global scheduling, allocating available 740 1.1 tron delivery agent processes to an appropriate message in the active 741 1.1 tron queue. </p> 742 1.1 tron 743 1.1 tron <p> Within the <a href="QSHAPE_README.html#active_queue">active queue</a>, (multi-recipient) messages are broken 744 1.1 tron up into groups of recipients that share the same transport/nexthop 745 1.1 tron combination; the group size is capped by the transport's recipient 746 1.1 tron concurrency limit. </p> 747 1.1 tron 748 1.1 tron <p> Multiple recipient groups (from one or more messages) are queued 749 1.1 tron for delivery grouped by transport/nexthop combination. The 750 1.1 tron <b>destination</b> concurrency limit for the transports caps the number 751 1.1 tron of simultaneous delivery attempts for each nexthop. Transports with 752 1.1 tron a <b>recipient</b> concurrency limit of 1 are special: these are grouped 753 1.1 tron by the actual recipient address rather than the nexthop, yielding 754 1.1 tron per-recipient concurrency limits rather than per-domain 755 1.1 tron concurrency limits. Per-recipient limits are appropriate when 756 1.1 tron performing final delivery to mailboxes rather than when relaying 757 1.1 tron to a remote server. </p> 758 1.1 tron 759 1.1 tron <p> Congestion occurs in the <a href="QSHAPE_README.html#active_queue">active queue</a> when one or more destinations 760 1.1 tron drain slower than the corresponding message input rate. </p> 761 1.1 tron 762 1.1 tron <p> Input into the <a href="QSHAPE_README.html#active_queue">active queue</a> comes both from new mail in the "incoming" 763 1.1 tron queue, and retries of mail in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. Should the "deferred" 764 1.1 tron queue get really large, retries of old mail can dominate the arrival 765 1.1 tron rate of new mail. Systems with more CPU, faster disks and more network 766 1.1 tron bandwidth can deal with larger <a href="QSHAPE_README.html#deferred_queue">deferred queues</a>, but as a rule of thumb 767 1.1 tron the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> scales to somewhere between 100,000 and 1,000,000 768 1.1 tron messages with good performance unlikely above that "limit". Systems with 769 1.1 tron queues this large should typically stop accepting new mail, or put the 770 1.1 tron backlog "on hold" until the underlying issue is fixed (provided that 771 1.1 tron there is enough capacity to handle just the new mail). </p> 772 1.1 tron 773 1.1 tron <p> When a destination is down for some time, the queue manager will 774 1.1 tron mark it dead, and immediately defer all mail for the destination without 775 1.1 tron trying to assign it to a delivery agent. In this case the messages 776 1.1 tron will quickly leave the <a href="QSHAPE_README.html#active_queue">active queue</a> and end up in the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> 777 1.1 tron (with Postfix < 2.4, this is done directly by the queue manager, 778 1.1 tron with Postfix ≥ 2.4 this is done via the "retry" delivery agent). </p> 779 1.1 tron 780 1.1 tron <p> When the destination is instead simply slow, or there is a problem 781 1.1 tron causing an excessive arrival rate the <a href="QSHAPE_README.html#active_queue">active queue</a> will grow and will 782 1.1 tron become dominated by mail to the congested destination. </p> 783 1.1 tron 784 1.1 tron <p> The only way to reduce congestion is to either reduce the input 785 1.1 tron rate or increase the throughput. Increasing the throughput requires 786 1.1 tron either increasing the concurrency or reducing the latency of 787 1.1 tron deliveries. </p> 788 1.1 tron 789 1.1 tron <p> For high volume sites a key tuning parameter is the number of 790 1.1 tron "smtp" delivery agents allocated to the "smtp" and "relay" transports. 791 1.1 tron High volume sites tend to send to many different destinations, many 792 1.1 tron of which may be down or slow, so a good fraction of the available 793 1.1 tron delivery agents will be blocked waiting for slow sites. Also mail 794 1.1 tron destined across the globe will incur large SMTP command-response 795 1.1 tron latencies, so high message throughput can only be achieved with 796 1.1 tron more concurrent delivery agents. </p> 797 1.1 tron 798 1.1 tron <p> The default "smtp" process limit of 100 is good enough for most 799 1.1 tron sites, and may even need to be lowered for sites with low bandwidth 800 1.1 tron connections (no use increasing concurrency once the network pipe 801 1.1 tron is full). When one finds that the queue is growing on an "idle" 802 1.1 tron system (CPU, disk I/O and network not exhausted) the remaining 803 1.1 tron reason for congestion is insufficient concurrency in the face of 804 1.1 tron a high average latency. If the number of outbound SMTP connections 805 1.1 tron (either ESTABLISHED or SYN_SENT) reaches the process limit, mail 806 1.1 tron is draining slowly and the system and network are not loaded, raise 807 1.1 tron the "smtp" and/or "relay" process limits! </p> 808 1.1 tron 809 1.1 tron <p> When a high volume destination is served by multiple MX hosts with 810 1.1 tron typically low delivery latency, performance can suffer dramatically when 811 1.1 tron one of the MX hosts is unresponsive and SMTP connections to that host 812 1.1 tron timeout. For example, if there are 2 equal weight MX hosts, the SMTP 813 1.1 tron connection timeout is 30 seconds and one of the MX hosts is down, the 814 1.1 tron average SMTP connection will take approximately 15 seconds to complete. 815 1.1 tron With a default per-destination concurrency limit of 20 connections, 816 1.1 tron throughput falls to just over 1 message per second. </p> 817 1.1 tron 818 1.1 tron <p> The best way to avoid bottlenecks when one or more MX hosts is 819 1.1 tron non-responsive is to use connection caching. Connection caching was 820 1.1 tron introduced with Postfix 2.2 and is by default enabled on demand for 821 1.1 tron destinations with a backlog of mail in the <a href="QSHAPE_README.html#active_queue">active queue</a>. When connection 822 1.1 tron caching is in effect for a particular destination, established connections 823 1.1 tron are re-used to send additional messages, this reduces the number of 824 1.1 tron connections made per message delivery and maintains good throughput even 825 1.1 tron in the face of partial unavailability of the destination's MX hosts. </p> 826 1.1 tron 827 1.1 tron <p> If connection caching is not available (Postfix < 2.2) or does 828 1.1 tron not provide a sufficient latency reduction, especially for the "relay" 829 1.1 tron transport used to forward mail to "your own" domains, consider setting 830 1.1 tron lower than default SMTP connection timeouts (1-5 seconds) and higher 831 1.1 tron than default destination concurrency limits. This will further reduce 832 1.1 tron latency and provide more concurrency to maintain throughput should 833 1.1 tron latency rise. </p> 834 1.1 tron 835 1.1 tron <p> Setting high concurrency limits to domains that are not your own may 836 1.1 tron be viewed as hostile by the receiving system, and steps may be taken 837 1.1 tron to prevent you from monopolizing the destination system's resources. 838 1.1 tron The defensive measures may substantially reduce your throughput or block 839 1.1 tron access entirely. Do not set aggressive concurrency limits to remote 840 1.1 tron domains without coordinating with the administrators of the target 841 1.1 tron domain. </p> 842 1.1 tron 843 1.1 tron <p> If necessary, dedicate and tune custom transports for selected high 844 1.1 tron volume destinations. The "relay" transport is provided for forwarding mail 845 1.1 tron to domains for which your server is a primary or backup MX host. These can 846 1.1 tron make up a substantial fraction of your email traffic. Use the "relay" and 847 1.1 tron not the "smtp" transport to send email to these domains. Using the "relay" 848 1.1 tron transport allocates a separate delivery agent pool to these destinations 849 1.1 tron and allows separate tuning of timeouts and concurrency limits. </p> 850 1.1 tron 851 1.1 tron <p> Another common cause of congestion is unwarranted flushing of the 852 1.1 tron entire <a href="QSHAPE_README.html#deferred_queue">deferred queue</a>. The deferred queue holds messages that are likely 853 1.1 tron to fail to be delivered and are also likely to be slow to fail delivery 854 1.1 tron (time out). As a result the most common reaction to a large <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> 855 1.1 tron (flush it!) is more than likely counter-productive, and typically makes 856 1.1 tron the congestion worse. Do not flush the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> unless you expect 857 1.1 tron that most of its content has recently become deliverable (e.g. <a href="postconf.5.html#relayhost">relayhost</a> 858 1.1 tron back up after an outage)! </p> 859 1.1 tron 860 1.1 tron <p> Note that whenever the queue manager is restarted, there may 861 1.1 tron already be messages in the <a href="QSHAPE_README.html#active_queue">active queue</a> directory, but the "real" 862 1.1 tron <a href="QSHAPE_README.html#active_queue">active queue</a> in memory is empty. In order to recover the in-memory 863 1.1 tron state, the queue manager moves all the <a href="QSHAPE_README.html#active_queue">active queue</a> messages 864 1.1 tron back into the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a>, and then uses its normal incoming 865 1.1 tron queue scan to refill the <a href="QSHAPE_README.html#active_queue">active queue</a>. The process of moving all 866 1.1 tron the messages back and forth, redoing transport table (<a href="trivial-rewrite.8.html">trivial-rewrite(8)</a> 867 1.1 tron resolve service) lookups, and re-importing the messages back into 868 1.1 tron memory is expensive. At all costs, avoid frequent restarts of the 869 1.1 tron queue manager (e.g. via frequent execution of "postfix reload"). </p> 870 1.1 tron 871 1.1 tron <h3> <a name="deferred_queue"> The "deferred" queue </a> </h3> 872 1.1 tron 873 1.1 tron <p> When all the deliverable recipients for a message are delivered, 874 1.1 tron and for some recipients delivery failed for a transient reason (it 875 1.1 tron might succeed later), the message is placed in the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a>. 876 1.1 tron </p> 877 1.1 tron 878 1.1 tron <p> The queue manager scans the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> periodically. The scan 879 1.1 tron interval is controlled by the <a href="postconf.5.html#queue_run_delay">queue_run_delay</a> parameter. While a deferred 880 1.1 tron queue scan is in progress, if an <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> scan is also in progress 881 1.1 tron (ideally these are brief since the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a> should be short), the 882 1.1 tron queue manager alternates between looking for messages in the "incoming" 883 1.1 tron queue and in the "<a href="QSHAPE_README.html#deferred_queue">deferred" queue</a>. This "round-robin" strategy prevents 884 1.1 tron starvation of either the <a href="QSHAPE_README.html#incoming_queue">incoming</a> or the <a href="QSHAPE_README.html#deferred_queue">deferred queues</a>. </p> 885 1.1 tron 886 1.1 tron <p> Each <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> scan only brings a fraction of the deferred 887 1.1 tron queue back into the <a href="QSHAPE_README.html#active_queue">active queue</a> for a retry. This is because each 888 1.1 tron message in the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> is assigned a "cool-off" time when 889 1.1 tron it is deferred. This is done by time-warping the modification 890 1.1 tron time of the queue file into the future. The queue file is not 891 1.1 tron eligible for a retry if its modification time is not yet reached. 892 1.1 tron </p> 893 1.1 tron 894 1.1 tron <p> The "cool-off" time is at least $<a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> and at 895 1.1 tron most $<a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a>. The next retry time is set by doubling 896 1.1 tron the message's age in the queue, and adjusting up or down to lie 897 1.1 tron within the limits. This means that young messages are initially 898 1.1 tron retried more often than old messages. </p> 899 1.1 tron 900 1.1 tron <p> If a high volume site routinely has large <a href="QSHAPE_README.html#deferred_queue">deferred queues</a>, it 901 1.1 tron may be useful to adjust the <a href="postconf.5.html#queue_run_delay">queue_run_delay</a>, <a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> and 902 1.1 tron <a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a> to provide short enough delays on first failure 903 1.1 tron (Postfix ≥ 2.4 has a sensibly low minimal backoff time by default), 904 1.1 tron with perhaps longer delays after multiple failures, to reduce the 905 1.1 tron retransmission rate of old messages and thereby reduce the quantity 906 1.1 tron of previously deferred mail in the <a href="QSHAPE_README.html#active_queue">active queue</a>. If you want a really 907 1.1 tron low <a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a>, you may also want to lower <a href="postconf.5.html#queue_run_delay">queue_run_delay</a>, 908 1.1 tron but understand that more frequent scans will increase the demand for 909 1.1 tron disk I/O. </p> 910 1.1 tron 911 1.1 tron <p> One common cause of large <a href="QSHAPE_README.html#deferred_queue">deferred queues</a> is failure to validate 912 1.1 tron recipients at the SMTP input stage. Since spammers routinely launch 913 1.1 tron dictionary attacks from unrepliable sender addresses, the bounces 914 1.1 tron for invalid recipient addresses clog the <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> (and at high 915 1.1 tron volumes proportionally clog the <a href="QSHAPE_README.html#active_queue">active queue</a>). Recipient validation 916 1.1 tron is strongly recommended through use of the <a href="postconf.5.html#local_recipient_maps">local_recipient_maps</a> and 917 1.1 tron <a href="postconf.5.html#relay_recipient_maps">relay_recipient_maps</a> parameters. Even when bounces drain quickly they 918 1.1 tron inundate innocent victims of forgery with unwanted email. To avoid 919 1.1 tron this, do not accept mail for invalid recipients. </p> 920 1.1 tron 921 1.1 tron <p> When a host with lots of deferred mail is down for some time, 922 1.1 tron it is possible for the entire <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> to reach its retry 923 1.1 tron time simultaneously. This can lead to a very full <a href="QSHAPE_README.html#active_queue">active queue</a> once 924 1.1 tron the host comes back up. The phenomenon can repeat approximately 925 1.1 tron every <a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a> seconds if the messages are again deferred 926 1.1 tron after a brief burst of congestion. Perhaps, a future Postfix release 927 1.1 tron will add a random offset to the retry time (or use a combination 928 1.1 tron of strategies) to reduce the odds of repeated complete deferred 929 1.1 tron queue flushes. </p> 930 1.1 tron 931 1.1 tron <h2><a name="credits">Credits</a></h2> 932 1.1 tron 933 1.1 tron <p> The <a href="qshape.1.html">qshape(1)</a> program was developed by Victor Duchovni of Morgan 934 1.1 tron Stanley, who also wrote the initial version of this document. </p> 935 1.1 tron 936 1.1 tron </body> 937 1.1 tron 938 1.1 tron </html> 939
Indexes created Mon Jun 08 00:24:58 UTC 2026