Home | History | Annotate | Line # | Download | only in Notes 
      1 **************************************************************************
      2 * The following are notes regarding the overheads of running DTrace.
      3 *
      4 * $Id: ALLoverhead.txt,v 1.1.1.1 2015/09/30 22:01:09 christos Exp $
      5 *
      6 * COPYRIGHT: Copyright (c) 2007 Brendan Gregg.
      7 **************************************************************************
      8 
      9 
     10 The following are notes regarding the overheads of running DTrace.
     11 
     12 * What are the overheads of running DTrace?
     13 
     14 Often negligible.
     15 
     16 It depends what the DTrace script does, in particular, the frequency of
     17 events that it is tracing.
     18 
     19 The following tips should explain what the overheads probably are,
     20 
     21 - if your script traces less than 1000 events per second, then the overhead
     22   is probably negligible. ie, less than 0.1% CPU.
     23 - if your script traces more than 100,000 events per second, then the
     24   overhead will start to be significant. If you are tracing kernel events,
     25   then perhaps this could be 10% per CPU. If you are tracing user land
     26   application events, then the overhead can be greater than 30% per CPU.
     27 - if your script produes pages of output, then the CPU cost of drawing
     28   this output to the screen and rendering the fonts is usually far greater
     29   than DTrace itself. Redirect the output of DTrace to a file in /tmp
     30   ("-o" or ">").
     31 - a ballpark figure for the overhead of a DTrace probe would be 500 ns.
     32   This can be much less (kernel only), or much more (many user to kerel
     33   copyin()s); I've provided it to give you a very rough idea. Of course,
     34   as CPUs become faster, this overhead will become smaller.
     35 
     36 If overheads are a concern - then perform tests to measure their magnitude
     37 for both your workload and the scripts applied, such as benchmarks with
     38 and without DTrace running. Also read the scripts you are using, and
     39 consider how frequent the probes will fire, and if you can customise the
     40 script to reduce the frequency of probes.
     41 
     42 For example, scripts that trace,
     43 
     44 	pid$target:::entry,
     45 	pid$target:::return
     46 
     47 would usually cause significant performance overhead, since they fire two
     48 probes for every function called (and can easily reach 100,000 per second).
     49 You could reduce this by modifying the script to only trace the libraries
     50 you are interested in. For example, if you were only interested in
     51 libsocket and libnsl, then change the above lines wherever they appeared to,
     52 
     53 	pid$target:libsocket::entry,
     54 	pid$target:libsocket::return,
     55 	pid$target:libnsl::entry,
     56 	pid$target:libnsl::return
     57 
     58 and you may notice the overheads are significantly reduced (especially anytime
     59 you drop libc and libdl). To go further, only list functions of interest,
     60 
     61 	pid$target:libsocket:connect:entry,
     62 	pid$target:libsocket:connect:return,
     63 	pid$target:libsocket:listen:entry,
     64 	pid$target:libsocket:listen:return,
     65 	[...]
     66 
     67 There are additional notes in Docs/Faq about the DTraceToolkit's scripts
     68 and performance overhead.
     69 
     70 
     71 * When are the overheads a problem?
     72 
     73 When they are significant (due to frequent events), and you are tracing
     74 in a production environment that is sensitive to additional CPU load.
     75 
     76 Overheads should be considered if you are measuring times (delta, elapsed,
     77 on-CPU, etc) for performance analysis. In practise, overheads aren't
     78 that much of a problem -- the script will either identify your issues
     79 correctly (great), or not (keep looking). Any it is usually easy to quickly
     80 confirm what DTrace does find by using other tools, or by hacking quick
     81 code changes. You might be using DTrace output that you know has a
     82 significant margin of error - but that becomes moot after you prove that
     83 the performance fix works through benchmarking a quick fix.
     84 
     85 At the end of the day, if DTrace helps find real measurable performance wins
     86 (and it should), then it has been successful.
     87 
     88 
     89 * When are overheads not a problem?
     90 
     91 When the script is not tracing extreamly frequent events.
     92 
     93 Also, when you are in development and tracing events for troubleshooting
     94 purposes (args to functions, for example), DTrace overheads are usually 
     95 not an issue at all.
     96 
     97