Discussion:
Comparing results of a netperf test with Solaris Tools and DTrace "tcpstat.d"
(too old to reply)
Thomas
2011-04-06 14:04:19 UTC
Permalink
Raw Message
Hello,

my first mail to the list :)

I run a set of tests with netperf:

 

for i in 1 2 3 4 ; do  netperf -H netserver -c -C -i 10 -P 0 -- -s 128k
-S 128k & done

!!! WARNING
!!! Desired confidence was not achieved within the specified iterations.
!!! This implies that there was variability in the test environment that
!!! must be investigated before going further.
!!! Confidence intervals: Throughput      : 15.680%
!!!                       Local CPU util  :
10.129%
!!!                       Remote CPU util : 6.731%

128104 128000 128000    10.00       212.92   56.40   
70.73    43.636  54.742
!!! WARNING
!!! Desired confidence was not achieved within the specified iterations.
!!! This implies that there was variability in the test environment that
!!! must be investigated before going further.
!!! Confidence intervals: Throughput      : 17.850%
!!!                       Local CPU util  :
10.128%
!!!                       Remote CPU util : 6.758%

128104 128000 128000    10.00       213.60   56.40   
70.74    43.595  54.697
!!! WARNING
!!! Desired confidence was not achieved within the specified iterations.
!!! This implies that there was variability in the test environment that
!!! must be investigated before going further.
!!! Confidence intervals: Throughput      : 12.001%
!!!                       Local CPU util  :
10.168%
!!!                       Remote CPU util : 6.761%

128104 128000 128000    10.00       220.80   56.39   
70.73    41.987  52.595
!!! WARNING
!!! Desired confidence was not achieved within the specified iterations.
!!! This implies that there was variability in the test environment that
!!! must be investigated before going further.
!!! Confidence intervals: Throughput      : 24.409%
!!!                       Local CPU util  :
10.398%
!!!                       Remote CPU util : 6.766%

128104 128000 128000    10.02       201.56   56.38   
70.73    46.549  58.390

[1]   Done                    netperf -H netserver -c
-C -i 10 -P 0 -- -s 128k -S 128k  (wd: ~)
(wd now: /OPSTEST/tackerma)
[2]   Done                    netperf -H netserver
-c -C -i 10 -P 0 -- -s 128k -S 128k  (wd: ~)
(wd now: /OPSTEST/tackerma)
[3]-  Done                    netperf -H netserver -c
-C -i 10 -P 0 -- -s 128k -S 128k  (wd: ~)
(wd now: /OPSTEST/tackerma)
[4]+  Done                    netperf -H netserver -c
-C -i 10 -P 0 -- -s 128k -S 128k  (wd: ~)

 

So, i assume that  about 850MB were transmitted.
212.92 + 213.60 + 220.80 + 201.56
848.87999999999988

I watched this test with several OS and system tools:

 

nicstat-1.22/nicstat.Solaris.sparc -n 10

    Time      Int   rKB/s   wKB/s   rPk/s   wPk/s   
rAvs    wAvs %Util    Sat
14:04:05     bge0    5.76    0.09   34.89    1.30  
169.2   68.31  0.00   0.00
14:04:15     bge0    6.90    0.09   39.31    1.40  
179.6   67.71  0.01   0.00
14:04:25     bge0 66432.7    0.06 44952.4    1.00  1513.3  
65.80  54.4   0.00
14:04:35     bge0 97336.8    0.07 65858.5    1.10  1513.4  
65.82  79.7   0.00
14:04:45     bge0  106039    0.09 71754.2    1.30  1513.3  
67.85  86.9   0.00
14:04:55     bge0  116455    0.08 78784.6    1.19  1513.6  
65.83  95.4   0.00
14:05:05     bge0  103871    0.07 70285.7    1.11  1513.3  
68.36  85.1   0.00
14:05:15     bge0  108125    0.28 73148.8    1.90  1513.6  
152.6  88.6   0.00
14:05:25     bge0  107270    0.11 72562.5    1.40  1513.8  
83.57  87.9   0.00
14:05:35     bge0  117035    0.09 79179.9    1.30  1513.6  
68.00  95.9   0.00
14:05:45     bge0  107024    0.11 72394.8    1.30  1513.8  
83.92  87.7   0.00
14:05:55     bge0  107191    0.07 72514.6    1.10  1513.7  
65.82  87.8   0.00
14:06:05     bge0 50169.2    0.07 33961.3    1.10  1512.7  
65.82  41.1   0.00
14:06:15     bge0    6.49    0.14   39.91    1.80  
166.5   81.50  0.01   0.00

./tcpstat.d_10s

    TCP_out   TCP_outRe      TCP_in   TCP_inDup    TCP_inUn
        984           0        
216           0           0
       2044           0   868107184          
0     9183872
       4478           0   978826459          
0    10973360
       3060           0  1051471652          
0     9635584
       3008           0  1050969704          
0     9083768
       3032           0  1039588008       
1448     9397584
       4624           0  1069841236          
0     9893776
       3294           0  1063289869          
0     9569424
       2996           0  1053051824          
0     9521664
       3173           0  1065730188          
0     8924416
       4390           0  1060844483          
0     9783496
       2148           0   213176464          
0     1949288
       1421           0        
549           0           0

./sysperfstat.pl 10

            ------ Utilisation ------     ------ Saturation
------
    Time    %CPU   %Mem  %Disk   %Net     CPU   
Mem   Disk    Net
14:04:12    2.34  11.44   0.00   0.04    0.60   0.00  
0.00   0.00
14:04:22   25.31  11.54   7.13  57.16    0.40   0.00  
0.00   0.00
14:04:33   62.30  11.58   0.00 167.63    0.40   0.00  
0.00   0.00
14:04:43   70.48  11.61   0.00 183.50    1.89   0.00  
0.00   0.00
14:04:53   65.37  11.65   8.31 175.27    0.80   0.00  
0.00   0.00
14:05:03   71.65  11.68   6.57 167.08    0.99   0.00  
0.00   0.00
14:05:13   65.48  11.71   0.00 195.03    0.69   0.00  
0.00   0.00
14:05:23   64.51  11.75   8.21 174.61    0.79   0.00  
0.00   0.00
14:05:33   65.69  11.86   0.00 175.13    0.40   0.00  
0.00   0.00
14:05:43   66.90  11.81   0.00 191.91    1.79   0.00  
0.00   0.00
14:05:53   64.84  11.85   7.10 174.58    0.60   0.00  
0.00   0.00
14:06:03   46.51  11.44   0.00 117.98    0.30   0.00  
0.00   0.00
14:06:13    2.19  11.44   0.00   0.04    0.60   0.00  
0.00   0.00

../bin/se nx.se 10

14:04:12 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp        0.7    1.0   0.03   0.12   0.00   0.00  
0.0   0.00   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0      37.7    1.2   6.48   0.08  0.000  0.000  
0.0   0.00   0.00
bge1      36.3    2.8   6.37   0.29  0.000  0.000  
0.0   0.00   0.00
bge2       2.2    2.4   0.18   0.18  0.000  0.000  
0.0   0.00   0.00
bge3       2.5    2.3   0.23   0.17  0.000  0.000  
0.0   0.00   0.00

14:04:22 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     22849.5    1.0 32231.14   0.20   0.00   0.00  
0.0   0.80   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    16358.0    1.0 24146.71   0.06  0.000  0.000  
0.0   0.00   0.00
bge1      31.2 2149.1   5.25 149.66  0.000  0.000   0.0  
0.00   0.00
bge2       2.7    2.3   0.27   0.19  0.000  0.000  
0.0   0.00   0.00
bge3       1.9    1.9   0.12   0.12  0.000  0.000  
0.0   0.00   0.00

14:04:32 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     70802.8    1.7 99860.03   0.44   0.00   0.00  
0.0   0.50   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    76169.3    1.2 112587.84   0.08  0.000  0.000  
0.0   0.00   0.00
bge1      36.5 10615.4   6.10 737.66  0.000  0.000   0.0  
0.00   0.00
bge2       3.1    2.7   0.30   0.21  0.000  0.000  
0.0   0.00   0.00
bge3       2.3    2.2   0.15   0.14  0.000  0.000  
0.0   0.00   0.00

14:04:42 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     70290.7    1.4 99111.98   0.30   0.00   0.00  
0.0   0.40   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    68262.3    1.3 100870.34   0.09  0.000  0.000  
0.0   0.00   0.00
bge1      34.0 11692.4   5.89 810.57  0.000  0.000   0.0  
0.00   0.00
bge2       3.0    3.2   0.35   0.35  0.000  0.000  
0.0   0.00   0.00
bge3       2.0    2.0   0.15   0.13  0.000  0.000  
0.0   0.00   0.00

14:04:52 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     74031.4    1.3 104419.59   0.30   0.00   0.00  
0.0   0.40   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    72090.2    1.2 106559.85   0.08  0.000  0.000  
0.0   0.00   0.00
bge1      33.7 9893.7   5.68 685.94  0.000  0.000   0.0  
0.00   0.00
bge2       2.0    2.3   0.13   0.17  0.000  0.000  
0.0   0.00   0.00
bge3       2.8    2.2   0.28   0.16  0.000  0.000  
0.0   0.00   0.00

14:05:02 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     71402.9    1.2 100687.96   0.29   0.00   0.00  
0.0   0.40   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    76589.5    1.1 113191.06   0.07  0.000  0.000  
0.0   0.00   0.00
bge1      36.4 12128.9   6.01 840.04  0.000  0.000   0.0  
0.00   0.00
bge2       2.2    2.1   0.14   0.13  0.000  0.000  
0.0   0.00   0.00
bge3       3.2    2.8   0.34   0.23  0.000  0.000  
0.0   0.00   0.00

14:05:12 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     75003.4    1.5 105785.48   0.32   0.00   0.00  
0.0   0.40   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    73186.5    1.4 108176.92   0.11  0.000  0.000  
0.0   0.00   0.00
bge1      35.0 10130.7   6.06 702.60  0.000  0.000   0.0  
0.00   0.00
bge2       2.0    2.1   0.13   0.14  0.000  0.000  
0.0   0.00   0.00
bge3       2.8    2.7   0.29   0.26  0.000  0.000  
0.0   0.00   0.00

14:05:22 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     74152.6    1.4 104598.08   0.43   0.00   0.00  
0.0   0.50   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    72394.3    1.7 107015.80   0.25  0.000  0.000  
0.0   0.00   0.00
bge1      33.2 10090.5   5.47 699.50  0.000  0.000   0.0  
0.00   0.00
bge2       2.5    1.9   0.23   0.12  0.000  0.000  
0.0   0.00   0.00
bge3       2.1    2.4   0.17   0.19  0.000  0.000  
0.0   0.00   0.00

14:05:32 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     74356.8    1.4 104877.81   0.32   0.00   0.00  
0.0   0.40   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    72267.1    1.4 106825.49   0.11  0.000  0.000  
0.0   0.00   0.00
bge1      34.3 9607.7   5.85 666.12  0.000  0.000   0.0  
0.00   0.00
bge2       2.8    2.0   0.28   0.13  0.000  0.000  
0.0   0.00   0.00
bge3       2.1    2.5   0.14   0.20  0.000  0.000  
0.0   0.00   0.00

14:05:42 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     73952.7    1.4 104306.72   0.31   0.00   0.00  
0.0   0.40   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    79403.5    1.4 117373.51   0.11  0.000  0.000  
0.0   0.00   0.00
bge1      36.5 11148.0   6.24 772.15  0.000  0.000   0.0  
0.00   0.00
bge2       3.4    3.0   0.37   0.31  0.000  0.000  
0.0   0.00   0.00
bge3       2.1    2.2   0.14   0.15  0.000  0.000  
0.0   0.00   0.00

14:05:52 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     73571.5    1.4 103783.93   0.43   0.00   0.00  
0.0   0.50   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    71733.9    1.2 106045.25   0.08  0.000  0.000  
0.0   0.00   0.00
bge1      34.8 9590.5   5.71 665.08  0.000  0.000   0.0  
0.00   0.00
bge2       2.4    2.4   0.20   0.19  0.000  0.000  
0.0   0.00   0.00
bge3       2.4    2.0   0.20   0.13  0.000  0.000  
0.0   0.00   0.00

14:06:02 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp     53402.0    0.8 75321.52   0.19   0.00   0.00  
0.0   0.00   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0    56056.9    1.1 82846.81   0.07  0.000  0.000  
0.0   0.00   0.00
bge1      37.2 7997.7   6.58 554.43  0.000  0.000   0.0  
0.00   0.00
bge2       2.0    2.3   0.13   0.19  0.000  0.000  
0.0   0.00   0.00
bge3       2.7    1.9   0.28   0.12  0.000  0.000  
0.0   0.00   0.00

14:06:12 Iseg/s Oseg/s InKB/s OuKB/s Rst/s  Atf/s  Ret%  Icn/s  Ocn/s
tcp        0.9    1.1   0.05   0.14   0.00   0.00  
0.0   0.00   0.00
Name    Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s
bge0      36.4    1.6   5.91   0.13  0.000  0.000  
0.0   0.00   0.00
bge1      34.8    2.8   5.77   0.29  0.000  0.000  
0.0   0.00   0.00
bge2       2.0    2.9   0.13   0.28  0.000  0.000  
0.0   0.00   0.00
bge3       3.0    2.1   0.33   0.14  0.000  0.000  
0.0   0.00   0.00

So far, it seems that most tools show about the same numbers - beside the
DTrace script "nx.se", which output i just do not understand.

 

ShoudnÂŽt "TCP_in" be number of bytes read from network?

Those numbers are by far too high!
For example, 1039588008 is 1,039,588,008 (one billion) ...

What shall that be?

Can anybody enlighten me about the results of "tcpstat.d"?

I am just not sure how to interprete the numbers - or if those results are comparable to other tools at all.

 

BTW, i created "tcpstat.d_10s" from "tcpstat.d" with changing "tick-1sec" to "tick-10sec".

But the numbers are the same for the original script!

 

Thank you very much!

[ Tolle Angebote findet ihr unter: http://www.uni.de/shopping ]
Thomas
2011-04-06 14:11:34 UTC
Permalink
Raw Message
Post by Thomas
So far, it seems that most tools show about the same numbers - beside
the DTrace script "nx.se", which output i just do not understand.
 

Ups.

I meant to write the DTrace ToolKit script "tcpstat.d", not the SE
ToolKit script "nx.se", sorry!

 
[ Tolle Angebote findet ihr unter: http://www.uni.de/shopping ]
Mike Harsch
2011-04-06 14:29:50 UTC
Permalink
Raw Message
BTW, i created "tcpstat.d_10s" from "tcpstat.d" with changing "tick-1sec" to "tick-10sec".
But the numbers are the same for the original script!
Hi Thomas,

Dividing the tcpstat.d output values by 10 brings them in line with the
rest of your measurements. Are you sure you didn't make a 10x error
when comparing your modified tcpstat.d with the original? (hint: the
key probe is the last one in the script where the counters are reset).

Mike
Thomas
2011-04-06 15:17:56 UTC
Permalink
Raw Message
Post by Mike Harsch
Dividing the tcpstat.d output values by 10 brings them in line with the
rest of your measurements.  Are you sure you didn't make a 10x error when
comparing your modified tcpstat.d with the original?  (hint: the key probe
is the last one in the script where the counters are reset).

 

Uhmm.

Yes. YouÂŽre right, kind of :)

Thank you!

 

I tried again with the regular script:

    TCP_out   TCP_outRe      TCP_in   TCP_inDup    TCP_inUn
        100           0  
107159168           0           0
        316           0  
103296672           0           0
        100           0  
105086552           0           0
        612           0  
102532224           0           0
        100           0  
102800916           0      110048
        100           0  
107525452           0           0
        100           0  
101727804           0      111000
        100           0  
108321680           0           0
        100           0  
104378992           0      116792
        100           0  
104001056           0      111000
        100           0  
105311944           0      122408
        100           0  
106907924           0           0
        100           0  
105918264           0           0
        612           0   
98864416           0           0
        100           0  
104936868           0           0
        100           0  
108334572           0           0
        100           0  
106463980           0           0
        100           0  
109640400           0           0
        100           0  
103656656           0      124032
        100           0  
104951880           0      121136

Now we have 105311944 = 105,311,944 which is 100 million.

But this is still too much compared to the the other tools, which show
about 70 to 80 MB/s, which seems to right for a 1Gb link.

Am i doing something wrong?

Or do i missinterprete the output of the script?

 

Thanx for any further enlightenment :)
[ Tolle Angebote findet ihr unter: http://www.uni.de/shopping ]
Thomas
2011-04-06 15:33:24 UTC
Permalink
Raw Message
Post by Thomas
But this is still too much compared to the the other tools, which show
about 70 to 80 MB/s, which seems to right for a 1Gb link.

Uhh.

Forget about this :)

It seems i already assumed a performance of about 80 MB/s and misread the
output of the other commands too (number of packages).

100 MB/s seems to be possible over 1Gb lines - which i did not expect to
happen.
[ Tolle Angebote findet ihr unter: http://www.uni.de/shopping ]
Mike Harsch
2011-04-06 16:16:26 UTC
Permalink
Raw Message
Post by Thomas
100 MB/s seems to be possible over 1Gb lines - which i did not expect
to happen.
100MBytes/sec is certainly possible on a 1Gbps link. If you get
results >125MBytes/sec, then you should be skeptical ;-)

Mike

Loading...