measuring NFS stack times

[Kyle Hailey]

I want to measure as many of the layers of the NFS stack starting with the
client IO request down to the NFS Server.
Measuring the NFS Server seems pretty easy with

nfsv3:::op-read-start ,nfsv4:::op-read-start
nfsv3:::op-read-done, nfsv4:::op-read-done

I can even go below the NFS server pretty easy. In my case I'm using ZFS so
I can use
zfs_read:entry to zfs_read:return

What I don't see how to do is measure the other layers of the stack. I'm
imagining the stack like
(this will only look correct in fixed width fonts)

user| |user
asks| |gets
for | |the
io | Machine A |IO
---- -----
1 |OS| |OS| 15
2 |NFS| |NFS| 14
3 |TCP| |TCP| 13
-----------------------------------------------------------------
network 4 |NET| |NET| 12
------------------------------------------------------------------
5 |TCP| Machine B |TCP| 11
6 |OS| |OS| 10
7 |NFS| |NFS| 9
8 |IO|

The layers I measure now
users start and end time
7 nfsv3:::op-read-start ,nfsv4:::op-read-start
8 zfs_read:entry to zfs_read:return
9 nfsv3:::op-read-done, nfsv4:::op-read-done

I'm wondering if and/or how I could measure the other layers.

[Adam Leventhal]

Hey Kyle,

user|                                                       |user
asks|                                                       |gets
for |                                                       |the
 io |                       Machine A                       |IO
----                                                         -----
  1 |OS|                                                 |OS| 15
    2  |NFS|                                         |NFS| 14
        3  |TCP|                                 |TCP| 13
-----------------------------------------------------------------
network     4  |NET|                        |NET| 12
------------------------------------------------------------------
               5   |TCP|    Machine B   |TCP| 11
                   6   |OS|          |OS| 10
                      7   |NFS|  |NFS| 9
                          8   |IO|
The layers I measure now
  users start and end time
7 nfsv3:::op-read-start ,nfsv4:::op-read-start
8 zfs_read:entry to zfs_read:return
9 nfsv3:::op-read-done, nfsv4:::op-read-done
I'm wondering if and/or how I could measure the other layers.

You can use the ip provider for the IP component, but you'll need to
resort to fbt to figure out the OS (what exactly are you looking
for?), NFS client, and TCP components (or the tcp provider on builds
142 and later). To figure out what fbt functions to trace, here are
some tips:

fbt:nfs:nfs3_*:entry/return will get the NFSv3 client operations.
For the TCP stuff, take a look at Alan McGuire's work (hg version
501806a754d21dae2120629e8d705674877328f2) to figure out the relevant
functions to instrument with the fbt provider.

I hope that helps.

Adam

--
Adam Leventhal, Delphix
http://dtrace.org/blogs/ahl

275 Middlefield Road, Suite 50
Menlo Park, CA 94025
http://www.delphix.com