Monitoring filessytem access

[Edward Ned Harvey]

I haven't looked into dtrace much yet, because it's obviously complex, and
I'm not sure if it's the right tool for the job. Can dtrace help me solve
this?



When somebody is hammering on the system, I want to be able to detect who's
doing it, and hopefully even what they're doing.



Everything I can find ... iostat, nfsstat, etc ... AFAIK, just show me
performance statistics and so forth. I'm looking for something more
granular. Either *who* the activity belongs to, or *what* files are active
(which might indirectly let me figure out who) ... etc.



If it makes the problem any easier (or harder) all the access is coming
across NFS. And maybe a little CIFS.

[Martin Mueller - Sun Germany SE]

Have a look at the Dtrace Toolkit from Brendan Gregg. rwsnoop should do
what you are asking for on files

Regards
Martin

Post by Edward Ned Harvey
I haven't looked into dtrace much yet, because it's obviously complex,
and I'm not sure if it's the right tool for the job. Can dtrace help me
solve this?
When somebody is hammering on the system, I want to be able to detect
who's doing it, and hopefully even what they're doing.
Everything I can find ... iostat, nfsstat, etc ... AFAIK, just show me
performance statistics and so forth. I'm looking for something more
granular. Either *who* the activity belongs to, or *what* files are
active (which might indirectly let me figure out who) ... etc.
If it makes the problem any easier (or harder) all the access is coming
across NFS. And maybe a little CIFS.
_______________________________________________
dtrace-discuss mailing list

--
Martin Mueller martin.mueller-xsfywfwIY+***@public.gmane.org
Systems Infrastructure Ambassador Tel.: [phone]
Sun Microsystems GmbH Fax: [phone]
D-40880 Ratingen Mobile: [phone]

Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551
Kirchheim-Heimstetten
Amtsgericht Muenchen: HRB 161028
Geschaeftsfuehrer: Juergen Kunz

[Edward Ned Harvey]

Post by Martin Mueller - Sun Germany SE
Have a look at the Dtrace Toolkit from Brendan Gregg. rwsnoop
should do what you are asking for on files

This seemed very promising when I first saw it ... But it doesn't seem to capture NFS activity. I sit here and watch it, meanwhile start "dd if=/dev/zero of=junk.file" on some other machine, writing over NFS, and ... No result appears on the rwsnoop output. ;-(

Any other suggestions?

Thank you for this one.

[Edward Ned Harvey]

Yessss....



man nfslogd

(and google nfslogd, etc)

totally nailed it. (Thanks to Cameron.)



Just incase anyone stumbles across this by search, here's the start-to-end
answer:

man nfslogd

First, edit /etc/default/nfslogd I am using:

MIN_PROCESSING_SIZE=1

IDLE_TIME=1

Assuming you already have svc:/network/nfs/server:default enabled

If you are using /etc/dfs/dfstab instead of the sharenfs property, just add
",log" as specified by man share_nfs

If you wish, run that share command now.

By simply adding ,log to the dfstab, nfslogd will automatically start
after reboot

or start it manually now: /usr/lib/nfs/nfslogd

If you are using the sharenfs property instead of dfstab, just figure out
how to add the ,log option

tail -f /var/nfs/nfslog



Again man nfslogd. Information in the log won't be complete until NFS
services are restarted, which is well documented in the man page.

[Richard Elling]

[Apologies for the delayed response]

NFS logging kills performance. I have witnessed more than 85% less
performance when NFS logging was enabled. The DTrace providers are
much more efficient.
00 rucgard

[Brendan Gregg - Sun Microsystems]

G'Day,

Post by Edward Ned Harvey

Post by Martin Mueller - Sun Germany SE
Have a look at the Dtrace Toolkit from Brendan Gregg. rwsnoop
should do what you are asking for on files

This seemed very promising when I first saw it ... But it doesn't seem
to capture NFS activity. I sit here and watch it, meanwhile start "dd
if=/dev/zero of=junk.file" on some other machine, writing over NFS,
and ... No result appears on the rwsnoop output. ;-(
Any other suggestions?

rwsnoop traces syscalls, and would work from the client. On the server,
NFS is kernel-based and accesses files without syscalls.

Use the NFS providers, if available, since they are a stable interface. Eg
(NFSv3):

# dtrace -n 'nfsv3:::op-read-start,nfsv3:::op-write-start { @[probefunc,
# args[1]->noi_curpath] = count(); }'

If they aren't available, you can try the fsinfo provider - if available. Eg:

# dtrace -n 'fsinfo:::read,fsinfo:::write { @[probefunc, args[0]->fi_pathname]
# = count(); }'

If that isn't available, you can dig it out of the (unstable) fbt provider,
and maybe sdt. You could probe NFS functions, the filesystem, or VFS. eg,
VFS on Solaris:

# dtrace -n 'fbt::fop_read:entry,fbt::fop_write:entry { @[probefunc,
# stringof(args[0]->v_path)] = count(); }'

Brendan