following a packet from sendto() to wire (and reverse)

[Mathias Koerber]

I am quite unfamiliar with DTrace (and even more with
Solaris/Linux/other) kernel internals.

I recently had to help look into an issue where an application
apparently sent a UDP packet, but it did not appear on the wire, nor
did snoop(1)ing the interface show it, so the assumption is that it
was lost (or misdirected) somewhere in the IP stack.

We did use Dtrace with syscall::sendto:entry/exit to prove that the
application did send the packet, but so far we have no clue how to
determine what happens to the packet afterwards.

Is there a Dtrace script that follows a packet queued bia send(2) or
sendto(2) through the kernel/stack and can indicate
whether some low-level function returns a failure, or
possibly hands it to a different interface (we snooped all except the
unsnoopable lo though w/o luck)..

I looked at the Dtrace Networking Guide but without more knowledge
of internals it was no help.

PS: The Solaris-10 machine I work on is old, so it has no
ip nor udp provider, but if it takes that I'd also like to know.

any hints on where to start would be welcome.

[[e-mail]]

You say the Solaris 10 machine is "old". How up-to-date is it (/etc/release) and how well patched? There have been many fixes in the history of Solaris 10, not least to its network stack. It would be a shame to invest a lot of effort in finding a bug that may have already been fixed. The patch database is worth a trawl, but do bear in mind that many bugs get fixed without be explicitly called out in bug reports.

----- Reply message -----
From: "Mathias Koerber" <***@koerber.org>
To: <dtrace-***@opensolaris.org>
Subject: [dtrace-discuss] following a packet from sendto() to wire (and reverse)
Date: Thu, Aug 25, 2011 01:40


I am quite unfamiliar with DTrace (and even more with
Solaris/Linux/other) kernel internals.

I recently had to help look into an issue where an application
apparently sent a UDP packet, but it did not appear on the wire, nor
did snoop(1)ing the interface show it, so the assumption is that it
was lost (or misdirected) somewhere in the IP stack.

We did use Dtrace with syscall::sendto:entry/exit to prove that the
application did send the packet, but so far we have no clue how to
determine what happens to the packet afterwards.

Is there a Dtrace script that follows a packet queued bia send(2) or
sendto(2) through the kernel/stack and can indicate
whether some low-level function returns a failure, or
possibly hands it to a different interface (we snooped all except the
unsnoopable lo though w/o luck)..

I looked at the Dtrace Networking Guide but without more knowledge
of internals it was no help.

PS: The Solaris-10 machine I work on is old, so it has no
ip nor udp provider, but if it takes that I'd also like to know.

any hints on where to start would be welcome.

[James Carlson]

Post by Mathias Koerber
We did use Dtrace with syscall::sendto:entry/exit to prove that the
application did send the packet, but so far we have no clue how to
determine what happens to the packet afterwards.

Can you show us the trace script you used and the output it showed? How
about details on the network configuration and the packet being sent?
Perhaps there's something in the details that someone here (or on
networking-discuss) will recognize.

For what it's worth, I was able to use the following trivial dtrace
script to follow a packet all the way from sendto in the kernel down to
the device driver on a Solaris 10 system, and see the MIB updates along
the way. MIB updates provide a fair indication of error activity.
(Note that tracing 'sendto' might not work for all UDP sockets ... if
the socket is connected and uses write(), you won't see any trace
activity.) (Also note that on a busy system, you may need to provide
more conditions on the initial probe [such as checking execname], and
may want to limit the module[s] in which you're tracing.)

#!/usr/sbin/dtrace -Fs -

syscall::sendto:entry
{
self->trace++;
}

fbt:::
/self->trace/
{
}

mib:::
/self->trace/
{
}

syscall::sendto:return
/self->trace/
{
self->trace = 0;
}