trace: error on enabled probe ID invalid address

[Hans-Peter]

Hi

I am trying to make a dtrace script that captures tcp packets sent by a specific process.

But I receive the message:
dtrace: error on enabled probe ID 3 (ID 35884: fbt:sockfs:sostream_direct:return): invalid address (0x106390000) in action #1 at DIF offset 12

Can someone explain why this happens?

regards HansP

#!/usr/sbin/dtrace -Cs
/*
* Command line arguments
*/
#include <sys/file.h>
#include <inet/common.h>
#include <sys/byteorder.h>
#include <sys/socket.h>
#include <sys/socketvar.h>

/*
* Print header
*/
dtrace:::BEGIN
{
/* starting values */
counts = COUNTER;
secs = INTERVAL;
TCP_out = 0;
TCP_in = 0;

printf("Tracing... Please wait.\n");

start = 0;
}

fbt:sockfs:sostream_direct:entry
/ pid == $1 && start == 0 /
{
self->sop = 1;
self->nsop = (struct sonode *)arg1;
self->tcpp = (tcp_t *)self->nsop->so_priv;
self->laddrs = self->nsop->so_laddr_sa;
start = 1;
printf("%50s : %10d\n","fbt:sockfs:sostream_direct:entry",self->nsop->so_sndbuf);
}

fbt:sockfs:sostream_direct:return
/ pid == $1 && start == 1 /
{
self->connp = (conn_t *)self->tcpp->tcp_connp;
/*printf("%50s %10d\n","fbt:sockfs:sostream_direct:return",self->laddr->soa_len); */
printf("%50s \n","fbt:sockfs:sostream_direct:return");
}

[Brian Utterback]

Post by Hans-Peter
Hi
I am trying to make a dtrace script that captures tcp packets sent by a specific process.
dtrace: error on enabled probe ID 3 (ID 35884: fbt:sockfs:sostream_direct:return): invalid address (0x106390000) in action #1 at DIF offset 12
Can someone explain why this happens?
regards HansP
#!/usr/sbin/dtrace -Cs
/*
* Command line arguments
*/
#include <sys/file.h>
#include <inet/common.h>
#include <sys/byteorder.h>
#include <sys/socket.h>
#include <sys/socketvar.h>
/*
* Print header
*/
dtrace:::BEGIN
{
/* starting values */
counts = COUNTER;
secs = INTERVAL;
TCP_out = 0;
TCP_in = 0;
printf("Tracing... Please wait.\n");
start = 0;
}
fbt:sockfs:sostream_direct:entry
/ pid == $1 && start == 0 /
{
self->sop = 1;
self->nsop = (struct sonode *)arg1;
self->tcpp = (tcp_t *)self->nsop->so_priv;
self->laddrs = self->nsop->so_laddr_sa;
start = 1;
printf("%50s : %10d\n","fbt:sockfs:sostream_direct:entry",self->nsop->so_sndbuf);
}
fbt:sockfs:sostream_direct:return
/ pid == $1 && start == 1 /
{
self->connp = (conn_t *)self->tcpp->tcp_connp;
/*printf("%50s %10d\n","fbt:sockfs:sostream_direct:return",self->laddr->soa_len); */
printf("%50s \n","fbt:sockfs:sostream_direct:return");
}

Well, you have a few potential problems with this script, for instance
the return probe is enabled via global variables, but dereferences
thread-local storage which may not be valid. However, the cause of your
immediate problem is that the argx variables are indexed from zero and
not one. I think you need

self->nsop = (struct sonode *)arg0;

[Katsunori FUJIWARA]

Hi, Hans:

Even if you would fix your D script as mentioned by Brian,
you may see same DTrace error("invalid address .....") at runtim.

If so, you may hit DTrace bug 6734858

# http://www.opensolaris.org/jive/thread.jspa?threadID=127996

The ways to avoid this problems are:

- use patched libctf.so (see above link for detail)

- (re-)define target(and nested) structure
with explict padding for alignment in your D script

- any other ?

[Hans-Peter]

Hi Brian,

Thanks for your reply.
I fixed it.

I think I need to read the manuals a bit better.

regards Hans-Peter

[Alan Maguire]

hi Hans-Peter

the good news is this will be a lot easier soon - we
recently integrated a DTrace tcp provider, and it
will be as easy as

tcp:::send
/ args[1]->cs_pid == $1 /
{
...
}


See:

http://wikis.sun.com/display/DTrace/tcp+Provider

...for more details.

In the interim (assuming you are using OpenSolaris
not Solaris 10), it might be worth tracing at the tcp
layer rather than at the socket layer (assuming
you're not doing anything timing-wise that
needs to be done at the socket layer):

fbt:ip:tcp_send_data:entry
/ args[0]->tcp_connp->conn_ixa->ixa_cpid == $1 /
{
printf("TCP send : pid %d laddr/port : %s/%d raddr/port %s/%d\n",
args[0]->tcp_connp->conn_ixa->ixa_cpid,
inet_ntoa6(&args[0]->tcp_connp->connua_v6addr.connua_laddr),
ntohs(args[0]->tcp_connp->u_port.connu_ports.connu_lport),
inet_ntoa6(&args[0]->tcp_connp->connua_v6addr.connua_faddr),
ntohs(args[0]->tcp_connp->u_port.connu_ports.connu_fport));
}

Sample output:

1 46446 tcp_send_data:entry TCP send : pid 132408
laddr/port : 129.150.120.230/46294 raddr/port 208.52.173.220/443

1 46446 tcp_send_data:entry TCP send : pid 132408
laddr/port : 129.150.120.230/34961 raddr/port 12.129.147.65/80

1 46446 tcp_send_data:entry TCP send : pid 132408
laddr/port : 129.150.120.230/34961 raddr/port 12.129.147.65/80


You'll need to use fbt::fuse-output:entry too for
TCP-fused localhost connections (where TCP
doesn't bother encapsulating localhost traffic
in TCP headers for performance reasons), and
this will also miss some TCP control segments,
but it may be good enough to see what's going on.
This will only work on reasonably recent builds
(130ish+) of OpenSolaris I suspect.

Hope this helps,

Alan

Post by Hans-Peter
Hi
I am trying to make a dtrace script that captures tcp packets sent by a specific process.
dtrace: error on enabled probe ID 3 (ID 35884: fbt:sockfs:sostream_direct:return): invalid address (0x106390000) in action #1 at DIF offset 12
Can someone explain why this happens?
regards HansP
#!/usr/sbin/dtrace -Cs
/*
* Command line arguments
*/
#include<sys/file.h>
#include<inet/common.h>
#include<sys/byteorder.h>
#include<sys/socket.h>
#include<sys/socketvar.h>
/*
* Print header
*/
dtrace:::BEGIN
{
/* starting values */
counts = COUNTER;
secs = INTERVAL;
TCP_out = 0;
TCP_in = 0;
printf("Tracing... Please wait.\n");
start = 0;
}
fbt:sockfs:sostream_direct:entry
/ pid == $1&& start == 0 /
{
self->sop = 1;
self->nsop = (struct sonode *)arg1;
self->tcpp = (tcp_t *)self->nsop->so_priv;
self->laddrs = self->nsop->so_laddr_sa;
start = 1;
printf("%50s : %10d\n","fbt:sockfs:sostream_direct:entry",self->nsop->so_sndbuf);
}
fbt:sockfs:sostream_direct:return
/ pid == $1&& start == 1 /
{
self->connp = (conn_t *)self->tcpp->tcp_connp;
/*printf("%50s %10d\n","fbt:sockfs:sostream_direct:return",self->laddr->soa_len); */
printf("%50s \n","fbt:sockfs:sostream_direct:return");
}

[Hans-Peter]

Well the bad news is that I am on Solaris 10. ;-)

Furthermore I am on a global zone (you cannot use dtrace in a local zone)
The traffic I see seems to be only traffic that is redirected to the local zones.

Using tcptop_snv the LADDR is always 0.0.0.0 or 1.0.0.127.

010 Jun 4 15:12:01, load: 13.36, TCPin: 71 KB, TCPout: 874 KB

UID PID LADDR LPORT RADDR RPORT SIZE NAME
0 0 1.0.0.127 6100 1.0.0.127 48002 108 <closed>
0 0 1.0.0.127 6100 1.0.0.127 48003 108 <closed>
4702 29611 0.0.0.0 0 10.1.73.116 0 390 opmn
4703 21444 0.0.0.0 0 10.1.73.117 0 780 emagent
4701 28642 0.0.0.0 0 10.1.73.116 0 6633 tnslsnr

Regards HansP