dtrace and kmem heap corruption ?

[Peter Shoults]

Hi,

I wrote a dtrace script that is tracking a couple of routines that
decrement a reference count, trying to catch the thread that
de-references the count down to -1. The script works fine, except for
some reason, I suspect it is not recording the last few decrements.
Probably best to show what I mean:

from msgbuf of corefile:

kernel memory allocator: buffer modified after being freed
modification occurred at offset 0x0 (0xdeadbeefdeadbeef replaced by
0xdeadbeeedeadbeef)
buffer=3004ec26af8 bufctl=3004ec65d20 cache: kmem_alloc_56

Now, grepping for that bufaddr on my dtrace script output...


1 20287 crgrprele:entry cred_ptr is 3004ec26af8, cred ref = 5
1 20287 crgrprele:entry cred_ptr is 3004ec26af8, cred ref = 5
1 20287 crgrprele:entry cred_ptr is 3004ec26af8, cred ref = 5
1 20287 crgrprele:entry cred_ptr is 3004ec26af8, cred ref = 4
1 20287 crgrprele:entry cred_ptr is 3004ec26af8, cred ref = 3
#

I have to believe we decremented down to zero and freed, then again,
maybe we did not decrement down to zero and still freed, and then when
buffer was on free list, we tried to decrement thinking we still had to
and we were still active. Either way - what can I do to catch what I
believe must have been at least one more decrement - it is seems as if
the decrement occurred, and we went belly up so quickly dtrace never had
a chance to record the decrement.

Anyway - if there is a way to boost what dtrace can capture, or ensure
it actually records when an event is triggered...that would be nice.

Pete

Peter

[Jonathan Adams]

Post by Peter Shoults
Hi,
I wrote a dtrace script that is tracking a couple of routines that
decrement a reference count, trying to catch the thread that
de-references the count down to -1. The script works fine, except for
some reason, I suspect it is not recording the last few decrements.
kernel memory allocator: buffer modified after being freed
modification occurred at offset 0x0 (0xdeadbeefdeadbeef replaced by
0xdeadbeeedeadbeef)
buffer=3004ec26af8 bufctl=3004ec65d20 cache: kmem_alloc_56

^^^^^^^^^^^

Have you done:

3004ec65d20::bufctl -v

and printed the "CONTENTS" field as a credgrp_t?

Post by Peter Shoults
Now, grepping for that bufaddr on my dtrace script output...
1 20287 crgrprele:entry cred_ptr is 3004ec26af8, cred ref = 5
1 20287 crgrprele:entry cred_ptr is 3004ec26af8, cred ref = 5
1 20287 crgrprele:entry cred_ptr is 3004ec26af8, cred ref = 5
1 20287 crgrprele:entry cred_ptr is 3004ec26af8, cred ref = 4
1 20287 crgrprele:entry cred_ptr is 3004ec26af8, cred ref = 3
#
I have to believe we decremented down to zero and freed, then again,
maybe we did not decrement down to zero and still freed, and then when
buffer was on free list, we tried to decrement thinking we still had to
and we were still active. Either way - what can I do to catch what I
believe must have been at least one more decrement - it is seems as if
the decrement occurred, and we went belly up so quickly dtrace never had
a chance to record the decrement.
Anyway - if there is a way to boost what dtrace can capture, or ensure
it actually records when an event is triggered...that would be nice.

There are DCMDs in mdb to dump the in-kernel dtrace state:

::dtrace_state

lists the active dtrace consumers, and

addr::dtrace

will process the in-kernel state for a particular consumer.

Cheers,
- jonathan