rt remote free backtrace (#2)

[Sepherosa Ziehau]

On Sun, Nov 2, 2008 at 6:25 AM, Peter Avalos <pavalos at theshell.com> wrote:
> On Thu, Oct 30, 2008 at 10:04:10PM +0800, Sepherosa Ziehau wrote:
>> On Thu, Oct 30, 2008 at 10:06 AM, Sepherosa Ziehau <sepherosa at gmail.com> wrote:
>> > On Thu, Oct 30, 2008 at 12:12 AM, Peter Avalos <pavalos at theshell.com> wrote:
>> >> On Thu, Sep 11, 2008 at 07:36:02PM +0800, Sepherosa Ziehau wrote:
>> >>> Hi all,
>> >>>
>> >>> I have committed a patch to HEAD, which will print backtrace if
>> >>> rtfree_remote() is called.  If you see this kind of messages on your
>> >>> console, please post them to bugs at .  If you think your box could
>> >>> afford panicking (mainly to get coredumps), you could set
>> >>> net.route.remote_free_panic to 1, and post the URL to the coredump so
>> >>> we could fetch and debug it.
>> >>>
>> >>
>> >> Here's a different one I got twice yesterday:
>> >>
>> >> rt remote free rt_cpuid 1, mycpuid 0
>> >> Trace beginning at frame 0xda6f5c90
>> >> rtfree_remote(c0371380,1,0,da6f5cf4,c023dc8d) at rtfree_remote+0x74
>> >> rtfree_remote(f059e120,1) at rtfree_remote+0x74
>> >> rtfree(f059e120,daa41080,da6f5d14,c023e36c,daa44680) at rtfree+0x2e
>> >> in_rtchange(daa44680,0) at in_rtchange+0x14
>> >> in_pcbnotifyall(c0421f58,91fa5e77,0,c023d65a) at in_pcbnotifyall+0x46
>> >> udp_ctlinput(f,d73c9bf8,0,0,da6f5d68) at udp_ctlinput+0x10f
>> >> netmsg_pru_ctlinput(d73c9b1c,0,c0421f24,da6f5d84,c0233f7a) at
>> >> netmsg_pru_ctlinput+0x13
>> >> netmsg_service(d73c9b1c,1,0,c03f3934,ff800000) at netmsg_service+0x44
>> >> netmsg_service_loop(c0421f24,0,0,0,0) at netmsg_service_loop+0x18
>> >> lwkt_exit() at lwkt_exit
>> >
>> > Ok, I see.  We need to locate the msgport based on 'cmd' paraeter
>> > passed to pr_ctlport too.  And forwarding in_pcbnotifyall() to all
>> > related msgports (i.e. all udp msgports or all tcp msgports).  I will
>> > give you a patch to try when I get home.
>> >
>>
>> Please test/review following patch:
>> http://leaf.dragonflybsd.org/~sephe/tcp_udp_ctlinput.diff
>>
>
> Panic:
>
> panic: netmsg_put_port: self-referential deadlock on netport
> mp_lock = 00000000; cpuid = 0
> Trace beginning at frame 0xda6f5ca0
> panic(da6f5cc4,da6f5cf4,2b063bc9,f,da6f5cc8) at panic+0x14d
> panic(c037181c,da6f5cf4,da6f5cdc,c01bdc9a,c041dfd0) at panic+0x14d
> netmsg_put_port(c041dfd0,da6f5cf4) at netmsg_put_port+0x3e
> lwkt_domsg(c041dfd0,da6f5cf4,0,0) at lwkt_domsg+0x4f
> udp_ctlinput(f,d73c9bf8,0,0,da6f5d68) at udp_ctlinput+0x151
> netmsg_pru_ctlinput(d73c9b1c,0,c0422664,da6f5d84,c0234082) at
> netmsg_pru_ctlinput+0x13
> netmsg_service(d73c9b1c,1,0,c03f4034,ff800000) at netmsg_service+0x44
> netmsg_service_loop(c0422664,0,0,0,0) at netmsg_service_loop+0x18
> lwkt_exit() at lwkt_exit
> boot() called on cpu#0
> Uptime: 15m44s
>
> (kgdb) bt
> #0  dumpsys () at ./machine/thread.h:83
> #1  0xc01b3c1d in boot (howto=260) at
> /usr/src/sys/kern/kern_shutdown.c:375
> #2  0xc01b3ee0 in panic (fmt=0xc037181c "netmsg_put_port:
> self-referential deadlock on netport") at
> /usr/src/sys/kern/kern_shutdown.c:800
> #3  0xc0234009 in netmsg_put_port (port=0xc041dfd0, lmsg=0xda6f5cf4) at
> /usr/src/sys/net/netisr.c:137
> #4  0xc01bdc9a in lwkt_domsg (port=0x0, msg=0xda6f5cf4, flags=0) at
> /usr/src/sys/sys/msgport2.h:52
> #5  0xc025321b in udp_ctlinput (cmd=15, sa=0xd73c9bf8, vip=0x0) at
> /usr/src/sys/netinet/udp_usrreq.c:700
> #6  0xc01e5bad in netmsg_pru_ctlinput (msg=0xd73c9b1c) at
> /usr/src/sys/kern/uipc_msg.c:648
> #7  0xc0233ca8 in netmsg_service (msg=0xd73c9b1c, mpsafe_mode=1,
> mplocked=0) at /usr/src/sys/net/netisr.c:286
> #8  0xc0234082 in netmsg_service_loop (arg=0xc0422664) at
> /usr/src/sys/net/netisr.c:333
> #9  0xc01bafee in lwkt_deschedule_self (td=Cannot access memory at
> address 0x8
> ) at /usr/src/sys/kern/lwkt_thread.c:228
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> kernel and vmcore are being uploaded to leaf. (*.22)

Please test this one:
http://leaf.dragonflybsd.org/~sephe/tcp_udp_ctlinput.diff1

Best Regards,
sephe

-- 
Live Free or Die