[DragonFlyBSD - Bug #3032] (New) IPFW3: memory leakage? objcache(xxx): Exhausted!

Sepherosa Ziehau sepherosa at gmail.com
Thu May 4 18:16:25 PDT 2017


On Thu, May 4, 2017 at 7:44 PM, Aaron LI <aaronly.me at outlook.com> wrote:
> Dear Bill,
>
> I'm running into problems with the IPFW3, which seems to cause memory
> leakages and lead to the "objcache(xxx): Exhausted!" warning, and
> finally the system became unresponsive and need reset.
>
> On the same VPS with IPFW3 disabled, it runs rather smoothly (already a
> week now since last reset); as for my other machine at home, it had very
> good uptime (>100 days).
>
> Therefore, could you please have a look at my bug report #3032 (detailed
> as below) when it is convenient for you?  Thanks!
>
> ----------------------------------------
>
> On the other hand, any other Dflyers ever came across such problems?

Since you are not using NAT etc, could you try ipfw?

>
>
> Cheers,
> --
> Aly
>
>
> On Mon, May 01, 2017 at 03:37, bugtracker-admin at leaf.dragonflybsd.org wrote:
>
>> Issue #3032 has been reported by liweitianux.
>>
>> ----------------------------------------
>> Bug #3032: IPFW3: memory leakage? objcache(xxx): Exhausted!
>> http://bugs.dragonflybsd.org/issues/3032
>>
>> * Author: liweitianux
>> * Status: New
>> * Priority: High
>> * Assignee:
>> * Category: Networking
>> * Target version:
>> ----------------------------------------
>> Recently, I setup a small VPS (512MB RAM) with DFly(v4.8)+Nginx+Postfix+Dovecot services as my personal email server, and configured IPFW3 as the firewall.  However, the system hang/freeze with these warnings after some time, and can only be reset:
>>
>> ------------------------------------------------------
>> Warning, objcache(mbuf pkt hdr): Exhausted!
>> Warning, objcache(mbuf pkt hdr + cluster): Exhausted!
>> ------------------------------------------------------
>>
>> The VPS can run rather smoothly without IPFW3 enabled.  So I tried to monitor the mbuf usages reported by "netstat -m", and found the "mbufs in use" was continuously increasing, and therefore, the mbufs are exhausted finally which freeze the whole system!!
>>
>> ======================================================
>>
>> In addition, I tried to reproduce the IPFW3 problem in my home NAS (DFly master at 2017-02-24, 8GB RAM, Nginx+ownCloud+PostgreSQL+transmission bt), and the system just paniced with following messages:
>>
>> ------------------------------------------------------
>> stack pointer           = 0x10:0xffffff81db763720
>> frame pointer           = 0x10:0xffffff81db763770
>> code segment            = base 0x0, limit 0xfffff, type 0x1b
>>                         = DPL 0, pres 1, long 1, def32 0, gran 1
>> processor eflags        = interrupt enabled, resume, IOPL = 0
>> current process         = Idle
>> current thread          = pri 12
>> trap number             = 9
>> panic: general protection fault
>> cpuid = 2
>> Trace beginning at frame 0xffffff81db7634e8
>> panic() at panic+0x236 0xffffffff805e5d56
>> panic() at panic+0x236 0xffffffff805e5d56
>> trap_fatal() at trap_fatal+0x2c3 0xffffffff80a58203
>> trap() at trap+0x19a 0xffffffff80a58a8a
>> calltrap() at calltrap+0x9 0xffffffff80a4125f
>> --- trap 0000000000000009, rip = ffffffff84840dd4, rsp = ffffff81db763710, rbp = ffffff81db763770 ---
>> ipfw_sync_send_state() at ipfw_sync_send_state+0x54 0xffffffff84840dd4
>> check_keep_state() at check_keep_state+0x196 0xffffffff84821d76
>> boot() called on cpu#2
>> Uptime: 12d1h18m34s
>> Physical memory: 8113 MB
>> Dumping 3889 MB:Warning, objcache(cluster mbuf): Exhausted!
>> ------------------------------------------------------
>>
>> For unknown reason, I failed to get a core dump.  I set "debug.debugger_on_panic=0" and have a 16GB swap partition (dm_crypt).
>>
>> An example "netstat -m" report looks like:
>> ------------------------------------------------------
>> 202336/293376 mbufs in use (current/max):
>> 522/17248 mbuf clusters in use (current/max)
>> 0/8368 mbuf jumbo clusters in use (current/max)
>>         202839 mbufs and mbuf clusters allocated to data
>>         19 mbufs and mbuf clusters allocated to packet headers
>> 102212 Kbytes allocated to network (56% of mb_map in use)
>> 0 requests for memory denied
>> 0 requests for memory delayed
>> 0 calls to protocol drain routines
>> ------------------------------------------------------
>>
>> With IPFW3 enabled, the above "mbufs in use" keeps increasing!  Flushing the firewall rules does not help either.
>> I also tried to *disable* the IPFW3 by setting "net.inet.ip.fw3.enable=1", and the mbuf usages *stop increase* (but don't decrease either).
>>
>> Therefore, I suspect there may be *memory leakages* within IPFW3??
>>
>> What's additional information can I provide to help solve this problem?
>>
>>
>> Cheers,
>> Aly
>>
>>
>> P.S., my IPFW3 rules (example output of "ipfw3 show"):
>> ------------------------------------------------------
>> 00010        0           0  allow via lo0
>> 00100 14446903 15349529336  check-state
>> 00200   168513   106435518  allow tcp from me out via em0 keep-state
>> 00201  5853336  6205262652  allow udp from me out via em0 keep-state
>> 00202        1          56  allow icmp from me out via em0 keep-state
>> 00301        3         168  deny from 172.16.0.0/12 in via em0
>> 00303        0           0  deny from 127.0.0.0/8 in via em0
>> 00304        3         984  deny from 0.0.0.0/8 in via em0
>> 00305        1         328  deny from 169.254.0.0/16 in via em0
>> 00306        0           0  deny from 192.0.2.0/24 in via em0
>> 00307        0           0  deny from 204.152.64.0/23 in via em0
>> 00308        0           0  deny from 224.0.0.0/3 in via em0
>> 00310     2012      239927  allow icmp in via em0 keep-state
>> 00315        0           0  deny tcp dst-port 113 in via em0
>> 00320        0           0  deny tcp dst-port 137 in via em0
>> 00321        0           0  deny tcp dst-port 138 in via em0
>> 00322        0           0  deny tcp dst-port 139 in via em0
>> 00323        0           0  deny tcp dst-port 81 in via em0
>> 00332     7033     1814028  deny tcp established in via em0
>> 00500       68       11257  allow tcp dst-port 8860 in via em0 keep-state
>> 00510        0           0  allow tcp dst-port 80 in via em0 keep-state
>> 00512        0           0  allow tcp dst-port 8800 in via em0 keep-state
>> 00513     9281     3765242  allow tcp dst-port 8801 in via em0 keep-state
>> 00700        0           0  allow tcp dst-port 22000 in via em0 keep-state
>> 00701      737       86229  allow udp dst-port 21027 in via em0 keep-state
>> 00800  8563201  9051221125  allow dst-port 51413 in via em0 keep-state
>> 00801      515      263178  allow tcp from 192.168.1.0/24 dst-port 9091 in via em0 keep-state
>> 60000     1253      185397  deny in via em0
>> 65535      414       59978  deny
>> ------------------------------------------------------
>
>
> --
> Aly



-- 
Tomorrow Will Never Die



More information about the Users mailing list