[DragonFlyBSD - Bug #3032] (Resolved) IPFW3: memory leakage? objcache(xxx): Exhausted!

bugtracker-admin at leaf.dragonflybsd.org bugtracker-admin at leaf.dragonflybsd.org
Sun Jun 11 08:16:18 PDT 2017

Issue #3032 has been updated by liweitianux.

Status changed from New to Resolved

There is a memory leakage bug with IPFW3 sync feature, which keeps consuming the mbuf, and finally causes this "objcache(xxx): Exhausted" issue.

The bug has been fixed by bycn82@ in commit 2032f72fa4bc9355fef4594376299c9aeb707530 on 2017-06-10.  I already did some tests and things are OK.

Generally speaking, the IPFW3 "sync" shouldn't be used (which is also useless) on a single isolated host.  But IPFW3 originally kept complaining that "ipfw3sync: centre not running", so I hacked to get rid of these annoying messages by "ipfw3 sync start centre".  Now, this complaining issue has also been fixed.

Bug #3032: IPFW3: memory leakage? objcache(xxx): Exhausted!

* Author: liweitianux
* Status: Resolved
* Priority: High
* Assignee: 
* Category: Networking
* Target version: 
Recently, I setup a small VPS (512MB RAM) with DFly(v4.8)+Nginx+Postfix+Dovecot services as my personal email server, and configured IPFW3 as the firewall.  However, the system hang/freeze with these warnings after some time, and can only be reset:

Warning, objcache(mbuf pkt hdr): Exhausted!
Warning, objcache(mbuf pkt hdr + cluster): Exhausted!

The VPS can run rather smoothly without IPFW3 enabled.  So I tried to monitor the mbuf usages reported by "netstat -m", and found the "mbufs in use" was continuously increasing, and therefore, the mbufs are exhausted finally which freeze the whole system!!


In addition, I tried to reproduce the IPFW3 problem in my home NAS (DFly master at 2017-02-24, 8GB RAM, Nginx+ownCloud+PostgreSQL+transmission bt), and the system just paniced with following messages:

stack pointer           = 0x10:0xffffff81db763720
frame pointer           = 0x10:0xffffff81db763770
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = Idle
current thread          = pri 12
trap number             = 9
panic: general protection fault
cpuid = 2
Trace beginning at frame 0xffffff81db7634e8
panic() at panic+0x236 0xffffffff805e5d56
panic() at panic+0x236 0xffffffff805e5d56
trap_fatal() at trap_fatal+0x2c3 0xffffffff80a58203
trap() at trap+0x19a 0xffffffff80a58a8a
calltrap() at calltrap+0x9 0xffffffff80a4125f
--- trap 0000000000000009, rip = ffffffff84840dd4, rsp = ffffff81db763710, rbp = ffffff81db763770 ---
ipfw_sync_send_state() at ipfw_sync_send_state+0x54 0xffffffff84840dd4
check_keep_state() at check_keep_state+0x196 0xffffffff84821d76
boot() called on cpu#2
Uptime: 12d1h18m34s
Physical memory: 8113 MB
Dumping 3889 MB:Warning, objcache(cluster mbuf): Exhausted!

For unknown reason, I failed to get a core dump.  I set "debug.debugger_on_panic=0" and have a 16GB swap partition (dm_crypt).

An example "netstat -m" report looks like:
202336/293376 mbufs in use (current/max):
522/17248 mbuf clusters in use (current/max)
0/8368 mbuf jumbo clusters in use (current/max)
        202839 mbufs and mbuf clusters allocated to data
        19 mbufs and mbuf clusters allocated to packet headers
102212 Kbytes allocated to network (56% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

With IPFW3 enabled, the above "mbufs in use" keeps increasing!  Flushing the firewall rules does not help either.
I also tried to *disable* the IPFW3 by setting "net.inet.ip.fw3.enable=1", and the mbuf usages *stop increase* (but don't decrease either).

Therefore, I suspect there may be *memory leakages* within IPFW3??

What's additional information can I provide to help solve this problem?


P.S., my IPFW3 rules (example output of "ipfw3 show"):
00010        0           0  allow via lo0
00100 14446903 15349529336  check-state
00200   168513   106435518  allow tcp from me out via em0 keep-state
00201  5853336  6205262652  allow udp from me out via em0 keep-state
00202        1          56  allow icmp from me out via em0 keep-state
00301        3         168  deny from in via em0
00303        0           0  deny from in via em0
00304        3         984  deny from in via em0
00305        1         328  deny from in via em0
00306        0           0  deny from in via em0
00307        0           0  deny from in via em0
00308        0           0  deny from in via em0
00310     2012      239927  allow icmp in via em0 keep-state
00315        0           0  deny tcp dst-port 113 in via em0
00320        0           0  deny tcp dst-port 137 in via em0
00321        0           0  deny tcp dst-port 138 in via em0
00322        0           0  deny tcp dst-port 139 in via em0
00323        0           0  deny tcp dst-port 81 in via em0
00332     7033     1814028  deny tcp established in via em0
00500       68       11257  allow tcp dst-port 8860 in via em0 keep-state
00510        0           0  allow tcp dst-port 80 in via em0 keep-state
00512        0           0  allow tcp dst-port 8800 in via em0 keep-state
00513     9281     3765242  allow tcp dst-port 8801 in via em0 keep-state
00700        0           0  allow tcp dst-port 22000 in via em0 keep-state
00701      737       86229  allow udp dst-port 21027 in via em0 keep-state
00800  8563201  9051221125  allow dst-port 51413 in via em0 keep-state
00801      515      263178  allow tcp from dst-port 9091 in via em0 keep-state
60000     1253      185397  deny in via em0
65535      414       59978  deny

You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: http://bugs.dragonflybsd.org/my/account

More information about the Bugs mailing list