[milters] Archive

Lists Index Date Thread Search

Article: 1281
From: Anthony Howe
Date: 2006-11-23 04:50:21 -0500
Subject: Re: Milter-link & size of VM

Removal...........: milters-request@milter.info?subject=remove
More information..: http://www.milter.info/#Support

Quentin Campbell wrote:
> I have been watching milter-link using 'top' on a number of our mail
> gateways and was struck by its behaviour:
> 1. On a single machine its VM requirement can apparently jump or fall by
> 50+ MBs during a 3 second interval,
> 2. but its resident size (working set?) is very small and
> 3. its average VM size varies enormously between gateways, 300+ MB on
> one gateway and 600+ MB on another and growing. I am running
> milter-link/0.3.16 on all the gateways.
> I am curious about its architecture and use of memory for its work.

sendmail's libmilter handles the creation and control of each thread. 
The default thread stack on most systems is 1MB per thread. Each SMTP 
connection into sendmail (one child process) will result in one thread 
in a milter. Currently libmilter provides no API control of thread stack 

Sendmail passes the message content to a milter in 64KB chunks.
milter-link uses no temporary files by doing on the fly in memory MIME 
parsing and decoding. However, this only requires at most 4KB for the 
structures and buffers. The DNS request might requires 2KB or so for 
their work. Berkeley DB memory requirements are unknown to me as it 
manages its own in-memory cache for lookups. If its not the Berkeley 
library, then I have no idea as to what might cause such significant 
jumps in VSZ.

You could save a message and try testing with the uri CLI in isolation 
from the milter. See com/snert/lib/util/uri for usage. For example

./uri -p -q -d multi.surbl.org,black.uribl.com,sbl-xbl.spamhaus.org \
       -f msgfile

You could try the supplied test file:

cd com/snert/lib/util
./uri -p -q -d multi.surbl.org,black.uribl.com,sbl-xbl.spamhaus.org \
       -f uri.txt

> I have been watching milter-link's behaviour because of an apparently
> intermittent problem with whitelisting on the seven mail gateways that
> we run. I am relying on the whitelisting by milter-link of mail from
> machines on our networks (1 x class B and 1 x class A) but the
> whitelisting seems to fail sometimes. 
> I have not been able to recreate the problem yet using small test
> messages containing the offending URLs. Nor have I been able yet to
> catch this in production with "verbose=info,trace,db" as I have to be

If you want to watch URI parsing and lookups add "debug" to the verbose 
option list.

> selective about the period during which I run with this level of
> debugging otherwise our very large Sendmail log files become even more
> unmanageable. 
> I use in the access file on each gateway:
> Connect:               RELAY
> Connect:10                      RELAY
> Connect:127                     RELAY
> Connect:128.240                 RELAY
> milter-link-Connect:10          OK
> milter-link-Connect:128.240     OK
> to affect whitelisting for mail from local hosts on our 128.240.*.* and
> the 10.*.*.* networks. It normally works as expected. Those milter-link
> entries cannot be "trumped" by other lookups as these have the highest
> precedence in lookup order.

Actually no. Typically, unless otherwise documented, the precedence 
tends to reflect the lookup order from low to high SMTP states: 
connection, MAIL, AUTH, RCPT; (AUTH will trump RCPT if matched or it 
will in the next release from what I see in my code). Within each SMTP 
state, the tag lookup order has its own high-to-low precedence starting 
with milter-NAME-tag: down to an untagged entry, depending on the SMTP 

Also in milter-link's case it also has milter-link-body tag lookups that 
can reject on or ignore specified domains from lookups.

> In relation to this, having whitelisted a message by virtue of a
> 'milter-link-Connect:...' entry that returned 'value="OK"' why does
> milter-link then proceed to do 'milter-link-To:' lookups for recipient
> addresses for the _same_ message, often with the result "default action
> SKIP"?

milter-link still looks for superior B/W entries. A SKIP though is 
neither black nor white, ie. continue processing. The milter handles 
white listing by skipping processing in filterBody() and 
filterEndMessage(); look for statements like:

	if (data->work.skipMessage) {

> I would have thought that once a message is whitelisted by a
> 'milter-link-Connect:client-IP' entry there is no point in doing any
> lower precedence 'milter-link-*' lookups.

Connect is low precedence, recipient is high precedence.

Anthony C Howe          Skype: SirWumpus                    SnertSoft
+33 6 11 89 73 78         AIM: SirWumpus    Sendmail Milter Solutions
http://www.snert.com/     ICQ: 7116561

Lists Index Date Thread Search