[milters] Archive

Lists Index Date Thread Search

Article: 1365
From: Quentin Campbell
Date: 2006-12-12 04:10:50 -0500
Subject: Re: Line lengths & milter-link

Removal...........: milters-request@milter.info?subject=remove
More information..: http://www.milter.info/#Support


Getting some useful debug info is proving to be the most frustrating

This morning I restarted 7 of the 8 mail gateways with
"verbose=info,trace,db". I had inadvertently left out the eigth. It
turns out that the only milter-link failure this morning was on that
eighth gateway so I have no debug info again. 

It appears that restarting sendmail & milter-link may mask the problem,
at least for 24 hours or so. For example I cannot recreate the problem
immediately after a restart by sending test messages with a known listed
URL. I have decided to leave all the mail gateways running with
""verbose=info,trace,db" for 24 hours in the hope this will catch the
problems when tomorrow's batch of proxy log messages get sent in the

The messages causing the problem are extracts from web proxy log files.
These extracts are carried in-line as the message body. A listed URL may
appear in more than one record and as yet I have not been able to
isolate a particular URL/record from a rejected message.

I will continue investigating this week but from 15th Dec until 3rd Jan
I will be away from work and will pick up again on the investigation
from that latter date.

It seems that I can avoid the problem by adding some additional
whitelisting entries in access.db. The original 'milter-link-Connect'
entries that I hoped would do the job are:

milter-link-Connect:10          OK
milter-link-Connect:128.240     OK

These cover our two internal network ranges (a class A and a class B

As the traffic I am having problems with is sent every day from the same
sender (via a redirection from an Exchange 2003 server) to the same
recipient I found that by whitelisting sender and recipient as well, the
messages always seem to get through OK. The additional access.db entries
that achieve this are:

# Deal with the problems TJR has with proxy log messages
milter-link-To:nwarn@burnmoor.ncl.ac.uk OK
milter-link-From:root@logproc.ncl.ac.uk OK
# Belt-n-braces
milter-link-body:sextracker.com OK   

The last line is the parent domain of some of the URLs that regularly
get caught; eg 'counter1.sextracker.com' (by .multi.surbl.org). 

All the mail gateways are RH AS4 systems. 'makemap' uses libdb-4.2.so.
'sendmail' also uses libdb-4.2.so. Ditto 'milter-link'. Doing a
'strings' on the last I get:

[root@cheviot4 mail]# strings /usr/local/sbin/milter-link | grep db
db_create() error: %s
Use stat() instead of fstat() to monitor .db file updates; experimental.
# Postfix with postmap(1) generated .db files; experimental.
Enable debugging of smdb routines.
mutex init in smdbOpen() failed: %s (%d)
mutex unlock in smdbGetValueInternal() failed: %s (%d)
sendmail db get "%s" error: %s
mutex lock in smdbGetValueInternal() failed: %s (%d)
Path to the access.db file.

I will try to isolate specific URLs/records if I can.

Thanks again for your help.


>-----Original Message-----
>From: milters-bounce@milter.info 
>[mailto:milters-bounce@milter.info] On Behalf Of Anthony Howe
>Sent: 09 December 2006 09:08
>To: milters@milter.info
>Subject: [milters] Re: Line lengths & milter-link
>Removal...........: milters-request@milter.info?subject=remove
>More information..: http://www.milter.info/#Support
>Quentin Campbell wrote:
>> Most of the time it correctly whitelists messages from hosts on our
>> internal network but sometimes it fails to. Thus a message with a
>> particular URL is whitelisted one day but the next day that 
>URL causes
>> the message to be rejected.
>Q: What is the complete URL?
>> I am desperately casting around for possible explanations for this
>> behaviour! The messages in question contain records 
>extracted from web
>> proxy logs. The record length can thus exceed 1k. Could the length of
>> records in a message cause milter-link to ignore 
>whitelisting entries in
>> access.db? 
>Assuming the access-db is configured and read correctly; I'm 
>not saying 
>it's not the source of the problem, just I'm doing a walk through here.
>In filterOpen() the smfAccessHost() will set true the flag 
>data->work.skipConnection if the client connection by IP or domain of 
>the reverse DNS is found white-listed (OK or RELAY) in the access map. 
>The only way override that would be to specify a negative REJECT 
>response for a sender, AUTH, or recipient.
>Q: Is your white listing specified by IP or domain?
>In filterMail(), this flag is carried over to 
>data->work.skipMessage. A 
>  A REJECT found by smfAccessMail() or smfAccessAuth() would result in 
>an immediate rejection with the message "sender blocked". 
>Similarly for 
>smfAccessRcpt() in filterRcpt().
>filterBody() and filterEndMessage() at the top of their 
>functions test 	
>	if (data->work.skipMessage) {
>and by pass the processing if it's true. I think the logic is correct. 
>In order perform URI processing we would have had to NOT find the 
>necessary access.db entries. So we have to challenge the 
>assumption that 
>access-db is not properly configured or a bug lies here.
>Q: Refresh my memory as to OS and version of Berkeley DB being used by 
>sendmail, makemap, and the milter?
>Your 2nd question concerning line record lengths.
>Q: Can you provide one or two records that trigger milter-link?
>In libsnert, uri.h specifies a 1024 byte hold buffer in which 
>a possible 
>URL is built up from input. RFC 2821 specifies that SMTP DATA 
>text line 
>lengths must not exceed 1000 bytes including the CRLF (use 
>quoted-printable soft-line breaks otherwise), but I doubt that 
>would be 
>an issue since its more than likely that the line record will contain 
>spaces and and other invalid URL characters, such that the hold buffer 
>would be reset several times while reading the input from a 
>line. Now on 
>the off chance that you actually have a URL greater than 1024 bytes, 
>then mimeGetUri() flushs the hold buffer and starts a new as noted in 
>the comments:
>	/* If the hold buffer is full, just dump it. The
>	 * buffer is larger that any _normal_ URL should
>	 * be and its assumed it would fail to parse.
>	 */
>You could try doubling the hold buffer size, but I think this 
>is not the 
>cause of your 1st problem with respect to white listing. Whether there 
>is a line length bug here I would need samples to test with.
>Anthony C Howe          Skype: SirWumpus                    SnertSoft
>+33 6 11 89 73 78         AIM: SirWumpus    Sendmail Milter Solutions
>http://www.snert.com/     ICQ:
7116561      http://www.snertsoft.com/

Lists Index Date Thread Search