[BarricadeMX] Archive

Lists Index Date Thread Search

Article: 25
From: Steve Freegard
Date: 2010-03-04 19:13:59 -0500
Subject: Re: Re: BMX log searching

On 04/03/10 23:39, Anthony Howe wrote:
> On 04/03/2010 23:33, Mike Bacher whispered from the shadows...:
>> On the search logs page, what does "Use index if available" mean?  If
>> there is something I need to do to "index" my logs, let me know.

We used to have a Perl job that indexed the log files to speed up the 
searching; but found that the indexing job itself placed considerable 
load on the machine when generating the indexes each day - so this was 
dropped in later versions.  The index functions in the web interface 
will disappear eventually.

>> Also, it would be nice if the search page would include the sendmail
>> lines in the "grep" search parameters, so you can see not only the
>> entries, but the corresponding sendmail lines as well...
> Note that BarricadeMX+ and the BarricadeMX web user interface (WUI)
> issues should go directly to FSL, though Steve Freegard of FSL does
> follow this list. This list focuses on the core engine smtpf aka
> BarricadeMX aka BMX, at least that was my intent when I created the
> list. FSL staff are welcome to comment on their areas.
> However, I think merging sendmail related lines may not be practical.
> Also note that BMX works with all manner of MTAs, which may or may not
> be local to the machine.
> Consider BMX on one machine forwarding mail directly to an Exchange
> machine, Postfix, Exim, etc. BMX also can forward to more than one
> MTA/machine at once. So to collect all the related logs from other
> machines would be next to impossible in real time.
> Even if the MTA is local to BMX one can't assume that the MTA is
> sendmail nor can BMX keep track of all possible MTA logging formats.
> Remember too that the log gathering is done in the WUI, not in the smtpf
> engine, so there are some time constraints as to how long a web browser
> will wait for an answer from the web server, typically 30 seconds.

The other problem is being able to 'link' log-lines together between 
smtpf and the MTA in use without having to scan the log file twice and 
incurring a big time penalty.

If you want something with a lot of features and speed - then take a 
look at Splunk which was specifically designed for this job.

> Now BMX+, which is the combination of sendmail, mailscanner, and
> mailwatch, etc., knows its components, so something might be possible
> there. Steve Freegard would have to answer that since that's his baby.

BMX+ puts a limited amount of log data into the PostgreSQL database 
which contains indexes on the connecting IP, full sender address or 
sender domain and allows searching on those items.  Anything else 
appears in MailWatch which also shows where the message was relayed 
(e.g. from Sendmail).


Lists Index Date Thread Search