PLA sylog configuration
PLA (PIX Logging Architecture) uses regular expressions (regex) to parse syslog messages received from Cisco firewalls and comes pre-configured to process the standard “syslogd” message format. Most current Linux distributions ship with rsyslog (able to log directly to MySQL) while some administrators prefer syslog-ng, any standard syslog daemon will work.
The PLA installation documentation assumes familiarity regex, so here you’ll see how to tweak PLA to parse your syslog log file.
Understanding the mechanics
Perl is actually used to tail the syslog log file looking for matches to messages described in the syslog_messages table, which are in turn loaded into the MySQL database. The processing script pla_parsed contains a regex filter that must be matched in order for the processing to occur correctly. Here, the applicable section from the PLA Documentation highlights the portion of code from the pla_parsed script we are concerned with:
### PIX Firewall Logs
### DEFINE YOUR SYSLOG LAYOUT HERE!
###$regex_log_begin = “(.*):(.*) (.*) (.*) (.*) (.*) (.*)“;
$var_pixhost=3;
$var_pixmonth=4;
$var_pixdate=5;
$var_pixyear=6;
$var_pixtime=7;
The variable regex_log_begin needs to match up all the log information up to the actual PIX, ASA, or FWSM message code in order to correlate date, time, and host for each message added to the syslog message file. Take a look at the provided sample log entry from the PLA Documentation, everything in red needs to be picked up by regex_log_begin, while the remainder is standard for Cisco firewalls and already handled by pla_parsed:
Oct 21 23:59:23 fwext-dmz-01 Oct 21 2006 23:58:23: %PIX-6-305011: Built dynamic TCP translation from inside:1.1.1.1/2244 to outside:2.2.2.2/3387
Harnessing regex
Explaining the operation of regex and wildcards is beyond the scope of this article; however, we have created a simple perl script that can be used to determine whether your syslog formating is supported by default or if changes are necessary. The pla_parser_regex_test script will let you insert new regex patterns and show you how they match up against your syslog messages. The file must be saved to your pla_parsing server and accessed in one of the following manners:
- perl pla_parser_regex_test.txt or
- execute the file directly ./pla_parser_regex_test.txt (after running chmod +x)
If you still need assistance with regex, numerous guides have been written to make regex understandable but remember: we’re working with the basics of regex here - nothing fancy. In our case, adjusting the default regex to match most any syslog format is now straight forward. After noting which characters match which pattern, we can determine what changes are needed by viewing the pla_parser_regex_test script’s output:
Running perl pla_parser_regex_test.txt the first time should give you:
You supplied the following log message:
Feb 21 10:59:32 Feb 21 2008 10:59:32 pix4us
Resulting in these matches using a default regex filter of:
(.*):(.*) (.*) (.*) (.*) (.*) (.*)
_________________________________________________________
]> match 1: Feb 21 10:59
]> match 2: 32
]> match 3: Feb (default pixhost)
]> match 4: 21 (default pixmonth)
]> match 5: 2008 (default pixdate)
]> match 6: 10:59:32 (default pixyear)
]> match 7: pix4us (default pixtime)
_________________________________________________________Verify the log string you provided is correct Adjust your
regex_log_begin or var_pix settings accordingly!If you have more than one substitution for each match, verify
the log message and the regex_log_begin filter string, you may
need to change something to get PLA runningIf the default settings match up, no changes are needed.
Just run pla_parsed to start PLA
Log parsing, traditionally, has been a problem for some trying to implement PIX Logging Architecture. The pla_parser_regex_test script is provided to help ease syslog-related installation problems. As such, pla_parser_regex_test will immediate provide feedback with each regex change and map out the deviations from the defaul PLA settings. I’m no perl programmer, so there is some manual intervention required to load your regex changes but it is very straight forward - so we’re not responsible for any problems created in the downloading, running, or editting of the pla_parser_regex_test script.
On to rsyslogd
Take this sample rsyslog entry and notice the difference from the standard syslogd format:
Feb 21 10:59:32 Feb 21 2008 10:59:32 pix4us : %PIX-6-110001: No route to 1.1.1.1 from 3.4.5.6
Feb 21 10:59:32 Feb 21 2008 10:59:32 pix4us (This is the relevant portion of the log.)
Oct 21 23:59:23 fwext-dmz-01 Oct 21 2006 23:58:23 (Note the hostname moved.)
The rsyslog entry includes the date twice followed by the hostname of the log source versus the default format expected by pla_parsed of date hostname date. The original regex is set to pickup the first time entry’s “minutes and seconds” and picks up the next 5 words/entries separated by spaces by using wildcards (.*). Each wilcard match is then translated into the necessary variables for pla_parsed:
$regex_log_begin = “(.*):(.*) (.*) (.*) (.*) (.*) (.*)“;
Oct 21 23:59:23 fwext-dmz-01 Oct 21 2006 23:58:23
As you experiment with different filters for the regex variable, be aware you may need to change the numerical references as well. In english, as you change the regex for different syslog formats, realize that the colors in the examples above may not line up correctly.
In order to process rsyslog, the pla_parsed must take into account the placement of the date/time. The initial (.*):(.*) is used to set a starting point in syslog message string. Since this new rsyslog format includes two date entries before the host name, the following can be used to allow pla_parsed to “see” the new syslog message string:
$regex_log_begin = “(.*):(.*) (.*) (.*) (.*) (.*) (.*)“;
Feb 21 10:59:32 Feb 21 2008 10:59:32 pix4us
The regex starts out the same, but looking at the colors you will notice the location of the information needed by pla_parsed to determine date, time, and host has moved. This time the 7th wildcard will have to be setup to match the $var_pixhost variable and the other $var_’s will “slide up” one position.
As a result of using this new log message format, the variables listed below the regex pattern must be modified to tell pla_parsed which (.*) contains which element. Once again the colors below will illustrate which pattern matches which element in the syslog message:
$regex_log_begin = “(.*):(.*) (.*) (.*) (.*) (.*) (.*)“;
$var_pixhost=7;
$var_pixmonth=3;
$var_pixdate=4;
$var_pixyear=5;
$var_pixtime=6;
The numbering happens left to right and the color coding should help this make sense. This same approach of keying off the timestamping can be applied to pla_parsed in order to allow processing of syslog-ng, ksyslogd, or any other syslog message format.
The pla_parser_regex_test will illustrate the same output, without the colors:
Here the $log variable inside the script has been updated with the new example:
You supplied the following log message:
Feb 21 10:59:32 Feb 21 2008 10:59:32 pix4us
Resulting in these matches using a default regex filter of:
(.*):(.*) (.*) (.*) (.*) (.*) (.*)
_________________________________________________________
]> match 1: Feb 21 10:59
]> match 2: 32
]> match 3: Feb (default pixhost)
]> match 4: 21 (default pixmonth)
]> match 5: 2008 (default pixdate)
]> match 6: 10:59:32 (default pixyear)
]> match 7: pix4us (default pixtime)
_________________________________________________________Verify the log string you provided is correct Adjust your
regex_log_begin or var_pix settings accordingly!If you have more than one substitution for each match, verify
the log message and the regex_log_begin filter string, you may
need to change something to get PLA runningIf the default settings match up, no changes are needed.
Just run pla_parsed to start PLA
Notice the output shows the matches don’t lineup with the default variables; however, the match number can be used to modify or verify the $var_ parameters.
An alternative method to match the rsyslog output involves using “(.*):(.*)” and “((.*):(.*):(.*))” to force a match on the time elements as seem below. Once again, the color coding should help clarify what is being done in the regex filter:
Sample: Feb 21 10:59:32 Feb 21 2008 10:59:32 pix4us
$regex_log_begin = “(.*):(.*) (.*) (.*) (.*) ((.*):(.*):(.*)) (.*)“;
$var_pixhost=7;
$var_pixmonth=3;
$var_pixdate=4;
$var_pixyear=5;
$var_pixtime=6;
The numbering happens left to right and the color coding should help this make sense. The ()’s around the grey time entry are grouped together and count as one match/entity, the sixth variable. The hostname is still detected properly as the last match.
Don’t forget syslog-ng!
Now that rsyslogd is now working, take a look at syslog-ng formatted with ISODATE (the log is pre-color-coded this time):
2008-02-29T12:43:16-05:00 pix-test Feb 29 2008 12:43:16: %PIX-6-110001: No route to 1.1.1.1 from 3.4.5.6
Notice the initial date/time stamp (ISODATE) is essentially one long phrase without spaces, regex can see that as a single match of (.*). syslog-ng has many options to format the syslog-ng output, not all include the timezone offset as part of the date/time stamp. One way to parse this initial message via regex is to use the following format:
$regex_log_begin = “(.*) (.*) (.*) (.*) (.*) (.*)“;
Remeber the portion of the syslog message being parsed is a subset:
2008-02-29T12:43:16-05:00 pix-test Feb 29 2008 12:43:16
The $var_ sequence will need to be adjusted to reflect these changes:
$var_pixhost=2;
$var_pixmonth=3;
$var_pixdate=4;
$var_pixyear=5;
$var_pixtime=6;
If your syslog-ng format is output using FULLDATE, this same entry would look like this:
2008 02 29 12:43:16 pix-test Feb 29 2008 12:43:16
This time, let’s use the pla_parser_regex_test to see what needs to be done:
You supplied the following log message:
2008 02 29 12:43:16 pix-test Feb 29 2008 12:43:16
Resulting in these matches using a default regex filter of:
(.*):(.*) (.*) (.*) (.*) (.*) (.*)
_________________________________________________________]> match 1: 2008 02 29 12:43
]> match 2: 16
]> match 3: pix-test (default pixhost)
]> match 4: Feb (default pixmonth)
]> match 5: 29 (default pixdate)
]> match 6: 2008 (default pixyear)
]> match 7: 12:43:16 (default pixtime)
_________________________________________________________Verify the log string you provided is correct Adjust your
regex_log_begin or var_pix settings accordingly!If you have more than one substitution for each match, verify
the log message and the regex_log_begin filter string, you may
need to change something to get PLA runningIf the default settings match up, no changes are needed.
Just run pla_parsed to start PLA
Each variable lines up, so the default regex and variable settings included with pla_parsed will correctly handle this format when used in syslog-ng.
Need help with a different format?
Have problems getting your PIX logs loaded? Paste in a sample message from your syslog server (IP Addresses santized please) in a comment below.
