Sec (1) Linux Manual Page

NAME

sec – simple event correlator

SYNOPSIS

sec: [–conf=<file pattern> …]
[–input=<file pattern>[=<context>] …]
[–input-timeout=<input timeout>]
[–timeout-script=<timeout script>]
[–reopen-timeout=<reopen timeout>]
[–check-timeout=<check timeout>]
[–poll-timeout=<poll timeout>]
[–socket-timeout=<socket timeout>]
[–blocksize=<io block size>]
[–bufsize=<input buffer size>]
[–evstoresize=<event store size>]
[–cleantime=<clean time>]
[–log=<logfile>]
[–syslog=<facility>]
[–debug=<debuglevel>]
[–pid=<pidfile>]
[–dump=<dumpfile>]
[–user=<username>]
[–group=<groupname> …]
[–umask=<mode>]
[–dumpfts | –nodumpfts]
[–quoting | –noquoting]
[–tail | –notail]
[–fromstart | –nofromstart]
[–detach | –nodetach]
[–jointbuf | –nojointbuf]
[–keepopen | –nokeepopen]
[–rwfifo | –norwfifo]
[–childterm | –nochildterm]
[–intevents | –nointevents]
[–intcontexts | –nointcontexts]
[–testonly | –notestonly]
[–help] [-?]
[–version]

DESCRIPTION

SEC is an event correlation tool for advanced event processing which can be harnessed for event log monitoring, for network and security management, for fraud detection, and for any other task which involves event correlation. Event correlation is a procedure where a stream of events is processed, in order to detect (and act on) certain event groups that occur within predefined time windows. Unlike many other event correlation products which are heavyweight solutions, SEC is a lightweight and platform-independent event correlator which runs as a single process. The user can start it as a daemon, employ it in shell pipelines, execute it interactively in a terminal, run many SEC processes simultaneously for different tasks, and use it in a wide variety of other ways.

SEC reads lines from files, named pipes, or standard input, matches the lines with patterns (regular expressions, Perl subroutines, etc.) for recognizing input events, and correlates events according to the rules in its configuration file(s). Rules are matched against input in the order they are given in the configuration file. If there are two or more configuration files, rule sequence from every file is matched against input (unless explicitly specified otherwise). SEC can produce output by executing external programs (e.g., snmptrap(1) or mail(1)), by writing to files, by sending data to TCP and UDP based servers, by calling precompiled Perl subroutines, etc.

SEC can be run in various ways. For example, the following command line starts it as a daemon, in order to monitor events appended to the /var/log/messages syslog file with rules from /etc/sec/syslog.rules:

/usr/bin/sec –detach –conf=/etc/sec/syslog.rules \
–input=/var/log/messages

Each time /var/log/messages is rotated, a new instance of /var/log/messages is opened and processed from the beginning. The following command line runs SEC in a shell pipeline, configuring it to process lines from standard input, and to exit when the /usr/bin/nc tool closes its standard output and exits:

/usr/bin/nc -l 8080 | /usr/bin/sec –notail –input=- \
–conf=/etc/sec/my.conf

Some SEC rules start event correlation operations, while other rules react immediately to input events or system clock. For example, suppose that SEC has been started with the following command line

/usr/bin/sec –conf=/etc/sec/sshd.rules –input=/var/log/secure

in order to monitor the /var/log/secure syslog file for sshd events. Also, suppose that the /etc/sec/sshd.rules configuration file contains the following rule for correlating SSH failed login syslog events:

type=SingleWithThreshold
ptype=RegExp
pattern=sshd\[\d+\]: Failed .+ for (\S+) from [\d.]+ port \d+ ssh2
desc=Three SSH login failures within 1m for user $1
action=pipe ‘%s’ /bin/mail -s ‘SSH login alert’ root [at] localhost
window=60
thresh=3

The pattern field of the rule defines the pattern for recognizing input events, while the ptype field defines its type (regular expression). Suppose that user risto fails to log in over SSH and the following message is logged to /var/log/secure:

Dec 16 16:24:59 myserver sshd[13685]: Failed password for risto from 10.12.2.5 port 41063 ssh2

This input message will match the regular expression pattern of the above rule, and the match variable $1 will be set to the string risto (see perlre(1) for details). After a match, SEC will evaluate the operation description string given with the desc field. This is done by substituting $1 with its current value which yields Three SSH login failures within 1m for user risto. SEC will then check if there already exists an event correlation operation identified with this string and triggered by the same rule. If the operation is not found, SEC will create a new operation for the user name risto, and the occurrence time of the input event will be recorded into the operation. Note that for event occurrence time SEC always uses the current time as returned by the time(2) system call, *not* the timestamp extracted from the event. Suppose that after 25 seconds, a similar SSH login failure event for the same user name is observed. In this case, a running operation will be found for the operation description string Three SSH login failures within 1m for user risto, and the occurrence time of the second event is recorded into the operation. If after 30 seconds a third event for the user name risto is observed, the operation has processed 3 events within 55 seconds. Since the threshold condition "3 events within 60 seconds" (as defined by the thresh and window fields) is now satisfied, SEC will execute the action defined with the action field — it will fork a command

/bin/mail -s ‘SSH login alert’ root [at] localhost

with a pipe connected to its standard input. Then, SEC writes the operation description string Three SSH login failures within 1m for user risto (held by the %s special variable) to the standard input of the command through the pipe. In other words, an e-mail warning is sent to the local root-user. Finally, since there are 5 seconds left until the end of the event correlation window, the operation will consume the following SSH login failure events for user risto without any further action, and finish after 5 seconds.

The above example illustrates that the desc field of a rule defines the scope of event correlation and influences the number of operations created by the rule. For example, if we set the desc field to Three SSH login failures within 1m, the root-user would be also alerted on 3 SSH login failure events for *different* users within 1 minute. In order to avoid clashes between operations started by different rules, operation ID contains not only the value set by the desc field, but also the rule file name and the rule number inside the file. For example, if the rule file /etc/sec/sshd.rules contains one rule

and the event

Dec 16 16:24:59 myserver sshd[13685]: Failed password for risto from 10.12.2.5 port 41063 ssh2

is the first matching event for the above rule, this event will trigger a new event correlation operation with the ID

/etc/sec/sshd.rules | 0 | Three SSH login failures within 1m for user risto

(0 is the number assigned to the first rule in the file, see EVENT CORRELATION OPERATIONS section for more information).

The following simple example demonstrates that event correlation schemes can be defined by combining several rules. In this example, two rules harness contexts and synthetic events for achieving their goal:

type=SingleWithThreshold
ptype=RegExp
pattern=sshd\[\d+\]: Failed .+ for (\S+) from [\d.]+ port \d+ ssh2
desc=Three SSH login failures within 1m for user $1
action=event 3_SSH_LOGIN_FAILURES_FOR_$1
window=60
thresh=3

type=EventGroup
ptype=RegExp
pattern=3_SSH_LOGIN_FAILURES_FOR_(\S+)
context=!USER_$1_COUNTED && !COUNTING_OFF
count=create USER_$1_COUNTED 60
desc=Repeated SSH login failures for 30 distinct users within 1m
action=pipe ‘%s’ /bin/mail -s ‘SSH login alert’ root [at] localhost; \
create COUNTING_OFF 3600
window=60
thresh=30

The first rule looks almost identical to the rule from the previous example, but its action field is different — after three SSH login failures have been observed for the same user name within one minute by an event correlation operation, the operation will emit the synthetic event 3_SSH_LOGIN_FAILURES_FOR_<username>. Although synthetic events are created by SEC, they are treated like regular events received from input sources and are matched against rules. The regular expression pattern of the second rule will match the 3_SSH_LOGIN_FAILURES_FOR_<username> event and start a new event correlation operation if no such events have been previously seen. Also, each time a synthetic event for some user name has matched the rule, a context with the lifetime of 1 minute for that user name is created (see the count field). Note that this prevents further matches for the same user name, since a synthetic event for <username> can match the rule only if the context USER_<username>_COUNTED *does not* exist (as requested by the boolean expression in the context field; see CONTEXTS AND CONTEXT EXPRESSIONS section for more information). The operation started by the rule sends an e-mail warning to the local root-user if 30 synthetic events have been observed within 1 minute (see the thresh and window fields). Note that due to the use of the USER_<username>_COUNTED contexts, all synthetic events concern different user names. After sending an e-mail warning, the operation will also create the context COUNTING_OFF with the lifetime of 1 hour, and will continue to run until the 1 minute event correlation window expires. After the operation has finished, the presence of the COUNTING_OFF context will keep the second rule disabled (as requested by the boolean expression in the context field). Therefore, at most one e-mail warning per 1 hour is issued by above rules.

The above examples have presented the event correlation capabilities of SEC in a very brief fashion. The following sections will provide an in-depth discussion of SEC features.

OPTIONS

–conf=<file_pattern>: expand <file_pattern> to filenames (with the Perl glob() function) and read event correlation rules from every file. Multiple –conf options can be specified at command line. Each time SEC receives a signal that forces a configuration reload, <file_pattern> is re-evaluated. See also INPUT PROCESSING AND TIMING section for a discussion on rule processing order for multiple configuration files.
–input=<file_pattern>[=<context>]: expand <file_pattern> to filenames (with the Perl glob() function) and use the files as input sources. An input file can be a regular file, named pipe, or standard input if – was specified. Multiple –input options can be specified at command line. Each time SEC receives the SIGHUP or SIGABRT signal, <file_pattern> is re-evaluated. If SEC experiences a system error when reading from an input file, it will close the file (use the –reopen-timeout option for reopening the file). If <context> is given, SEC will set up the context <context> each time it reads a line from input files that correspond to <file_pattern>. This will help the user to write rules that match data from particular input source(s) only. When there is an –input option with <context> specified, it will automatically enable the –intcontexts option. See INTERNAL EVENTS AND CONTEXTS section for more information.
–input-timeout=<input_timeout>, –timeout-script=<timeout_script>: if SEC has not observed new data in an input file during <input_timeout> seconds (or the file was closed <input_timeout> seconds ago), <timeout_script> will be executed with command line parameters 1 and <the name of the input file>. If fresh data become available again, <timeout_script> will be executed with command line parameters 0 and <the name of the input file>. Setting <input_timeout> to 0 disables this behavior (this is also the default). Note that –input_timeout and –timeout_script options can be used as synonyms for –input-timeout and –timeout-script, respectively.
–reopen-timeout=<reopen_timeout>: if an input file is in the closed state (e.g., SEC fails to open the file at startup, because it has not been created yet), SEC will attempt to reopen the file after every <reopen_timeout> seconds until open succeeds. Setting <reopen_timeout> to 0 disables this behavior (this is also the default). This option has no meaning when the –notail option is also specified. Note that –reopen_timeout is a synonym for –reopen-timeout.
–check-timeout=<check_timeout>: if SEC has not observed new data in an input file, the file will not be polled (both for status and data) during the next <check_timeout> seconds. Setting <check_timeout> to 0 disables this behavior (this is also the default). Note that –check_timeout is a synonym for –check-timeout.
–poll-timeout=<poll_timeout>: a real number that specifies how many seconds SEC will sleep when no new data were read from input files. Default is 0.1 seconds. Note that –poll_timeout is a synonym for –poll-timeout.
–socket-timeout=<socket_timeout>: if a network connection to a remote peer can’t be established within <socket_timeout> seconds, give up. Default is 60 seconds. Note that –socket_timeout is a synonym for –socket-timeout.
–blocksize=<io_block_size>: the number of bytes SEC will attempt to read at once from an input file. Default is 8192 bytes (i.e., read from input files by 8KB blocks).
–bufsize=<input_buffer_size>: set all input buffers to hold <input_buffer_size> lines. The content of input buffers will be compared with patterns that are part of rule definitions (i.e., no more than <input_buffer_size> lines can be matched by a pattern at a time). If <input_buffer_size> is set to 0, SEC will determine the proper value for <input_buffer_size> by checking event matching patterns of all SEC rules. Default is 0 (i.e., determine the size of input buffers automatically).
–evstoresize=<event_store_size>: set an upper limit to the number of events in context event stores. Default is 0 which sets no limit.
–cleantime=<clean_time>: time interval in seconds that specifies how often internal event correlation and context lists are processed, in order to accomplish time-related tasks and to remove obsolete elements. See INPUT PROCESSING AND TIMING section for more information. Default is 1 second.
–log=<logfile>: use <logfile> for logging SEC activities. Note that if the SEC standard error is connected to a terminal, messages will also be logged there, in order to facilitate debugging.
–syslog=<facility>: use syslog for logging SEC activities. All messages will be logged with the facility <facility>, e.g., local0 (see syslog(3) for possible facility values). Warning: be careful with using this option if SEC is employed for monitoring syslog log files, because message loops might occur.
–debug=<debuglevel>: set logging verbosity for SEC. Setting debuglevel to <debuglevel> means that all messages of level <debuglevel> and lower are logged (e.g., if <debuglevel> is 3, messages from levels 1-3 are logged). The following levels are recognized by SEC:
1 – critical messages (severe faults that cause SEC to terminate, e.g., a failed system call)
2 – error messages (faults that need attention, e.g., an incorrect rule definition in a configuration file)
3 – warning messages (possible faults, e.g., a command forked from SEC terminated with a non-zero exit code)
4 – notification messages (normal system level events and interrupts, e.g., the reception of a signal)
5 – informative messages (information about external programs forked from SEC)
6 – debug messages (detailed information about all SEC activities)
Default <debuglevel> is 6 (i.e., log everything). See SIGNALS section for information on how to change <debuglevel> at runtime.
–pid=<pidfile>: SEC will store its process ID to <pidfile> at startup.
–dump=<dumpfile>: SEC will use <dumpfile> as its dump file for writing performance and debug data. With the –dumpfts option, a timestamp suffix is appended to the dump file name. See SIGNALS section for more information. Default is /tmp/sec.dump.
–user=<username>, –group=<groupname>: if SEC is started with effective user ID 0, it will drop root privileges by switching to user <username> and group <groupname>. The –group option can’t be used without the –user option. If the –user option is given without –group, primary group of the user <username> is assumed for <groupname>. If several groups are provided with multiple –group options, SEC switches to the first group with other groups as supplementary groups.
–umask=<mode>: set file mode creation mask to <mode> at SEC startup, where <mode> is a value from the range 0..0777 (see also umask(2)). Octal, decimal, hexadecimal, and binary values can be specified for <mode> (e.g., octal mask 0027 can also be expressed as 23, 0x17, and 0b000010111).
–dumpfts, –nodumpfts: if the –dumpfts option is specified, a timestamp suffix (seconds since Epoch) is appended to the dump file name that reflects the file creation time. Default is –nodumpfts.
–quoting, –noquoting: if the –quoting option is specified, operation description strings that are supplied to command lines of shellcmd, spawn, and cspawn actions will be put inside single quotes. Each single quote (‘) that strings originally contain will be masked. This option prevents the shell from interpreting special symbols that operation description strings might contain. Default is –noquoting.
–tail, –notail: if the –notail option is specified, SEC will process all data that are currently available in input files and exit after reaching all EOFs. If all input is received from a pipe and the –notail option is given, SEC terminates when the last writer closes the pipe (EOF condition). Please note that with named pipes –notail should be used with –norwfifo. With the –tail option, SEC will jump to the end of input files and wait for new lines to arrive. Each input file is tracked both by its name and i-node, and input file rotations are handled seamlessly. If the input file is recreated or truncated, SEC will reopen it and process its content from the beginning. If the input file is removed (i.e., there is just an i-node left without a name), SEC will keep the i-node open and wait for the input file recreation. Default is –tail.
–fromstart, –nofromstart: these flags have no meaning when the –notail option is also specified. When used in combination with –tail (or alone, since –tail is enabled by default), –fromstart will force SEC to read and process input files from the beginning to the end, before the ‘tail’ mode is entered. Default is –nofromstart.
–detach, –nodetach: if the –detach option is specified, SEC will disassociate itself from the controlling terminal and become a daemon at startup (note that SEC will close its standard input, standard output, and standard error, and change its working directory to the root directory). Default is –nodetach.
–jointbuf, –nojointbuf: if the –jointbuf option is specified, SEC uses joint input buffer for all input sources (the size of the buffer is set with the –bufsize option). The –nojointbuf option creates a separate input buffer for each input file, and a separate buffer for all synthetic and internal events (the sizes of all buffers are set with the –bufsize option). The –jointbuf option allows multiline patterns to match lines from several input sources, while the –nojointbuf pattern restricts the matching to lines from one input source only. See INPUT PROCESSING AND TIMING section for more information. If the size of input buffer(s) is 1 (either explicitly set with –bufsize=1 or automatically determined from SEC rules), –jointbuf option is enabled, otherwise the default is –nojointbuf.
–keepopen, –nokeepopen: if the –keepopen option is specified, SEC will keep input files open across soft restarts. When the SIGABRT signal is received, SEC will not reopen input files which have been opened previously, but will only open input files which are in the closed state. The –nokeepopen option forces SEC to close and (re)open all input files during soft restarts. Default is –keepopen.
–rwfifo, –norwfifo: if the –norwfifo option is specified, named pipe input files are opened in read-only mode. In this mode, the named pipe has to be reopened when the last writer closes the pipe, in order to clear the EOF condition on the pipe. With the –rwfifo option, named pipe input files are opened in read-write mode, although SEC never writes to the pipes. In this mode, the pipe does not need to be reopened when an external writer closes it, since there is always at least one writer on the pipe and EOF will never appear. Therefore, if the –notail option has been given, –norwfifo should also be specified. Default is –rwfifo.
–childterm, –nochildterm: if the –childterm option is specified, SEC will send the SIGTERM signal to all its child processes when it terminates or goes through a full restart. Default is –childterm.
–intevents, –nointevents: SEC will generate internal events when it starts up, when it receives certain signals, and when it terminates gracefully. Specific rules can be written to match those internal events, in order to accomplish special tasks at SEC startup, restart, and shutdown. See INTERNAL EVENTS AND CONTEXTS section for more information. Default is –nointevents.
–intcontexts, –nointcontexts: SEC will create an internal context when it reads a line from an input file. This will help the user to write rules that match data from particular input source only. See INTERNAL EVENTS AND CONTEXTS section for more information. Default is –nointcontexts.
–testonly, –notestonly: if the –testonly option is specified, SEC will exit immediately after parsing the configuration file(s). If the configuration file(s) contained no faulty rules, SEC will exit with 0, otherwise with 1. Default is –notestonly.
–help, -?: SEC will output usage information and exit.
–version: SEC will output version information and exit.

Note that options can be introduced both with the single dash (-) and double dash (–), and both the equal sign (=) and whitespace can be used for separating the option name from the option value. For example, -conf=<file_pattern> and –conf <file_pattern> options are equivalent.

CONFIGURATION FILES

Each SEC configuration file consists of rule definitions which are separated by empty lines, whitespace lines and/or comment lines. Each rule definition consists of keyword=value fields, one keyword and value per line. Values are case insensitive only where character case is not important (like the values specifying rule types, e.g., ‘Single’ and ‘single’ are treated identically). The backslash character (\) may be used at the end of a line to continue the current rule field in the next line. Lines which begin with the number sign (#) are treated as comments and ignored (whitespace characters may precede #). Any comment line, empty line, whitespace line, or end of file will terminate the preceding rule definition. For inserting comments into rule definitions, the rem keyword can be used. For example, the following lines define two rules:

type=Single
rem=this rule matches any line which contains \
three consecutive A characters and writes the string \
"three A characters were observed" to standard output
ptype=SubStr
pattern=AAA
desc=Three A characters
action=write – three A characters were observed
# This comment line ends preceding rule definition.
# The following rule works like the previous rule,
# but looks for three consecutive B characters and
# writes the string "three B characters were observed"
# to standard output
type=Single
ptype=SubStr
pattern=BBB
desc=Three B characters
action=write – three B characters were observed

Apart from keywords that are part of rule definitions, label keywords may appear anywhere in the configuration file. The value of each label keyword will be treated as a label that can be referred to in rule definitions as a point-of-continue. This allows for continuing event processing at a rule that follows the label, after the current rule has matched and processed the event.

The points-of-continue are defined with continue* fields. Accepted values for these fields are:

TakeNext: after an event has matched the rule, search for matching rules in the configuration file will continue from the next rule.
GoTo <label>: after an event has matched the rule, search for matching rules will continue from the location of <label> in the configuration file (<label> must be defined with the label keyword anywhere in the configuration file *after* the current rule definition).
DontCont (default value): after an event has matched the rule, search for matching rules ends in the *current* configuration file.
EndMatch: after an event has matched the rule, search for matching rules ends for *all* configuration files.

SEC rules from the same configuration file are matched against input in the order they have been given in the file. For example, consider a configuration file which contains the following rule sequence:

type=Single
ptype=SubStr
pattern=AAA
rem=after this rule has matched, continue from last rule
continue=GoTo lastRule
desc=Three A characters
action=write – three A characters were observed

type=Single
ptype=SubStr
pattern=BBB
rem=after this rule has matched, don’t consider following rules, \
since ‘continue’ defaults to ‘DontCont’
desc=Three B characters
action=write – three B characters were observed

type=Single
ptype=SubStr
pattern=CCC
rem=after this rule has matched, continue from next rule
continue=TakeNext
desc=Three C characters
action=write – three C characters were observed

label=lastRule

type=Single
ptype=SubStr
pattern=DDD
desc=Three D characters
action=write – three D characters were observed

For the input line "AAABBBCCCDDD", this ruleset writes strings "three A characters were observed" and "three D characters were observed" to standard output. If the input line is "BBBCCCDDD", the string "three B characters were observed" is written to standard output. For the input line "CCCDDD", strings "three C characters were observed" and "three D characters were observed" are sent to standard output, while the input line "DDD" produces the output string "three D characters were observed".

If there are two or more configuration files, rule sequence from every file is matched against input (unless explicitly specified otherwise). For example, suppose SEC is started with the command line

/usr/bin/sec –input=- \
–conf=/etc/sec/sec1.rules –conf=/etc/sec/sec2.rules

and the configuration file /etc/sec/sec1.rules has the following content:

type=Single
ptype=SubStr
pattern=AAA
desc=Three A characters
action=write – three A characters were observed

type=Single
ptype=SubStr
pattern=BBB
continue=EndMatch
desc=Three B characters
action=write – three B characters were observed

Also, suppose the configuration file /etc/sec/sec2.rules has the following content:

type=Single
ptype=SubStr
pattern=CCC
desc=Three C characters
action=write – three C characters were observed

If SEC receives the line "AAABBBCCC" from standard input, rules from both configuration files are tried, and as a result, the strings "three A characters were observed" and "three C characters were observed" are written to standard output. Note that rules from /etc/sec/sec1.rules are tried first against the input line, since the option –conf=/etc/sec/sec1.rules is given before –conf=/etc/sec/sec2.rules in the SEC command line (see also INPUT PROCESSING AND TIMING section for a more detailed discussion). If SEC receives the line "BBBCCC" from standard input, the second rule from /etc/sec/sec1.rules produces a match, and the string "three B characters were observed" is written to standard output. Since the rule contains continue=EndMatch statement, the search for matching rules will end for all configuration files, and rules from /etc/sec/sec2.rules will not be not tried. Without this statement, the search for matching rules would continue in /etc/sec/sec2.rules, and the first rule would write the string "three C characters were observed" to standard output.

PATTERNS, PATTERN TYPES AND MATCH VARIABLES

Patterns and pattern types are defined with pattern* and ptype* rule fields. Many pattern types define the number of lines N which the pattern matches (if N is omitted, 1 is assumed). If N is greater than 1, the scope of matching is set with the –jointbuf and –nojointbuf options. With –jointbuf, the pattern is used for matching N last input lines taken from the joint input buffer (the lines can come from different input sources). With –nojointbuf, the source of the last input line is identified, and the pattern is matched with N last input lines from the input buffer of the identified source.

SubStr[N]: pattern is a string that is searched in the last N input lines L1, L2, …, LN. If N is greater than 1, the input lines are joined into a string "L1<NEWLINE>L2<NEWLINE>…<NEWLINE>LN", and the pattern string will be searched from it. If the pattern string is found in input line(s), the pattern matches. Backslash sequences ,
,
, \s, and

sec (1) Linux Manual Page