A week ago, there have been many thousands of requests which were constantly rejected by the smtp service. The weird thing was that, even though it was clear (proper error codes) to the bots that the email could not be relayed through the server, they continued to try.

The following graph shows the number of the rejected requests during the last week:

The following graph shows the inbound/outbound traffic on the router on a random day during the last week:

The following graph shows the DNS failures on a random day during the last week:

Actions

Actually, I decided not to commit any action in order to stop the bots. Despite the continuous rejections, they kept on trying and trying.

I let them go for a week and they gave me 10000+ IPs. These IPs belong to hosts from which those requests were originating. After a week, I added all these IPs in the postfix access table in a way that those hosts were denied access, for example:

XXX.YYY.ZZZ.CCC REJECT

Within the following 30 minutes the number of requests dropped to a minimum! The following graph confirms that:

This clearly shows that those bots are not designed to consider a rejection due to a HELO/recipient/sender etc restriction as a reason to stop trying, but they only stop when they are denied access. This makes sense in a way, as the “access denied” error means that the administrator is aware of what the remote host was trying to do, but, on the other hand, I don’t quite understand the logic behind keeping on trying despite all the non-access-denied rejections.

My conclusion is that the MTA’s access control mechanism is by far more important than I had initially thought. At least, postfix’ access control facility provides all the flexibility one would need.

Anyway, I love logs. The maillogs are being archived and I will have them further analyzed whenever I have the time.

Related Articles

• Email Notifications from a Linux System
• Check Server HTTP Headers with CURL
• Free Personal Email Certificates Program discontinued by Thawte
• Issues with the feeds are now resolved
• Setup the SSH server to use keys for authentication

aureport has a shortcoming. It can only display one report at a time. So, in order to have daily SELinux reports in our inbox aureport needs to be called several times from within a cron job. This is why aureportgen.py was written. It can be configured to execute aureport several times collecting the pre-defined reports for a given period of time and then print a concatenation of these reports to the stdout.

Configuration

Open the aureportgen.py script in a text editor and scroll down to the configuration section. The available options are:

The path to the aureport executable:

aureportpath = "/sbin/aureport"

The list of the reports to collect:

reports = ["a", "mf", "ms", "lf", "ls"]

The line above might be a little cryptic so here is the convention on how to specify the report names:

1. Each report name may consist of up to two letters. The first, which is mandatory, specifies the report type and the second, which is optional, specifies whether a successes or a failures report will be returned.
2. All the aureport command line switches that specify a report type can be used as the first letter.
3. Either the letter “f” or “s” can be used as the second letter to indicate a success-only or failure-only report.
4. If the second letter is omitted, then a report that contains both successful and failed events is returned.

By using the configuration above, the following reports will be returned:

1. AVC denials (successes or failures not applicable to this report)
2. Account modification failures
3. Account modification successes
4. Login failures
5. Login successes

Usage

This script was written in order to be called from within a cron job. It is mandatory to specify the number of days in the past the reports will be generated for. This is done with the –days command line option.

An example daily cron script is listed below:

#! /bin/bash
python /path/to/aureportgen.py --days 1 | mail -s "SELinux Reports by aureport" root
exit 0

The same as a cronjob would be:

0 0 * * *    root    python /path/to/aureportgen.py --days 1 | mail -s "SELinux Reports by aureport" root

This will scan the audit logs and the pre-defined reports will be generated for the last 24 hours. The output will be emailed to root.

WARNING: It should be possible to run this script directly on the CLI and have multiple reports displayed in the standard output, but if the login shell’s locale is different than the crn job’s, then there could be a problem. In such cases, you will still be able to run it from the CLI without errors by setting the extra configuration option fix_date to True:

fix_date = True

License

This project is released under the terms of the GNU General Public License.

Download

You can download this script from the following link:
aureportgen-0.1.tar.gz

Note that this is an alpha version, it is released without any warranties or support.

Related Articles

• How to integrate seaudit-report in logwatch
• Script for Apache Error Report
• Using setenforce to switch SELinux mode wisely
• Get my kernel headers script
• Email Notifications from a Linux System

• -I : when used, CURL prints only the server response’s HTTP headers, instead of the page data.
• -L : if the initial web server response indicates that the requested page has moved to a new location (redirect), CURL’s default behaviour is not to request the page at that new location, but just print the HTTP error message. This switch instructs CURL to make another request asking for the page at the new location whenever the web server returns a 3xx HTTP code.

The combination of the above two switches results in having all the server responses’ headers printed to the terminal, until CURL receives a code other than 3xx.

So, here is a real-life example. I have made two major changes to my web site so far; one included a URL structure modification when I moved from a pure HTML web site to WordPress, while the second was a domain change from the free domain raoul.shacknet.nu to the current paid domain name. So, this makes at least two permanent redirections. I say “at least”, because other necessary redirections could take place as well, for example, redirections of the example.org version of the domain to the www.example.org version, or redirections required for permalinks.

So, here is CURL’s output when I requested an old page of my web site:

$ curl -I -L http://raoul.shacknet.nu/servers/vnc.html

HTTP/1.1 301 Moved Permanently
Date: Fri, 06 Oct 2006 01:49:06 GMT
Server: Apache/2.2.2
Location: http://www.raoul.shacknet.nu/servers/vnc.html
Connection: close
Content-Type: text/html; charset=iso-8859-1
HTTP/1.1 301 Moved Permanently
Date: Fri, 06 Oct 2006 01:49:06 GMT
Server: Apache/2.2.2
Location: http://www.g-loaded.eu/servers/vnc.html
Connection: close
Content-Type: text/html; charset=iso-8859-1
HTTP/1.1 301 Moved Permanently
Date: Fri, 06 Oct 2006 01:49:06 GMT
Server: Apache/2.2.2
Location: http://www.g-loaded.eu/2005/11/10/configure-vnc-server-in-fedora/
Connection: close
Content-Type: text/html; charset=iso-8859-1
HTTP/1.1 200 OK
Date: Fri, 06 Oct 2006 01:49:06 GMT
Server: Apache/2.2.2
X-Powered-By: PHP/5.1.4
X-Pingback: http://www.g-loaded.eu/xmlrpc.php
Status: 200 OK
Connection: close
Content-Type: text/html; charset=UTF-8

The above output shows that the client has to send 4 HTTP requests and receive 4 respective responses from the server in order to reach that old page’s final location:

1. The first server response informs the client that the page has been permanently moved to the www version of the old domain.
2. The second response indicates that the page has been permanently moved to the www version of the new domain using the old URL structure.
3. The third indicates a permanent move to the new URL structure.
4. The final response informs the client with a 200 OK code that it has reached the page’s final location.

In this example, CURL provides a clear view of what situation a browser, a search engine bot or any other HTTP client encounters when it tries to reach an old page.

I am not an expert, but I guess that too many redirects are not a good thing, not only by considering the small increase of the server load, but also that search engine bots might not like them very much. On the other hand, redirections are a necessary evil if you want links from other websites pointing to your old domain or your old URL structure to continue to be valid, which in fact adds “value” to your website, as search engine experts would say. So it’s a matter of choice.

Related Articles

• HTTP headers
• HTTP Status Codes
• The If-Modified-Since HTTP Header
• Use wget or curl to download from RapidShare Premium
• Set up an anonymous FTP server with vsftpd in less than a minute

1. completely override the default audit script by placing one with the same name in /etc/logwatch/scripts/services/
2. create a new custom service for seaudit-report

I decided to follow the second way. The setools source package contains the necessary files for logwatch, but they needed some customization to reflect my setup. I use the auditd service to collect SELinux related messages, which are saved in /var/log/audit/audit.log.

The file that contains information about which log files should logwatch analyze is named seaudit-report-group.conf and needs to be put in the /etc/logwatch/conf/logfiles/ directory. As I mentioned previously, the logged audits are written in /var/log/audit/audit.log, but /var/log/messages still keeps some info about SELinux, eg policy reloads etc. So, the logwatch log-group configuration file should contain the following lines:

LogFile = audit/audit.log
Archive = audit/audit.log.*.gz
LogFile = messages
Archive = messages.*.gz

The logwatch service configuration file, seaudit-report-service.conf, should be put in /etc/logwatch/conf/services/ and should contain the following:

Title = "SELinux Audit"
LogFile = seaudit-report-group

The seaudit-report utility, by default, does not accept input from stdin, so a wrapper script must be used in order to launch this utility with the proper options. The setools source package contains such a script, seaudit-report-service, which should be put in the /etc/logwatch/scripts/services/ directory. The following code is the same as the original script, apart from the line that sets seaudit-report‘s location. In some systems this utility is located in /usr/bin/ and in others, like mine, in /usr/sbin/ etc:

#!/bin/sh
SEAUDITREPORT=$(which seaudit-report)
OPTS="--stdin --malformed"
echo "Date Range: $LOGWATCH_DATE_RANGE"
echo "Detail Level: $LOGWATCH_DETAIL_LEVEL"
echo "Temp Dir: $LOGWATCH_TEMP_DIR"
echo "Debug Level: $LOGWATCH_DEBUG"
${SEAUDITREPORT} ${OPTS}
if [ $? -ne 0 ]; then
    RC=$?
    echo >&2 "Failed while executing seaudit-report.\n"
    exit $RC
fi
exit 0

It is also needed to set the executable bit on this script:

# chmod u+x /etc/logwatch/scripts/services/seaudit-report-service

By issuing the following command, the SELinux report should be printed to stdout:

# logwatch --service seaudit-report-service --range all --print

Finally, it is possible to disable logwatch’s default SELinux analysis service (audit) by adding the following line in /etc/logwatch/conf/logwatch.conf:

Service = "-audit"

From now on, the logwatch report should contain, among others, only the report produced by seaudit-report.

Related Articles

• Logwatch and Dovecot 1.x series in FC5
• SELinux audit reports script
• Script for Apache Error Report
• A real-time log viewer
• Email Notifications from a Linux System

1. Fedora 5 includes dovecot v1.x, which has some differences in its log output format compared to older versions,
2. the stock logwatch package in Fedora contains a dovecot log parser that supports dovecot up to the 0.99.x version.

This issue’s resolution is rather simple.

First of all, get the updated dovecot log parser from the logwatch CVS repository. I used the latest revision, 1.4 at the moment of writing. Create a services/ directory inside the /etc/logwatch/scripts/ directory, put the updated parser in there and set its executable bit:

# chmod u+x /etc/logwatch/scripts/services/dovecot

This script will be the one to parse the Dovcecot log file instead of the default one.

Next, it is required to override a setting in the dovecot service’s configuration file. So, create a text file, named dovecot.conf, in /etc/logwatch/conf/services/ and write the following line:

*OnlyService = (imap-login|pop3-login|dovecot)

Now, test if it works:

# logwatch --service dovecot --range yesterday --detail 10 --print

This will print the dovecot report to stdout.

There are some things that you should take a note of. The logwatch Dovecot report will work only if dovecot is configured to record its log entries to the system log (syslog), which in turn moves these entries to /var/log/maillog. If you have configured Dovecot to record its log entries to a separate file, other than syslog, then it is very unlikely that the whole thing will work for any Dovecot version. This is because of the different way log entries are written to the separate log.

Related Articles

• How to integrate seaudit-report in logwatch
• Creative PC-CAM Series webcams in linux
• A real-time log viewer
• Fedora sound recording problems… solved
• SELinux audit reports script

How ModSecurity Works

A definition of ModSecurity, as it appears in the official website, would be:

ModSecurity(TM) is an open source intrusion detection and prevention engine for web applications. It can also be called a web application firewall. It operates embedded into the web server, acting as a powerful umbrella, shielding applications from attacks.

The analysis of the client request and the server response is performed in stages:

1. In the first stage, the request’s format is analyzed by a series of built-in checks (implicit validations). These checks can be controlled using configuration directives.
2. In the second stage, the request goes through a series of user-defined input-filters. Whenever there is a match, a list of user-defined actions is performed.
3. The request is processed by Apache.
4. If output filtering is enabled, then the output goes through a series of user-defined output-filters. If there is a match, then the specified actions are performed.

Installation

Install the mod_security Apache module as you install every other package. On Fedora, for example, you can use yum:

# yum install mod_security

Basic Configuration

Ususally, there is a separate configuration file for each Apache module, which is imported into the main Apache configuration. On Fedora, ModSecurity’s configuration file exists in /etc/httpd/conf.d/mod_security.conf.
The basic configuration would be:

<IfModule mod_security.c>
    SecFilterEngine On
    SecAuditEngine RelevantOnly
    SecAuditLog logs/audit_log
    # Default action list
    SecFilterDefaultAction "deny,log,status:406"
    # Implicit validations
    SecFilterCheckURLEncoding On
    SecFilterCheckUnicodeEncoding On
    SecFilterForceByteRange 1 255
    SecFilterNormalizeCookies On
    #SecFilterCookieFormat 1
    # POST Payload scanning
    SecFilterScanPOST On
    # Output Filtering
    #SecFilterScanOutput On
    #SecFilterOutputMimeTypes "(null) text/html text/plain"
</IfModule>

All the directives that are used in the above block of code are called “Configuration Directives” in ModSecurity terminology. Below, there is some brief explanation for each one of them:

• General configuration directives: ModSecurity’s operation involves two engines that work independently, the filter engine and the audit engine. Both engines can be turned On or Off.

  SecFilterEngine 

  Turn the filtering engine On or Off.

  SecAuditEngine 

  Turn the audit engine On or Off. This can be done on a per-server (virtualhost context in the Apache configuration) or on a per-directory basis (directory or .htaccess context in the Apache configuration). If this directive is set explicitly to On, all requests will be logged. If it is set to RelevantOnly, only those requests that were matched by a filter are logged. This is the recommended setting.

  SecAuditLog 

  The path to the audit log file.

• Everytime a filter matches a request, an action is performed. These actions can be defined on a per-rule basis, but also a general default action can be specified.

  SecFilterDefaultAction 

  This directive sets the default action to be performed when a filter catches a request. The general syntax of actions is used, but some actions cannot be set in this directive (see ModSecurity Documentation). For example, the default action above will configure the engine to log each rule match and reject the request with status code 406.

• General request validations (URL encoding, Unicode encoding, cookie format, byte range checks) can be performed by some configuration directives. These are built-in checks and are called implicit validations in ModSecurity terminology. Note, that in case you want to run ModSecurity in detect-only mode without rejecting any requests, for example by setting all the per-filter primary actions to pass, it is also required to set the default action to one that will not reject the request (eg pass), so that, even if the request fails to pass through these validations, the request analysis can continue on to the checks defined by the filters. An alternative for detect-only operation would be to explicitly turn off these built-in validations. For normal operation though, it is highly recommended that these validations are turned on:

  SecFilterCheckURLEncoding 

  Make sure that URL encoding is valid (On|Off).

  SecFilterCheckUnicodeEncoding 

  Make sure that Unicode encoding is valid (On|Off).

  SecFilterForceByteRange 

  Performs byte range checks. Only bytes that are in the specified range will be allowed in the request. Usually, the 1-255 range is accepted, but if you need to make it even more strict, you can set it to allow only bytes in the 32-126 range, for example: SecFilterForceByteRange 32 126

  SecFilterNormalizeCookies 

  By default, ModSecurity will not try to normalize cookie names and values. Some web applications encode the cookie content, but that can be normalized by setting this directive to On. The directive SecFilterCheckCookieFormat is completely deprecated in v1.9.x and has no effect.

  SecFilterCookieFormat 

  By default, ModSecurity supports normalization on cookies that are in Netscape format. It can be configured though to support version 1 cookies (as defined in RFC-2965). To enable version 1 cookie support set this directive to 1.

• By default, ModSecurity can scan only GET variables. Scanning POST variables (POST Payload) is disabled by default.

  SecFilterScanPOST 

  By setting this directive to On the post payload can also be scanned.

• By default, ModSecurity does not perform output filtering.

  SecFilterScanOutput 

  Set this configuration directive to On in order to enable output filtering (only Apache 2.x). Note that, when output filtering is enabled, only responses that have no content type, or whose content type is text/plan or text/html will be scanned. This behaviour can be modified with the SecFilterOutputMimeTypes directive.

  SecFilterOutputMimeTypes 

  Set which content types will be scanned in output filtering.

Specifying Actions

Whenever a filter catches a request, then an action, or better, a list of actions is performed. The general syntax for action lists is (no spaces are allowed between actions):

"primary_action,secondary_actions,flow_action or parameter:value action"

For example:

SecFilterDefaultAction "deny,log,status:406"

Action lists can be defined in three places:

1. SecFilterDefaultAction directive: Whenever a user-defined rule, which follows this directive, is matched, then these default actions are performed, unless they are overridden by a per-ruleset or per-rule action list (see below).
2. SecFilterSignatureAction directive: This directive may appear several times in the configuration and is used in order to specify a per-ruleset action list. A ruleset consists of all the rules that immediately follow the SecFilterSignatureAction directive. Action lists that are defined in this directive are merged with the default action list. Note that this merging procedure may result in action overriding. It depends on the type of the actions used. See the documentation for more information on this. Also, note that this action list will not be inherited by child contexts.
3. SecFilter and SecFilterSelective directives: These directives accept an optional action list to be performed whenever the rule is matched. These are per-rule actions. Actions defined in these directives are merged with the actions that are defined in the other two directives above. Note that this merging procedure may result in action overriding. It depends on the type of the actions used. See the documentation for more information on this.

Another action-related directive is SecFilterActionsRestricted. When it is set to On, all the per-rule actions, except for the metadata actions (id, msg, rev, severity), are ignored. This is particularly useful when importing 3rd party rules, which also contain per-rule action lists, but you want to define your own action-list to be performed when any of these rules is matched.

In short, the most commonly used actions and what they actually do on a filter match are outlined below:

pass

Do nothing and continue on with the next rule. Useful when running ModSecurity in detect-mode.

allow

Allow the request, but do not continue the request or server-response analysis.

deny

Deny the request and return the error document which corresponds to the defined status code (see below).

log

Log to the Apache error log and to the adit log if the audit engine is enabled.

nolog

Log nothing to the Apache error log or to the audit log.

status

(status:CODE): Specifies the HTTP error code that will be returned if the request is rejected.

redirect

(redirect:URL): Redirect to specified page. Always overrides the status and deny actions.

exec

(exec:/path/to/script): Execute the specified script. This action is always performed in addition to the primary action (if one is defined). The script must write its output to stdout.

Also, a flow action, which affects the order in which the rules are processed, may be defined:

skipnext

(skipnext:N): Skips next N rules on filter match.

chain

Combine two or more filters together. The last filter is the one that will affect the request, but in order to reach the last filter, all previous chained filters must be matched.

One of the most useful features of ModSecurity is that metadata (rule id, revision number, a text message, severity information) may be defined for each filter. This metadata is defined as parameter:value, where action can be one of id, rev, msg, severity. For example:

SecFilter ".*admin.*" id:3,severity:1

Finally, a filter or a chain of filters can be explicitly marked for inheritance in child contexts with the mandatory action. For example:

SecFilter ".*admin.*" mandatory

or

SecFilter ".*admin.*" mandatory,chain
SecFilter ".*login.*"

For more information refer to the ModSecurity documentation.

Specifying Filters

Modsecurity supports writing filters in the ways outlined below. For further information refer to the documentation.

• Simple Input Filter Syntax:

SecFilter KEYWORD [ACTIONS]
SecFilter !KEYWORD [ACTIONS]

• Advanced Input Filter Syntax:

SecFilterSelective LOCATION KEYWORD [ACTIONS]
SecFilterSelective LOCATION !KEYWORD [ACTIONS]

• Advanced Output Filter Syntax:

SecFilterSelective OUTPUT KEYWORD [ACTIONS]
SecFilterSelective OUTPUT !KEYWORD [ACTIONS]

KEYWORD 

is a regular expression.

 ! 

means that the regular expression is inverted, like the NOT logical operator.

LOCATION 

is a location identifiers or a list of location identifiers separated with the pipe | symbol. The location identifiers might be for example, the remote party’s IP (REMOTE_ADDR) or hostname (REMOTE_HOST) etc. For a detailed list of all the possible location identifiers refer to the “Advanced Filtering” section of the ModSecurity documentation.

OUTPUT 

indicates that the request will first be processed by Apache and the output will be checked against this filter. Requires that the directive SecFilterScanOutput has been set to On.

A special usage of advanced filtering is when the ARGS and the ARGS_someformfield are used. In this case, ARGS_someformfield supports inverted usage, so that all form fields are checked, except for the field that is defined in the ARGS_someformfield statement. For example:

SecFilterSelective "ARGS|!ARG_firstname" "Jack"

This one checks if any of the form fields, except for field “firstname“, is set to “Jack”.

Filter Inheritance

The filter inheritance scheme in ModSecurity follows the rules outlined below.

• By default, all filters, together with their per-rule action lists, are inherited by child contexts.
• The default action list is also inherited by child contexts.
• Action lists, defined in the SecFilterSignatureAction directive (per-ruleset actions), are never inherited by child contexts.

It is possible to customize this scheme by using one or more of the directives SecFilterInheritance, SecFilterInheritanceMandatory, SecFilterImport, SecFilterRemove or by setting the per-rule action “mandatory” on a specific filter or filter chain.

Here are some notes about these directives:

SecFilterInheritance
Controls the inheritance of rules from the parent context. By default, its value is On. By setting it to Off in a context, eg virtualhost, none of the filters defined in parent contexts is inherited. This directive needs to be explicitly set in every context in which you do not want to inherit any rules. For example:

<Directory /path/to/some/dir>
    SecFilterInheritance Off
</Directory>

SecFilterImport
This directive works in conjuction with SecFilterInheritance and has a meaning only if the latter has been set to Off in a particular context. It accepts a space-delimited list of rule IDs and can be used to explicitly import filters from parent contexts. For example:

<Directory /path/to/some/dir>
    SecFilterInheritance Off
    SecFilterImport 1001 1002 1003
</Directory>

SecFilterRemove
This directive is the exact opposite of SecFilterImport. It has a meaning only if SecFilterInheritance has not been disabled in a particular context and works only for filters that have not been marked for mandatory inheritance (see below). It accepts a space-delimited list of rule IDs and can be used to explicitly disable inherited filters. For example:

<Directory /path/to/some/dir>
    SecFilterRemove 10 11 12
</Directory>

SecFilterInheritanceMandatory
Controls the inheritance of rules for the child contexts. By default, its value is Off. By setting it to On in a context, eg virtualhost, all of the filters defined in this particular context will be inherited by force by child contexts, despite the fact that filter inheritance might be disabled in those child contexts. This directive needs to be explicitly set in every context whose filters need to be always in-effect in subcontexts. For example:

<Virtualhost 192.168.0.1:80>
	SecFilterInheritanceMandatory On
	SecFilter ".*admin.*" "id:10"
	<Directory /path/to/some/dir/in/this/vhost>
		SecFilterRemove 10
	</Directory>
</Virtualhost>

In the above example, the filter with ID 10 will still be in effect in the directory context, because all rules have been marked for mandatory inheritance in its parent context.

Some times, it is needed to mark only specific rules for mandatory inheritance and not all of the current context’s rules. This can be achieved by using the mandatory action in a per-rule action list. For example:

<Virtualhost 192.168.0.1:80>
	SecFilter ".*admin.*" "id:10,mandatory"
</Virtualhost>

The rule with ID 10 will always be inherited by child contexts.

It is recommended that some critical filters are marked for mandatory inheritance, especially in environments where there is no trust between the users.

Per-Virtualhost ModSecurity Logging

It is possible to use Apache’s custom logging feature in order to log requests, which matched a ModSecurity filter, on a per-virtualhost basis. The key for this to work is the fact that ModSecurity defines the environment variable mod_security-relevant whenever a rule is matched.

So, by adding the following statement in the virtualhost context, Apache will record information about ModSecurity’s activity for the specific virtualhost. This statement is taken from the official documentation:

<IfModule mod_security.c>
    CustomLog /path/to/logs/modsec_custom_log \
        "%h %l %u %t \"%r\" %>s %b %{mod_security-message}i" \
        env=mod_security-relevant
</IfModule>

Recommended Filters

Accepted Encoding Types
As it is stated in the docs, ModSecurity supports two encoding types for the request body:

1. application/x-www-form-urlencoded – used to transfer form data
2. multipart/form-data – used for file transfers

In order to be sure that the web server will only accept requests with these two encoding types, a selective filter can be added. Note that GET requests are excluded from this rule because some (automated) clients supply “text/html” as Content-Type. Also, keep in mind that some web applications make use of the XMLRPC libraries in order to perform inter-application communication (communication between different web sites), send pingbacks/trackbacks for example. In this case, these features will not work, unless the text/xml encoding is accepted by ModSecurity.

# Accepted encoding types for request
SecFilterSelective REQUEST_METHOD "!^GET$" chain
SecFilterSelective HTTP_Content-Type "!(^$|^application/x-www-form-urlencoded|^multipart/form-data|^text/xml)"

Chunked Transfer Encoding
Reject requests whose body is delivered in chunks. This will not affect the server’s ability to send responses using the chunked transfer encoding (Rule taken from the ModSecurity Documentation):

# Reject Requests With Chunked Transfer Encoding
SecFilterSelective HTTP_Transfer-Encoding "!^$"

Missing User-Agent or Host Headers
Reject requests on which the User-Agent or Host headers are empty. This will only reject poorly made bots that do not define a user-agent string. Note that bots can easily use a fake user-agent string so to pretend that are common internet browsers. Nothing can be done to prevent this ability.

# Require HTTP_USER_AGENT and HTTP_HOST headers
SecFilterSelective "HTTP_USER_AGENT|HTTP_HOST" "^$"

Missing Content-Length Header
Reject POST requests that do not provide the Content-Length header (Rule taken from the default mod_security.conf supplied on Fedora):

# Require Content-Length to be provided with every POST request
SecFilterSelective REQUEST_METHOD "^POST$" chain
SecFilterSelective HTTP_Content-Length "^$"

Path Traversal Attacks
Reject requests which attempt to confuse the web server and make it traverse system paths and possibly execute shell commands. Note that this general filter might break the normal operation of web applications which use shell commands:

# Reject Path Traversal attacks
SecFilter "\.\./"

Cross-Site-Scripting Attacks (weak rule)
This filter rejects Cross-Site-Scripting Attacks (XSS) by preventing the usage of javascript statements (Javascript injection). This rule is safe to use, unless your application uses javascript in GET/POST request variables, which is very unlikely. (Rule taken from the default mod_security.conf supplied on Fedora):

# Weak rule to reject Cross-Site-Scripting Attacks (XSS) using javascript
SecFilter "<( |\n)*script"

Cross-Site-Scripting Attacks (stronger rule)
This filter rejects XSS Attacks by preventing HTML and Javascript injection attempts. This filter is a lot more general and stronger than the above and its usage is not recommended for most content management and web-forum systems, as there is a high possibility that it wll break their normal operation. (Rule taken from the default mod_security.conf supplied on Fedora):

# Stronger rule to reject XSS Attacks (HTML/Javascript)
SecFilter "<(.|
)+>"

ModSecurity Resources

There are many web sites that offer ready ModSecurity rulesets for various web applications. In the following links you can find some very useful general rulesets, but keep in mind that they need a bit of customization in order to work with your web application.

Some rule resources:

• http://www.modsecurity.org/projects/rules/index.html
• http://www.gotroot.com/tiki-index.php?page=mod_security+rules

Useful information about the attacks:

• http://www.modsecurity.org/db/resources/

Further Reading

The official documentation was used as a base for this overview and, in my opinion, is the best resource for help when coniguring ModSecurity. Everything is written in great detail in there:

1. The ModSecurity documentation

Also, you might want to have a look at a web interface that aims to provide great help when writing filters for ModSecurity. This is available here.

Notes

I only recently started using mod_security. The first things that need to be done in order to be able to create effective filters that actually protect a web application is to study the application itself. Setting non-fatal actions for certain filters and checking the audit logs regularly is a good start. The goal is to protect the web application from being mis-used, but without breaking its normal operation.

This document is the second revision of the article, but it is still a draft. It needs to be enhanced with more examples. I also intend to write two extra articles related to this document, which will contain custom filter sets for WordPress and MediaWiki, which are the web applications I use. Links to those articles will appear in the “Further Reading” section of this document.

Related Articles

• Use mod_deflate to Compress Web Content delivered by Apache
• Check Server HTTP Headers with CURL
• A quick AWstats guide
• .htaccess Cheat Sheet
• High traffic on the email server

How It Works For Me

The script is written around the logic that a normal comment submitter usually posts a rational number of comments in a 30-day period. All blog owners have an idea of the comment traffic on their blogs. At least on this blog, most people won’t post more than 2-3 comments during a month. Even if they do not fill all the required fields on the comment form and they have to re-submit, which is almost rare, the number of the POST HTTP requests they send is not greater than 10 or 12.

On the other hand, a spammer, or better a spambot, usually tries to do more than that. After doing some research myself, I have concluded to the following: Even if they use dynamic IP addresses, which is a fact in most cases, the number of the POST HTTP requests they send during a 30-day period to the wp-comments-post.php file is by far greater than the most regular commenter’s POST requests on this blog. And I’m talking about 40-60 or even more POST HTTP requests from the same remote host in a month. This sounds like a spambot to me (correct me if I’m wrong).

The Script

This script, although it was written for my own use, contains some configuration parameters, so to be customized for different setups. You direct it to the directory that contains the rotated apache log files and it searches for the POST HTTP requests in all of them. My logs are rotated daily, but recycled monthly. Every day’s log file is also kept in my main apache log archive (yes, I keep everything). So, this script will give me a report for a whole month.

#! /bin/bash
 
##############################################################
# Configuration BEGIN
####################
# The minimum number of POST HTTP requests a remote host
# may have sent, without getting into this list.
# ADJUST ACCORDING TO YOUR BLOG'S COMMENT TRAFFIC
# 20 is just an example!!
MinPostReq=20
# Where to send the report
MailTo="me@example.com"
 
# Which hosts to exclude. Separate each host with an escaped |, eg \|
fexclude="desktop.example.com\|server.example.com"
 
# The directory path that contains the Apache Logs
# NO TRAILING SLASH
path="/path/to/rotated/apache/logs"
 
# The page that accepts the comments. For WordPress, this is the following:
page="/wp-comments-post.php"
# Set it to empty "" to see the remote hosts that have sent POST requests on
# any page of your web site (you will be amazed!)
#page=""
 
################
# Configuration END
#############################################################
 
if [ -z $fexclude ]; then
    # If no hosts are excluded, then set it to a random value,
    # else grep -v will excude everything
    fexclude="dfahjgf32eDFSDFaFaD"
fi
 
(
echo -e "Req    Remote Host\n===================================================="
(
for i in $path/access_log.*.gz; do
    zcat $i | grep '"POST '$page | grep -v '^\('$fexclude'\)' | awk '{ print $1 }'
done
) | sort | uniq -c | sort -rg \r
    | awk '{ if(int($1)>'$MinPostReq') { printf(" %-5s %s\n",$1,$2) } }'
) | mail -s "Report Of Potential Spammers" $MailTo
 
exit 0

Configuration

Here is some info about the configuration options:

• MinPostReq : The number of POST HTTP requests a remote host has sent to the /wp-comments-post.php file. This is the most critical option. Adjust it according to your blog’s comment submission traffic. For example if the most regular commenter posts 5 comments/month, set this a lot higher, eg 20 or more. Setting a good number here can reduce the error.
• MailTo : The email to send the report.
• fexclude : Which hosts to exclude. For example you can put all the hosts from which you administer WordPress. Make sure you separate them with an escaped | , eg: |
• path : the path to the directory that contains the rotated apache logs. All log files of the form: access_log.#.gz will be searched.
• page : The WordPress page where the comments are submitted to. This is the /wp-comments-post.php file. You can set this option to nothing in order to see which hosts have sent POST HTTP requests and the total number of these requests. You will be amazed with the result.

The Report

This script will send a report to your email address, which contains a list of the remote hosts that have sent POST HTTP requests to the /wp-comments-post.php and the number of these requests.

These MAY NOT be all spammers. Some of them could be readers who have submitted a lot of comments relevant to your blog. This report is just an overview of the POST HTTP requests towards WordPress and it will take further investigation to determine if a remote host has actually spammed you or tried to.

Read The Following Section At Least 10 Times

THIS REPORT IS NOT A LIST OF SPAMMERS.
You have been warned! Make sure you don’t accuse your innocent readers/commenters!! This will be your own mistake and not this script’s or my fault.

By using the above script or part of it, you explicitely accept the above statement.

What You Can Do

If, after your own investigation, you are 100% sure that a remote host has spammed you, you can use any whois service to find the remote host’s internet provider and send them the relevant parts of your apache log files, the contents of the comments and any other information you have collected that proves that you were spammed from this address at a specific time. And remember: be polite. These people have no other relation to the spammer, apart from the fact that the latter is their client.

What If It Does Not Work?

Consider this script as a note you have seen. This is no release! I do not even care if it works or not. I’ve already stated that all this may be totally useless for you or for your setup. If this is the case, write a script for your own setup.

Related Articles

• SELinux audit reports script
• bbPress for WordPress
• Script for Apache Error Report
• Mass download
• High traffic on the email server

Article Goals

The goals of this document are to:

1. Install AWstats in a custom location as a normal user. Although it is possible to have a system-wide installation, I chose this approach for completeness. The differences between the two methods are just the scripts’ locations. The rest of the configuration stays the same.
2. Create a configuration file for our web site (Apache Virtual Host) for as accurate statistics as possible.
3. Parse this host’s log file and create a database with statistical data.
4. Use this statistical data to generate web site traffic reports. We will focus on the creation of static HTML reports, but some info on how to use awstats.pl as a CGI is also provided.
5. Make a quick introduction to user-defined charts.

Prerequisites

This document assumes that:

• Our web site is configured to have its own log file.
• The log file is written in the "combined" format (NCSA combined/XLF/ELF log format)
• You have configured the Apache web server to do reverse DNS lookups (HostnameLookups On). This means that the log files contain the visitors’ hostnames instead of their IP addresses in the HOST field. This is not necessary though.
• We have access to the log file.

Actually, only the last one is a necessity, as awstats can be configured to generate statistics even from heavily customized log formats. For this article we will use an apache log file in the "combined" format.

Custom installation in our Home directory

We will install the AWstats package in our Home directory. So, download the latest awstats version from the Project Page and extract it:

# tar -xzvf awstats-X.X.tar.gz -C /home/jsmith/

A new directory (awstats-X.X) is created in our Home. This is where all scripts and other supplemental files are installed. You may want to rename this to just awstats:

# mv awstats-X.X awstats

We will need to create two more directories, one for the awstats statistical data and one for the traffic reports (static HTML pages). So, we create the first one inside the awstats installation directory:

# mkdir /home/jsmith/awstats/statdata

The directory which will hold the traffic reports can be located inside our web site’s root directory, so that they are accessible from a web browser. Assuming that our DocumentRoot is /home/jsmith/public_html/, we create a new directory in there:

# mkdir /home/jsmith/public_html/traffic

Using this installation scheme, we avoid exposing the awstats scripts to the internet. Only the traffic reports will be accessible through a web browser. This means that it will not be possible to use awstats.pl as a CGI script to generate reports dynamically (directly from our statistical data), but this behaviour can easily be changed.

We also need to copy some images, which are used in the HTML or PDF traffic reports, to the traffic directory:

# cp -R /home/jsmith/awstats/wwwroot/icon/ /home/jsmith/public_html/traffic/

The last part of the installation process is to set the appropriate permissions to the AWstats directories and files. So, we set the mode to 0755 for directories and 0644 for all files. Because the Perl scripts (*.pl files) need to be executable, we set their mode to 0755. The following lines do all this:

# find /home/jsmith/awstats/ -type d | xargs chmod 0755
# find /home/jsmith/awstats/ -type f | xargs chmod 0644
# find /home/jsmith/awstats/ -type f -name *.pl | xargs chmod 0755

We also need to set the appropriate permissions to the directory which will hold the reports and which will be accessible from the internet:

# find /home/jsmith/public_html/traffic/ -type d | xargs chmod 0705
# find /home/jsmith/public_html/traffic/ -type f | xargs chmod 0604

That’s enough with the installation.

Configuration

We need to create a configuration file for our web site. This file will be read by awstats.pl in order to generate the statistical data or the traffic reports. There is a sample configuration file in the /home/jsmith/awstats/wwwroot/cgi-bin/ directory, named awstats.model.conf. We make a copy of this file in the same directory and replace the "model" part of the name with one that will represent our web site:

# cp awstats.model.conf awstats.mysite.conf

We will work on the copy. Although, modifying only a few basic directives, such as the logfile path and the statistical data directory path, would be enough, we will modify some more, so that our statistics are as accurate as possible and our reports look the way we want.

Open the awstats.mysite.conf file in your favourite text editor and let’s start customizing it.

Note: I would suggest that you do not use relative paths whenever needed, but rather absolute ones.

LogFile="/home/jsmith/logs/access_log"
LogType=W
LogFormat=1
LogSeparator=" "

These are log file specific directives. If your log file is in the "combined" format, all you have to modify is its path.

SiteDomain="www.mysite.com"
HostAliases="mysite.com"

Here we set our web site’s URL and all the aliases that can be used to reach the site with a web browser. Separate all aliases with a "space".

DNSLookup=0

By setting this directive to 0, no reverse DNS lookup requests will be sent to the nameserver. I have set the Apache web server to do these lookups, so a value of 0 is the proper one. You can set this to 1, which will lead to numerous lookup requests to the nameserver, or 2, which will make awstats do the resolving by examining a DNS cache file, if it exists. Keep in mind that having awstats do the reverse DNS lookups will slow the statistics update process dramatically.

DirData="/home/jsmith/awstats/statdata"

Set the directory where awstats will keep its statistical data. This is one of the directories we had created in the installation process.

DirCgi="/home/jsmith/awstats/wwwroot/cgi-bin"

This is the directory that contains the awstats.pl script.

DirIcons="icon"

Remember that we had previously copied the awstats icons to the directory which will hold our reports? That’s why we do not need to specify an absolute path for these. Just set it to icon.

CreateDirDataIfNotExists=0

If you had previously created the directory which will hold the statistical data, then a value of 0 will do. Otherwise set it to 1 to have the directory you have specified in the DirData directive created.

KeepBackupOfHistoricFiles=1

It’s a good habit to have awstats keep a backup of the historic data during the update process.

DefaultFile="index.php"

Here we define the index file for our web site. In other words, our home page. This depends on your site.

SkipHosts="OUR OWN PCs' HOSTNAMES"

This is a very important directive ragarding the accuracy of the statistics. Usually, we are our web site’s most regular visitor and it’s obvious that we do not want to be counted as a visitor. This directive can take IP addresses or hostnames as values, separated with a space. Regular expressions can be used in the form of REGEX[value]. IP addresses cannot be mixed with hostnames, so, if the DNS lookups take place at the web server level, then we have to use hostnames as values, otherwise we have to use IP addresses. Usually we need to set the IPs or hostnames of all our LAN computers or computers we use to edit the website, so that they are ignored. Below are some examples:

SkipHosts="localhost REGEX[^.*\.example\.dyndns\.org$] test.mysite.com windowspc1"
OR
SkipHosts="127.0.0.1 REGEX[^192\.168\.] REGEX[^10\.]"

SkipUserAgents=""

If you use any custom spiders or bots to test or analyze your web site, but you don’t want their access to be included in the stats, then you should add their "User Agent String" as a value to this directive. Again, regular expressions can be used in the form of REGEX[value].

NotPageList="css js class gif jpg jpeg png bmp ico swf"

This is another important directive. Here we set what file extensions will not be counted as Page Views or Downloads, but only as Hits. Usually, this list includes files that are part of a web page (images, stylesheets, flash animations, java applets etc.).

URLWithQuery=0
URLWithQueryWithOnlyFollowingParameters=""
URLWithQueryWithoutFollowingParameters=""
URLReferrerWithQuery=0

These do not need to be modified, unless you want the query string to be included in the web page URLs or Referrer URLs in the traffic reports. Enabling the URLWithQuery directive is important in the case your web page URLs are of the form /index.php?p=10, so that it is clear in the traffic reports which page was viewed. On the other hand, if your page URLs are of the above form, but you use permalinks, then it’s not needed to modify the default values for these directives.
Including the query string in the referrer URLs is not important, in fact it can lead to lengthy meaningless referrer lists, which is not so convenient. I just provide this info here for completeness.

LevelForWormsDetection=2

By default, the detection of worms that have crawled your web site is disabled. You may want to enable this by setting the above directive’s value to 2.

Lang="en"

If you need the reports to be in a specific language, set it here. A list of supported languages exists in the configuration file.

ShowAuthenticatedUsers=PHBL

If your reports are private, you may set this directive’s value to PHBL (details about what each letter represents can be found inside the conf file), so that a section with details about your web site’s authenticated users is included in the reports.

ShowWormsStats=HBL

If you had previously enabled the worm detection, then you may want to include a detailed section about worms in the reports.

IncludeInternalLinksInOriginSection=1

By setting this to 1, a summary of how many links to another internal page have been followed from your site’s pages is included in the reports.

MaxNbOfDomain = 10
MaxNbOfHostsShown = 10
MaxNbOfLoginShown = 10
MaxNbOfRobotShown = 10
MaxNbOfPageShown = 10
MaxNbOfOsShown = 10
MaxNbOfBrowsersShown = 10
MaxNbOfRefererShown = 10
MaxNbOfKeyphrasesShown = 10
MaxNbOfKeywordsShown = 10

Here follows some info about the reports. You can create only one main page with a summary of the web site’s traffic, but you can also generate some supplemental pages which have full lists of the visited pages, referrers, countries, search engines etc. Each section in the main page includes a predefined number of entries that are displayed. For example, it displays by default the top 10 referrers. This number can be customized by modifying the directives above.

ShowLinksOnUrl=1

By default, each URL shown in the reports is a clickable hyperlink. If you do not want them to be actual hyperlinks, then set this to 0.

MaxRowsInHTMLOutput=1000
DetailedReportsOnNewWindows=1

With these directives you set the number of entries each of the supplemental reports can have and if you want these supplemental reports to be opened in a new browser window.

LoadPlugin="tooltips"
LoadPlugin="decodeutfkeys"

The AWstats package includes some plugins you can enable. I found the above two to be helpful. The first one enables the display of some descriptive tooltips and the second one makes it possible to show keywords and keyphrases correctly using national characters.
There are some other interesting plugins inside the awstats package, but also some more from other contributors. You can find the latter at the project’s web site. Keep in mind, that each plugin may require certain Perl modules to be installed.

Update the statistics database

Now that we have finished customizing our web site’s configuration file, we can finally have awstats.pl parse our log file and create statistical data:

# perl /home/jsmith/awstats/wwwroot/cgi-bin/awstats.pl -config=mysite -update -showcorrupted

Notice that we do not use the whole configuration file’s filename (awstats.mysite.conf) to define our configuration, but only the part between awstats. and .conf.

The -showcorrupted option is not necessary. A total number of corrupted records would be displayed anyway. This just provides detailed info.

It would be convenient if you set cron to execute the above command on a daily or hourly basis. Here is a small BASH script that can be run through cron:

#! /bin/bash
# Update the statistics database
perl /home/jsmith/awstats/wwwroot/cgi-bin/awstats.pl -config=mysite -update -showcorrupted
# Calculate Total Visits for all months
TotVisits=$(grep ^TotalVisits /home/jsmith/awstats/statdata/*.txt | sed 's/^.*awstats.*TotalVisits.//' | awk '{sum += $1} END {print sum}')
# Export a small GIF image with the number of total visits
text2gif -t "$TotVisits" > /home/jsmith/public_html/traffic/counter.gif
# Set proper permissions on the GIF image
chmod 0604 /home/jsmith/public_html/traffic/counter.gif
exit 0

This small script updates the statistical data, calculates the total visits for all months and exports a small B&W GIF image which can be used as our custom counter in our web site. It’s not a real-time counter, but it’s better than nothing… Anyway, this just an example. The text2gif utility is part of the libungif-progs package.

Generate traffic reports

There are two methods to generate reports. Either by using awstats.pl directly or by using a helper script, named awstats_buildstaticpages.pl.

To generate the main report for November 2005 using awstats.pl, you can issue the following command:

# perl /home/jsmith/awstats/wwwroot/cgi-bin/awstats.pl -config=mysite -month=11 -year=2005 -output -staticlinks > /home/jsmith/public_html/traffic/awstats.mysite.200511.html

If the options -month and -year are omitted, then the report is generated for the current month. You can also generate a report for a whole year, by setting these two options to -month=all and -year=2005.

You can view the page with your web browser at:
http://www.mysite.com/traffic/awstats.mysite.200511.html

Furthermore, you can create supplemental reports (lengthy lists of referrers, countries etc.) or even apply filters. This info is covered in detail in the awstats documentation. See the relevant section here.

A quick way to create full reports (main and supplemental pages) is to use the helper script, awstats_buildstaticpages.pl. This can be used in the following way:

# perl /home/jsmith/awstats/tools/awstats_buildstaticpages.pl -configdir=/home/jsmith/awstats/wwwroot/cgi-bin -config=mysite -awstatsprog=/home/jsmith/awstats/wwwroot/cgi-bin/awstats.pl -dir=/home/jsmith/awstats/statdata -month=11 -year=2005 -builddate=200511

Here is an explanation for some of the options:
-configdir: Sets the path of the directory which contains the configuration files.
-awstatsprog: Sets the path to the awstats.pl script.
-dir: Sets the directory where the report files should be saved.
-builddate: Adds month and year info in the report’s filename.

Again, if the options -month and -year are omitted, then the report is generated for the current month and year.

Other options that can be used are:
-update: Updates the awstats statistics database before generating any reports.
-buildpdf: Creates a PDF file, after the generation of the HTML pages is done.

In order to generate PDF files, the package htmldoc needs to be installed in the system.

It would be more convenient if you set cron to execute the above command.

AWstats Extra Section Configuration

AWstats can be configured to include user-defined charts in the reports. These are defined in the "Extra Section" in the awstats.mysite.conf file. An explanation for each directive is included withing the conf file. Here I provide two examples that work together with some notes, just to get you started with custom charts.

Keep in mind the following two things:

1. Every time you define a new extra chart, you have to increment the number in the name of each directive. For example, for the first extra chart the directive that defines the chart’s name would be ExtraSectionName1, for the second extra chart it would be ExtraSectionName2 etc.
2. Every time you define a new extra chart, but you want it to include info from already parsed log files, you have to recreate the awstats historical statistical data. You can simply delete the contents of the /home/jsmith/awstats/statdata directory and parse all your log files again.

At least a basic knowledge of Regular Expressions is required in order to configure extra charts.

Top 50 RPM Downloads

This user-defined chart displays the Top 50 RPM package downloads (used for the current web site):

ExtraSectionName1="Top 50 RPM Downloads"
ExtraSectionCodeFilter1="200 304"
ExtraSectionCondition1=""
ExtraSectionFirstColumnTitle1="Package Name"
ExtraSectionFirstColumnValues1="URL,\/packages\/(.*)\.rcn.*\.rpm$"
ExtraSectionFirstColumnFormat1="%s"
ExtraSectionStatTypes1=HB
ExtraSectionAddAverageRow1=0
ExtraSectionAddSumRow1=1
MaxNbOfExtra1=50
MinHitExtra1=1

Top 100 Referrers by Domain

This user-defined chart displays the Top 100 Referrers by Domain. It also merges referrer URLs of the form www.domain.com and domain.com to just domain.com.

ExtraSectionName2="Top 100 Referrers by Domain"
ExtraSectionCodeFilter2="200 304"
ExtraSectionCondition2=""
ExtraSectionFirstColumnTitle2="Referring Domain"
ExtraSectionFirstColumnValues2="REFERER,^http:\/\/www\.([^\/]+)\/||REFERER,^http:\/\/([^\/]+)\/"
ExtraSectionFirstColumnFormat2="%s"
ExtraSectionStatTypes2=PHBL
ExtraSectionAddAverageRow2=0
ExtraSectionAddSumRow2=1
MaxNbOfExtra2=100
MinHitExtra2=1

Some notes

User-defined charts add much more flexibility to AWstats. Sometimes, even non-professional webmasters need to "dig" into the server logs for some special info about their web site. This can be perfectly achieved by using custom scripts, but the extra charts are a better way of doing this.

Three are the most important directives in the extra chart configuration:

• ExtraSectionCodeFilter: This filters the log entries according to the HTTP code that the web server returned after a page or file request.
• ExtraSectionCondition: With this we can set some rules that define which entries will pass or not. The rules are of the form "URL, regular expression" and they can be separated with "||", which means "OR". Instead of the URL field, other fields like the User Agent string or the Referrer URL can be checked. These are documented in the configuration file’s comments. This directive can be left blank
• ExtraSectionFirstColumnValues: This defines what is the value that will be displayed in the custom chart. This is the same as the ExtraSectionCondition, but it could be considered as a third level of filtering. This directive cannot be left blank. An important thing to take a note of is that you need to specify a group in the regular expression. This means that a part or all of the regular expression must be in parenthesis. Whatever this group matches will be the value in the chart.

It’s clear that the knowledge of regular expressions is the absolute key in configuring an extra chart. This document is not intended to be a REGEX guide. I am not an expert on this anyway, so it would be pointless. Some helpful links can be found in the "Further Reading" section of this document.

Apache Configuration (optional)

Using this AWstats installation and configuration guide, there is no need for any special configuration at the web server level.

But, if you have not created the directory that holds the traffic reports (/home/jsmith/public_html/traffic) inside your DocumentRoot, then adding an Alias in your Apache VirtualHost configuration is necessary. For example, if you have created the traffic directory in /home/jsmith/traffic, then the following Alias must be added in your Apache or Virtual Host configuration file, so that the reports are accessible from a web browser:

Alias /traffic /home/jsmith/traffic
<Directory /home/jsmith/traffic>
	AllowOverride AuthConfig
	Options None
</Directory>

Access control directives can be added inside the <Directory> tags or in an .htaccess file, but this will not be covered in this document.

On the other hand, if you want to use awstats.pl as a CGI script in order to create the traffic reports dynamically from the web browser, then the addition of a ScriptAlias in your Apache or Virtual Host configuration is necessary. Assuming that you have followed the custom installation instructions of this guide, then this ScriptAlias could be:

ScriptAlias /traffic-bin/ "/home/jsmith/awstats/wwwroot/cgi-bin/"
<Directory "/home/jsmith/awstats/wwwroot/cgi-bin">
	AllowOverride None
	Options None
	Order allow,deny
	Allow from all
</Directory>

Now, point your web browser at:

http://www.mysite.com/traffic-bin/awstats.pl?config=mysite

All the awstats.pl options, except for -staticlinks, are supported, so you can try the following:

http://www.mysite.com/traffic-bin/awstats.pl?config=mysite&month=08&year=2005

Using awstats.pl as a CGI script, the reports are created in real-time from the statistical data, so it might be slow. This adds unnecessary load on the server. Furthermore, AWstats had some security related issues in the past, so using it as a CGI script is not recommended, unless you are sure that these problems have been solved or you implement access restrictions.

Further Reading

Here are some documents you might find useful:

1. The AWstats Documentation
2. The AWstats FAQ
3. HTTP Status Code Definitions
4. Regular Expression HOWTO
5. Syntax of regular expressions in Perl (Perl Documentation)
6. How to build HTMLDOC RPM package for Fedora Core by Thomas Chung

Related Articles

• Track ‘em Down!
• Maxmind’s GeoIP.dat.gz location change
• Mass download
• Check Server HTTP Headers with CURL
• Red Hat RPM Guide

*.*                /dev/tty9

Remember to use tabs instead of spaces. Then save and restart the syslog service:

# service syslog restart

From now on, by pressing Ctrl-Alt-F9 you can watch all the info that goes to syslog in real-time.

Related Articles

• Creative Commons v3.0 Licenses Launched
• How to integrate seaudit-report in logwatch
• Logwatch and Dovecot 1.x series in FC5
• Namespace Declarations With cElementTree
• Bot-Allow-Content and CC-Configurator plugin updates

# ps ax | less

See a list of all the running processes using BSD syntax:

# ps aux | less

See real-time information about running processes:

# top

Pressing the L, T, M keys while top is running you can toggle the display of Average Load, CPU and Memory usage respectively. The Space key refreshes the display and Q key quits the program. There are a lot more, so you should check the man page.

Memory and SWAP space usage:

# free -m

Use the -m parameter to show the amount of memory in MB instead of bytes.

Check if a program is running:

# ps ax | grep PROGRAM | grep -v grep

Show filesystem and disk space usage:

# df -h

Show disk usage under the current directory

# du -hs

The -h parameter shows sizes in human readable formats (eg MB, GB) and the -s gives only a total. If it’s not used, then information about each subdirectory is shown.

There are a lot more. I’ll add more commands to this post in a future update. Stay tuned!

Related Articles

• Change the console resolution
• Epiphany Python Console – Open New Tab
• Change console font in Fedora
• Making a directory writable by the webserver
• Search for a string in multiple files