Honeytrap Module Guide

WARNING: When set up correctly with a firewall or similar this module will slow down and block traffic to your website. This potentially could include yourself, website owners, maintainers and legitimate visitors to your website. If you use this module then you do so at your own risk.

Summary

The Honeytrap module allows site owners and system administrators to monitor web crawlers that do not follow the rules set out in the robot.txt file or via the RobotsText module or similar method and as a result put an unnecessarily high load on servers.

This is especially important for large, very high traffic and/or high profile sites where the activity of these non-compliant crawlers can bring servers to their knees. If these crawlers are not blocked or slowed down quickly enough then these crawlers can result in servers being knocked completely offline.

Note:

The Honeytrap module does not directly block or slow down offending IP addresses itself; it only logs and reports them, leaving you in full control of how you want to deal with them.

Requirements

Views 2 module
A configurable firewall (ideally)
A running Drupal cron task (ideally)
RobotsTxt module or access to the robots.txt file.
This module will not currently work on a Windows environment because it assumes that the '/' character separates path components.

Installation

Install as usual, see drupal.org/node/70151 for further information.

List of Terms

To make things as clear as possible I have included a list of common terms that are used in this guide.

Crawlers - A web crawler is a computer program that browses web sites in a methodical, automated manner. This process is called web crawling or spidering. Crawlers are also referred to as "web crawlers", "spiders", "bots", "ants" "indexers", "automatic indexers" and "web robots".
robots.txt - A file that web site owners can use to give instructions about their site to crawlers. You may want to use the RobotsTxt module rather than creating your own robot.txt file. Amongst other things this file can be used to "Disallow" crawler access to certain urls or folders and to set a "Crawl-delay" to specify the maximum rate that a crawler should load pages from the site.See www.robotstxt.org for a good explanation.
Compliant crawler - A crawler that obeys the Robot Exclusion Standard. Amongst other things these crawlers obey various entries in the robots.txt file.
Non-compliant crawler - These crawlers do not obey the Robot Exclusion Standard and ignore the robot.txt entries including the "Disallow" paths and more importatly the "Crawl-delay" entry which is used to reduce crawler load on a website's infrastructure. These are also referred to as "bad crawlers" and similar terms in this guide.
Trap - The trap is a link on the website that is hidden from normal users and compliant crawlers. These traps are used by the Honeytrap module to monitor for any non-compliant crawlers that are likely to be putting an unnecessarily high load on the web server(s) and supporting infrastructure.
Firewall - A firewall is a device or set of devices designed to permit, deny or restrict access to a network/website based on a set of rules. For more on information on firewalls see http://en.wikipedia.org/wiki/Firewall_(computing).
List - A group of IP addresses created by the Honeytrap module. These list can be read by the firewall system for use with its access rules.

Configuration

Configure the following Honeytrap user permissions in Administration » People » Permissions » Honeytrap module:
- edit honeytraps - Users in roles with the "edit Honeytraps" permission will be able to see and alter the Honeytrap settings via Administration » Site configuration » Honeytrap
- view honeytrap lists - Users with this permission can view the Honeytrap lists via dedicated urls of the format Honeytrap/list/list_name (e.g. the black list can be seen at Honeytrap/list/black). Users in roles with the "Access administration menu" permission may also view the Honeytrap lists on the admin pages under the "View lists" tab.
If configured, ensure that your web service has read and write access to the location of the list files:
- List files - The list files can either be read from urls of the format honeytrap/lists/list_name or from the files which contain the lists. These files allow other processes such as firewalls to be instantly updated. If you enable the list file functionality then you must ensure that your web service has write access to these folders.
If you don't have the RobotsTxt module enabled then ensure that you add the correct entries to your robots.txt module as indicated on the settings page of the Honeytrap module.

Customization

To alter the output format of the Honeytrap lists you will need to define your own honeytrap_list_item theme.
Lots of other settings can be altered on the administration settings tab for the Honeytrap module. This tab can be found at admin/settings/honeytrap/settings.

Suggested Usage

The Honeytrap module makes use of three lists. For automatic and optimal site performance these lists should be used in conjunction with a firewall or similar system as follows:

naughty list - Firewalls should throttle IP addresses on this list to slow down their access to your site. Items that trigger a trap are added to the naughty list unless they are on the black or white list.
black list - Firewalls should block IP addresses on this list. You can manually add or edit IP addresses to put them on the black list.
white list - Firewalls should ignore this list. You can manually add or edit IP addresses to put them on the white list. The white list is used to prevent the specified IP addresses from appearing on any other list. You should normally ensure that you add your own IP address to this list.

Tips

Make sure that you add a "Crawl-delay" entry to your robot.txt file. This can be used to slow down the crawl rates of compliant web crawlers.
Add your IP address to the Honeytrap white list.
Set up a firewall to enforce the Honeytrap's naughty and black lists.
Test your setup first by manually visiting a trap url then by using a real, non-compliant web crawler (see the FAQ section).

Troubleshooting

Below is a list of common problems that you may encounter. Each problem is followed by a list of likely causes which should help you to resolve the problem.

Items on the naughty list are never expiring

You have not set the auto expiry time on the settings tab
The cron task is not running
You have specified a cron whitelist which does not include the cron tasks IP
The IP was not automatically added, it was manually added or edited

Items on the naughty list are expiring, but later than I expected

If this is an issue then you need to run the cron task more often

Everything seem to be hitting my traps, even things like googlebot

The entries in your robot.txt entries do not match the Honeytrap settings
The disallowed entries in your robot.txt file are not correct

A Yahoo crawler is hitting my traps

Some of the Yahoo crawlers are reportedly ignoring the robot.txt file. Try searching for "yahoo ignoring robot.txt".

Addresses on the black list are not getting blocked

Your firewall is not set up to block items on the black list
The firewall is not reading the black list file
The firewall has not read the black list file yet

Addresses on the naughty list are not getting throttled

Your firewall is not set up to throttle items on the naughty list
The firewall is not reading the naughty list file
The firewall has not read the naughty list file yet

Addresses on the naughty list are getting blocked rather than throttled

Your firewall is set up to black list items on the naughty list
Your firewall is using the naughty list for the black list

FAQs

Q: How can I test that my traps work?

A: The easiest way to test your traps is simply by visiting a trap in the browser. You will find an example URL for a trap on the Settings tab. Once you are happy with this you can carry out more "real life" tests using something like "wget" or by downloading a web crawler that will ignore the robot.txt file.

WARNING: Be careful that you don't block yourself, especially if you have a firewall fully connected up to the Honeytrap. This is even more important if you only have access to your site via a single IP address. I suggest that you add your IP address to the white list first before carrying out any of these tests. You will be able to monitor hits to the trap urls via the watchog log even if your IP address is on the white list.

I can see IP addresses appearing on the naughty/black lists but they are not getting blocked, why?

A: Have you set up your firewall to read in the list files created by the Honeytrap? If so, check that the files exist where you expect them and that they contain the anticipated IP addresses. If they do, then check your firewall setup is correct and that it is actually reading the files. Also check that the list file format is what your firewall expects.

CREDITS

Created by:

Mike Jessop a.k.a. Mikey Bunny (don't ask)