Dear all,

two days ago my forum was flooded with requests from the facebook-crawler. Likely someone linked to the forum on Facebook (can’t check cause I don’t have a FB account and no intentions to get one). First I saw only a bizarre high number of online ‘users’ and at the end my server gave up (likely due to too many database-connections) and responded with an HTTP 500 (Internal Server Error).

Search for the user-agent string facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) in your server’s access.log to check. My daily logs grew from ≈10MB to more than 150MB. A Google-search showed that I was not alone…
The crawler is aggressive and doesn’t give a shit about the robots.txt. Therefore,

User-agent: FacebookBot
Disallow: /

does not help.

Finally I used the workaround suggested at stackoverflow.