Fingerprint surface-based detection of web bot detectors

This website provides additional materials to our research published at ESORICS 2019. A preprint version is available here. This project aims to explore web bot detection based on technical properties by fingerprinting bots.

We base our research on a reverse analysis of a commercial bot detector. This showed that unique properties web clients exist which that sufficient to identify a web bot. We developed a more general approach to find system properties of web bots that allow us to distinguish these from human-controlled web clients. Our developed methodology comprises browser fingerprinting and comparisons of resulting fingerprints of a regular browser and web bot frameworks that belong to the same browser family. The derived fingerprint surfaces of web bots were used to conduct the first measurement of the prevalence of web bot detection in the Web. Responsible for this project are: Responsible for this project are:

Gabry Vlot,
Hugo Jonker and
Benjamin Krumnow

Fingerprint surface of web bots

On following links we provide an overview of the setup of fingerprints we compared and specific details about deviations we found. We further share the complete fingerprints as database, which can be downloaded. Finally, our modified version of fingerprintJS2 and tools to compare fingerprints for deviations can be found on Github.

A detector for web bot detection

Based on our findings what makes web bots visible to websites, we developed a mechanism to find scripts that attempt to identify web bots. We build a scanner equipped with this mechanism to detect web bot detectors. In our study, we used this bot to scan the Alex Top 1M, which led to the identification of over 127K sites with detectors. Our modified version of the OpenWPM framework can be found under the following link:

Bot detector scanner