Are CAPTCHAs Still Bot-hard? Generalized Visual CAPTCHA Solving with Agentic Vision Language Model

We infiltrated a human-driven CAPTCHA farm to evaluate Halligan's ability to generalize to unforeseen visual CAPTCHAs. We selected 2Captcha as our target for three reasons: (1) it allows anyone to join as a worker; (2) it ranks as the second-most popular CAPTCHA-solving service according to domain popularity metrics, enabling us to collect more data; and (3) its API documents solvers for many CAPTCHA services, increasing our likelihood of encountering new types of CAPTCHAs.

Using 2Captcha's Windows software, we created a throwaway account and deployed Halligan to interact with tasks. Additionally, we performed a man-in-the-middle (MITM) attack using mitmproxy to intercept the communication between the worker software and 2Captcha's server, which gives us more insight on each task. Below is an example of the intercepted data from an incoming CAPTCHA task:

1. data contains all task-related information, including images, the sitekey, and the target website URL, necessary to load the visual CAPTCHA.
2. proxy and cookie contain spoofing-related data that enables workers to solve tasks using the same identity as the site visitor (task requester or customer). This approach helps bypass behavioral and IP-based checks, increasing the task success rate.
3. reputation, balance, simple_solving are internal metrics that reflect the current worker's performance and status.

        {
          "status": 1,
          "saverecaptcha": 0,
          "data": {
            "captcha_id": 228766692,
            "pageurl": "...",
            "sitekey": "...",
            "rate": 0.001,
            "useragent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
            "timeout": 120,
            "captchatype": "prosopo",
            "id_dup": 0
          },
          "salt": false,
          "thread": 0,
          "proxy_ip": "",
          "proxy_port": "",
          "proxy_user": "",
          "proxy_pass": "",
          "proxy_type": "HTTPS",
          "necessarilyproxy": 0,
          "necessarilyuseragent": 0,
          "send_screenshot": 0,
          "screenshot_interval": 0,
          "user_cookie": {
            "status": 0,
            "cookie_id": 0,
            "cookie": ""
          },
          "reputation": 966,
          "simple_solving_percentage": 83,
          "simple_solving_accuracy": 85,
          "balance": 0.54063000002375,
          "sandbox": 0,
          "time": 0.072
        }
      

The figure below illustrates the distribution of tasks by CAPTCHA service over the course of the study. Hover over any area to see a detailed breakdown of the numbers, and hover over the legend to highlight specific sections. To zoom in on a particular period, click and drag; double-click anywhere to reset the zoom to its default view.

The figure below displays the total number of tasks, grouped by CAPTCHA service and further divided by CAPTCHA type. Four out of the nine services (xcaptcha, prosopo, amazon, and 2captcha), representing 17 of the 26 CAPTCHA types, are not included in the benchmark. Hover over the figure for more details.

The figure below displays the target domains, organized by CAPTCHA service. Please note that the intercepted data may be incomplete, and some services (e.g., xcaptcha, 2captcha, mtcaptcha) do not track the page URL, so the data should be considered an estimate. Arkose has the highest concentration of popular domains, with all ranked within the top 2000 and over half in the top 500. Amazon has a large portion of government-related domains, while geetest has a large portion of finance and cryptocurrency-related domains.