hCaptcha is a reCAPTCHA clone that has been growing in popularity over 2020 and 2021, in particular due to Cloudflare’s conversion of their nag screens from Google’s reCAPTCHA to hCaptcha. Although hCaptcha advertises itself as being a privacy-conscious alternative to reCAPTCHA, there’s also an incentive for websites to switch over: hCaptcha will pay websites each time one of their users completes a hCaptcha challenge.

<p>hCaptcha is a reCAPTCHA clone that has been growing in popularity over 2020 and 2021, in particular due to Cloudflare’s conversion of their nag screens from Google’s reCAPTCHA to hCaptcha. Although hCaptcha advertises itself as being a privacy-conscious alternative to reCAPTCHA, there’s also an incentive for websites to switch over: hCaptcha will pay websites each time one of their users completes a hCaptcha challenge.</p> <p>Now the question is: how does you completing a captcha earn anyone money? Of course, hCaptcha is a VC-funded business, so it can afford to burn money in the pursuit of market share; nonetheless there needs to be a plausible business model there, and it’s not obvious at first sight.</p> <p>If you read the <a href="https://www.hcaptcha.com">hCaptcha website</a>, they suggest that AI startups will pay them to label their images for them. <sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> Labelling images is a labour-intensive task and required for some current-generation machine learning approaches. AI startups are well-funded and have money to spend on labelling, so this sounds like a reasonable case of selling shovels during a gold rush. But the output from solving CAPTCHAs isn’t obviously isomorphic to the type of labelling required for machine learning, which is often quite specific and requires a very low error rate.</p> <p>Complex CAPTCHA challenges are not possible, as web users turn out to be drunk, blind, 3 years old, or just randomly clicking buttons to get this infernal thing to go away. Accordingly, hCaptcha challenges are simple: select the images that match a simple 1-3 word prompt from a 3x3 grid. This is fortunately easy for most real people. <sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup> <sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup></p> <p>The most common prompts seem to be selecting buses, trucks, boats or trains out of the grid.<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">4</a></sup> The market demand for this sort of simple labelling must be rather limited, even if challenges have to be repeated many times and cross-checked to get an acceptable error rate.</p> <p>So far, a little inscrutable but all seems sensible enough. But then it all gets interesting when you actually take a look at the images in a little more detail:</p> <p><img src="https://bearbin.net/images/captcha/1.png" alt="hCaptcha example" /></p> <p>Starting from the top left and going right, we have:</p> <ul> <li>A boat that appears to have been painted by Dalí, with a mast drooping like a wet noodle.</li> <li>A plane with tricycle landing gear, except it’s got two sets of wheels at the front and one at the back. That’s not normal!</li> <li>A normal looking plane with some odd-looking clouds above.</li> <li>A bus with an axle in front of the door, and another behind it, and another at the back. Hmm</li> <li>A boat in a marina made of splodges.</li> <li>A normal-looking boat on a normal-looking sea, except - look at that horizon! How did that happen.</li> <li>A single-decker london bus with a ghost of it’s double-decker cousin above. And a giant moth perched on it at the back.</li> <li>Another ghostly upper deck on a regional bus.</li> <li>A sailing boat with some oddly stylised “alien” writing on the sail.</li> </ul> <p>These images are obviously AI-generated. They have all the hallmarks of GAN output, with typical artifacts and oddities. <a href="https://bearbin.net/images/captcha/2.png">Have</a> <a href="https://bearbin.net/images/captcha/3.png">some</a> <a href="https://bearbin.net/images/captcha/4.png">more</a> and see if you can spot the same things in these other challenges - it’s not hard at all, is it!</p> <p>The question then is why? Why would hCaptcha be generating these challenges - aren’t they supposed to be labelling real life, not some AI mirages? You know the labels before you generate them, what’s the point in using humans to re-label them again… And why are the results so bad - these are definitely not state of the art!</p> <p>The only explanation that makes sense is that hCaptcha is not really doing this whole AI-labelling business at all, or if they are it’s only in very limited fashion. Most of the time they’re just using a GAN to generate images that defeat the bots’ image recognition AI. And the GAN isn’t trained to optimise human recognition, rather to confound the bots in an arms race, leading to the bad image quality.</p> <p>If you have any better ideas I’d be glad to hear them because this whole thing doesn’t really make much sense.</p> <p>Footnotes:</p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1" role="doc-endnote"> <p>If you look closer, they have an <a href="https://medium.com/hcaptcha-blog/hcaptcha-technical-architecture-high-level-design-4373a8c944b2">article that purports to explain the “technical architecture of hCaptcha”</a> which is a supreme example of buzzword-stuffing blockchain-washed nothing. There is less than zero need for a blockchain to track customer requests, much less the public Ethereum blockchain, but it’s the buzzword of the month so it must go in. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2" role="doc-endnote"> <p><em>Most</em> real users, that is. There are some users for whom the challenge is actually too hard, or who’ve been blackholed and are interpreting bad IP reputation as poor skill. But the ones who fall down most often are those who try too hard and analyse the prompt and challenge in too much detail. The real way to solve these image challenges is to answer what you think <em>other people will answer</em>, rather than the <em>correct answer</em>. And don’t take too long either, just a quick glance is all your competition are giving! Anecdotally, this isn’t too common with hCaptcha, but reCAPTCHA challenges are extremely prone to this failure if you think too hard. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3" role="doc-endnote"> <p>Unfortunately <a href="http://dx.doi.org/10.1109/SPW53761.2021.00061">this is also quite easy for bots</a>, somewhat subverting the point of a CAPTCHA, so that’s how browser fingerprinting and IP reputation creep in to get reasonable enough results. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4" role="doc-endnote"> <p>These prompts are so common that a <a href="https://news.ycombinator.com/item?id=29838908">front-page post on Hacker News</a> consisted of this observation (and prompted me to write up my thoughts on the topic from the past few months). <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div>

Following on from my post yesterday about an edge case in YouTube, I thought I’d write about a class of edge cases perhaps even more strange that I’ve been exploring recently:

<p>Following on from <a href="drinking-from-the-firehose-youtube-music">my post yesterday about an edge case in YouTube</a>, I thought I’d write about a class of edge cases perhaps even more strange that I’ve been exploring recently:</p> <p>Search engines are a fact of daily life for most of the population nowadays. Google (sub your preferred provider) is an extension of the brain, imagined as giving you access to the sum of the world’s information at the click of a button. But a search engine isn’t just a Ctrl-F for the internet with a nice interface and ads; rather it’s a tremendously complicated system with lots of features and interactions between those features. And all you need to explore the system yourself is some well-tuned search queries.</p> <p>I recently had an epiphany: search engines are designed to find you results for <em>something</em> and that’s a job they perform well. But there’s nothing stopping you from searching for <em>nothing</em>! And the search engines will still give you results!</p> <p>And what results they are - have a go on the links below:</p> <p>An empty query on DDG: <a href="https://duckduckgo.com/?q=%2B&quot;&quot;">https://duckduckgo.com/?q=+””</a><br /> A different empty query on DDG: <a href="https://duckduckgo.com/?q=(&quot;&quot;)">https://duckduckgo.com/?q=(“”)</a><br /> An empty query on Google: <a href="https://www.google.com/search?q=(&quot;&quot;)">https://www.google.com/search?q=(“”)</a><br /> An empty query on Google News: <a href="https://www.google.com/search?q=%22%22&amp;tbm=nws">https://www.google.com/search?q=”“&amp;tbm=nws</a></p> <p>And have you ever thought about doing an <em>anything but</em> search? Normally you can add negations to the end of your search term to remove unwanted results, but there’s nothing stopping you from having a search term consisting entirely of negations!</p> <p>Here’s one on DDG: <a href="https://duckduckgo.com/?q=-&quot;an+entirely+negated+query&quot;">https://duckduckgo.com/?q=-“an entirely negated query”</a><br /> On Bing: <a href="https://www.bing.com/search?q=-%22an%2Bentirely%2Bnegated%2Bquery%22">https://www.bing.com/search?q=-“an entirely negated query”</a><br /> And on Google Books: <a href="https://www.google.com/search?q=-%22nothing%2Bto%2Bsee%2Bhere%22&amp;tbm=bks">https://www.google.com/search?q=-“nothing to see here”&amp;tbm=bks</a></p> <h2 id="commentary">Commentary</h2> <p>Google appears to have some half-effective filtering for these empty search queries so you’ll mostly get the same two YouTube videos as a result - is this an Easter egg? Although Google News and Books don’t have any filter, and you do get some odd results there!</p> <p>DuckDuckGo doesn’t appear to have any filtering at all, although it’s obvious just how much DDG relies on Bing’s whitelabel product for its results by looking at how similar the two are.</p> <p>If you can think of a deeper reason for these results, please do leave a comment and lets try and explain some of the mystery away.</p>