EQL Breaks Convention on Bot Mitigation: Why It Works

A curious thing happened in early 2023. Word was spreading around the sneaker community about a mythical shoe Nike was working on. Rumors swirled, and heat began to build. Then it happened.

LeBron James was spotted wearing the Tiffany & Co. Nike Air Force 1 to Madison Square Gardens. Carmelo Anthony flexed on Instagram, and what was already one of the most anticipated sneaker launches of the year turned into a massive. hyped. frenzy.

But when the shoes dropped online, the website running the launch didn’t crash. It didn’t throw fans into an endless queue. And no carts were robbed by bad actors…

This isn’t how the story normally goes.

Sneaker fans are fuelled by passion. They scour the internet for upcoming news of hot releases and congregate on forums to trade rumors. Some pay a monthly subscription to cook groups, communities, or apps to get early access to launch links and tips and tricks on how to maximize their chances of obtaining products before they sell out. When a big launch is announced, they set their alarm clocks and participate in a mad rush to enter as soon as it opens.

But unfortunately, they aren’t the only ones watching.

An entire army of sneaker bots are waiting in the wings. They scrape and monitor websites or various social media networks to detect when new launches are added, usually before they are publicly announced. They use browser automation to fill in shopping carts at superhuman speeds. They even leverage paid captcha solver services that pay cheap outsourced labor a fraction of a cent to “prove” that the bot is human. To put it bluntly, real fans don’t stand a chance.

For brands and retailers attempting to run these launches themselves… it’s utter chaos. The traffic load on their website goes from zero to half the internet and back down to zero again in the space of a few minutes. Running a shopping cart? It’s going to be nearly impossible to avoid overselling. Running a raffle? You’ll unfortunately likely find yourself wading through a spreadsheet of thousands (or even millions!) of entries trying to spot the fakes.

When humans and bots collide, things fall apart

Modern websites are built on cloud computing platforms, which have “rate limiting” that activate when they detect large or abnormal spikes in usage to prevent a single customer from overloading the service. The cruel irony is that these limits rear their head at precisely the worst moment - when the most eyeballs are watching the site.

It’s very hard for tech teams to predict the heat and simulate these traffic spikes, and service providers aren’t usually swayed by requests along the lines of “Please increase our rate limits by 1000x because we think we’re going to be popular”.

Worse still, every service has a “hard limit” that is often undocumented and no amount of begging and pleading can bypass. After a while, it starts to dawn on you that your e-commerce site simply wasn’t built for high-heat launches. It’s just not that common for a website to sign up thousands of users per second.

So how did the Tiffany x Nike AF1 launch stay up?

To answer that question we need to go back to the first launch that EQL ever ran, the Air Jordan I Mid SE Fearless Maison Château Rouge back in 2019. Full of enthusiasm and youthful naivety, we built a modern tech stack that adhered to all the industry best practices and watched as it crumbled to its knees in the first 5 minutes after launch because a third-party component couldn’t mint auth tokens fast enough. Thoroughly chastened, we went back to the drawing board and swore this would never happen again…

4 years later, we’ve managed to avoid a repeat occurrence, despite running an increasingly wide range of high-heat launches across sneakers, collectibles, art, whiskey, wine, golf accessories, apparel, and more.

So what’s the secret?

The first thing we did was to take a good hard look at every third-party dependency in our tech stack. Did we truly need it? Could we build and scale it to meet the unique demands of high-heat launches better ourselves? Over time as our company and expertise have grown, the answer to that question is increasingly yes.

Of course, no startup is an island - we still need technical partners. For the core dependencies that remain, we forged much deeper relationships with our service providers and made sure that they understood and were willing to support the unique demands of our platform.

Hot stuff

The first few minutes after a launch goes live is the critical moment where everything has the potential to go bananas. A key revelation was that whilst a huge amount of processing is required to make sure that the outcomes of a launch are fair (we’ll get to this in a moment), very little of this work needs to happen up-front.

We completely re-architected our system to be as lazy as possible - deferring **everything else to ensure that we reserve all of our critical processing power to keep those entries flowing in. Your parents were wrong - procrastination is a virtue!

We also baked in as many layers of resilience and reliability as possible, evolving our system so that it automatically detects when third-party systems go down and fills in the details later.

Thou shalt not pass

One of the great things about solving for massive scale is it allowed us to fully embrace a core tenet of our platform: never reject an entry. Even when it’s coming from a bot; especially when it’s coming from a bot.

This is the key to understanding how EQL works.

Silence is a virtue

We’re engaged in a never-ending battle of cat-and-mouse with ingenious bot makers who are trying to reverse engineer our system and constantly attempting all sorts of tricks to slip under the radar, and the last thing we want to do is give them a nice helpful “Gotcha!” signal every time they’re detected. By silently accepting their entry, all the signal they get is yet another L, with no clue as to whether they were detected or just unlucky.

A nice side-effect of accepting all those bot entries is we accumulate a comprehensive repository of bot attacks, which is a brilliant resource for improving the future bot defenses of our platform, which ultimately results in a fairer experience for real fans.

(Sorry botters, we appreciate the hustle but we gotta stand up for the fans.)

Embracing the grey zone

The old-school approach to handling bots is to stick a gatekeeper in front of your website and implement IP blocking to block anyone who looks slightly suspicious. The problem with this bot management approach is that you have to make a binary decision about whether to let someone in or not. If you’re too lenient, you let in all the bots. If you’re too strict, you’ll block real fans. Neither approach works well in practice.

EQL works differently: the more you try to game the system, the lower your chances of winning. Whereas if you play fair, we do everything we can to give you a fair chance of winning. This allows us to embrace the murky space between “obviously bad” and “obviously good” and just focus on making sure that real fans get the highest chance of winning.

So how do we detect bots?

We avoid hard and fast rules, such as “no more than 3 entries from a single IP address” because they’re ripe for abuse - botters are very good at figuring out out these sorts of filters and ensuring that they sit just below the threshold. Plus these rules always have the potential to unfairly penalise real fans. 50 entries from a single IP address could be a lazy botter, but it could also be a shopping mall’s public wifi, and everyone entering in-store is clipped... oops.

And then, there are those that spend hours manually submitting entries without using a bot, in the hope that this will help them fly under the radar. We catch them too, but not without shedding a tear for the futile human effort!

It’s worth keeping in mind that having a bot isn’t enough to game the system - attackers also need all their fake entries to fly under the radar. This is interesting because it gives us two chances to catch abuse - firstly by the presence of a bot, and secondly by detecting suspicious signals in the information that the user agents actually type into the entry form. Significantly, this is something that traditional anti-bot tools simply cannot do, because they don’t inspect user-submitted data.

If, like some sites out there, your anti-bot measure is just a human eye-balling a massive spreadsheet of entries, it’s relatively easy to fly under the radar. “Address jigging” is a well-known approach where you slightly alter (or deliberately misspell!) your address in ways that the postman will still deliver to your house.

Our internal fraud and abuse engine, lovingly known as Clippy has the job of putting all the clues together and making sure that real fans, the ones not trying to game the system, get the highest chance of winning (the meta-joke here is that Clippy doesn’t actually ‘clip’ anyone).

Clippy draws on a mind-bogglingly large number of signals to perform its assessment: it looks at classic old-school signals such as browser automation or whether an entry is coming from a known proxy IP address; it looks at the behavior of accounts over time; and it looks for suspicious patterns between entries to spot clusters originating from the same person trying to unfairly increase their chances of winning.

Clippy is an AI+human collab. We combine machine learning models with expert human oversight to ensure that all of those signals are interpreted in the fairest possible way, to spot anomalies, and to make sure we can rapidly adapt to new attacks. It’s been an insanely intricate process to build (which is why we can confidently tell you about it).

To paraphrase Gary Kasparov, the first chess grandmaster to famously lose to a computer chess engine, Artificial Intelligence might have shown that it can beat humans, but Augmented Intelligence, humans plus AI, is the ultimate combination.

Don’t feed the bots

Some may not know, but getting your hands on a bot can be pretty tricky – some of them are scarce products in their own right – and can cost hundreds or sometimes thousands of dollars. Cook groups teach their members how to use them, with the promise that "hey, look you can make money!" But if the learning curve is too steep you can always buy a slot from someone else who will run the bot for you - welcome to the bot gig economy.

It’s easy to get scammed when buying a bot too. There’s no better way to spot a bad bot than to see who advertises fake success after a big launch alongside a link to buy their bot license.

One of the cruel ironies of this space is that when a site is being cleaned out by bots, it encourages more bots to join the feeding frenzy, which results in even bigger traffic spikes and chaos for website operators. On the flip side, what we often see when we start working with new partners is a dramatic drop in their top-of-funnel web traffic once the bots pack up and go elsewhere. And in their wake, a joyful flood of real people celebrating first-time wins.

Putting it all together

EQL is commerce built for passion. We understand that the demand for these products (and trying inventive ways to get them) comes with the territory. Bots are one way people try to get their hands on the world’s most in-demand products – one that has been undeniably successful in the past but leads to unmitigated chaos for website operators and heartbreak for less savvy fans.

The end goal of our tech is to make launches more reliable, fair, and memorable, and we’re well on our way.