Are Bots Good Or Bad For Online Business?

Bots tend to get a lot of bad press. And considering that bots tend to be involved in all kinds of malicious internet activity – including devastating DDoS attacks – it isn’t altogether unwarranted. However, the actual story behind bots isn’t quite so one-sided.

Robot image from Shutterstock

Think of your website as a garden. When you’re in the throes of a hairless caterpillar infestation, you might toy with the idea of blasting your garden with the most potent chemical pesticide available, thereby banishing all insects from your plants. But what about all the beneficial insects? What about bees? Your garden won’t thrive without them.

Before you go banishing all bots from your website you need to be aware of just how many good bots there are roaming the internet. Keep reading for the details on good bots, bad bots, and how you can keep the former coming to your site while keeping the latter away.

Lots of bots

A bot is simply a software application that runs automated tasks (scripts) over the internet. As much as you may think internet traffic is primarily composed of Redditors and people watching cat gifs, in actuality a full 48.5% of all internet traffic comes from bots. Of that 48.5%, 19.5% are good bots while the remaining 29% are bad.

For small and medium websites (small being websites receiving up to 1000 visitors per day, and medium receiving up to 10,000 visitors per day) the breakdown skews even more heavily in bots’ favor. On small websites, 85.4% of traffic comes from bots. Medium websites fare a bit better, with 71.1% of traffic being accounted for by bots.

What the good guys are up to

The reason you can’t just set up website security that tells all bots to hit the bricks is that good bots are essentially responsible for keeping the internet running in good order. Good bots keep search engines operating (and keep your website in those search engine results), power APIs, monitor websites and scan for vulnerabilities, among other duties.

One of the main problems with bad bots stems from the fact that you (and every other website owner) obviously wants good bots on their website. Website security is designed to allow good bots through to your site so they can go about their benevolent business. This in and of itself is fine. What isn’t fine is the way cyber criminals can work this to their advantage with bad bots.

Bad bot types

Of the 29% of internet traffic made up of bad bots, a full 24.5% are what’s known as impostor bots – malicious bots pretending to be good bots like Googlebots, attempting to take advantage of the access granted to good bots. Advanced malicious bots are also able to pass themselves off as human visitors, bypassing security that would otherwise bounce the baddies.

Other types of bad bots include scrapers and spammers. Scrapers are engineered to steal information like pricing and other original content from websites in order to post it on another website. They tend to run rampant on ecommerce sites. Spambots are more likely to be seen on blogs and other content websites, filling up comment sections with gibberish and links to other sites in the hopes of getting a website’s visitors to click on the links.

Spambots can also be used to artificially boost a website’s SEO and other traffic-related statistics. This will more often than not result in the website in question being penalized in search engine rankings and can even result in the site being blacklisted or removed from search results.

What bad bots are doing when they aren’t scraping and spamming

As bad as content scraping and spamming can be, it isn’t the worst of what bad bots get up to. Malicious bots are also used as hacking tools and can be used in intrusions that steal confidential data, such as your customers’ financial information, or your intellectual property.

Bad bots are also a major driving force behind DDoS attacks, the distributed denial of service attacks that seek to overwhelm your website with malicious traffic in order to knock it offline or slow it down so much that it’s virtually unusable. But the consequences of a DDoS attack extend far beyond an unavailable website. These attacks not only cause you to lose short-term business, but can do long-term damage to your reputation and credibility, result in major bills from your hosting provider, and can even compromise your data or damage your hardware, software and other aspects of your business.

Bad bots becoming increasingly brilliant

Internet security is essentially a high-stakes game of chess between cyber criminals and security firms, with each group trying to stay one step ahead of the other. As such, bad bots keep getting smarter, sneakier and all the more lethal.

Bad bots are now more likely than ever to successfully pass themselves off as human visitors by mimicking human behavior. This is thanks to an increase in sophistication from the developers behind the bots. Internet security firm Incapsula has noted that over 50% of the DDoS bots they’ve seen recently support cookies and JavaScript, a decidedly new development.

Incapsula has also observed a large step up in bot activity in the form of massive application layer DDoS attacks that are ranging in the hundreds of thousands of requests per second from 10,000 to hundreds of thousands of IP addresses, sometimes supporting cookies or JavaScript, as previously mentioned.

2015 Global Bot traffic report summery [Source: Incapsula]

Saying goodbye to bad bots

Security that protects your site against bad bots while letting good bots and actual human visitors through unfettered has to take a multilateral approach to analysing and identifying your website visitors.

This multilateral approach has to include the use of a static analysis tool that can quickly determine a bot’s true identity by examining structural web requests as well as header information, a challenge-based approach that makes use of proactive security components like CAPTCHA requests, and a behavioral-based approach that examines a bot’s activity to see if the bot has characteristics that vary from its parent program, as any anomalies may point to malicious activity.

Essentially, effective bot security will analyse every single visitor to your website to understand exactly what it’s dealing with in order to let human visitors and good bots through while blocking bad bots from your site. Just like your dream garden pest control would inspect all incoming insects to bounce the mealworms and welcome the bees, resulting in a growing season where you would not throw an entire chewed up tomato plant in anguish.