What is a bot?
A bot is a software program designed to automate specific, repetitive tasks. Bots range from helpful to malicious: search engines use bots to crawl the web and add new pages to their index; businesses use chatbots to answer customer questions; and attackers use bots to brute-force logins or spread spam.
More about bots
A bot (short for robot) is any program that performs specific, repetitive tasks automatically, and the category is broad: search crawlers, chatbots, uptime monitors, spam scripts, and attack tools are all bots. Bots generate more traffic than people do. The Thales 2026 Bad Bot Report measured automated traffic at 53% of all internet traffic in 2025, with 40% coming from bad bots and 13% from good ones.
Search engines use bots such as Googlebot to discover new pages and add them to the search index, businesses use chatbots to answer customer questions, and attackers use bots to break into websites.
How bots work
A bot is a script or program that sends automated requests over the internet, running around the clock at machine speed and scale far beyond a human. A request may carry a user-agent string, a short label that tells the server what software is asking, but that header is optional and easy to fake. Honest bots announce themselves in it, and well-behaved ones check your robots.txt file before fetching a page and skip whatever it disallows. Dishonest bots send a browser’s user-agent instead, which is why reliable bot detection never trusts the label alone: it also weighs request rate, the paths being requested, the IP addresses and networks the traffic comes from, the published IP ranges of verified bots, and behavior over time.
Search engine crawlers
A crawler (also called a spider) is one kind of bot, and it’s how search engines find content. The bot starts from pages it already knows, follows every link it finds while crawling, and sends each new page back for indexing. Those visits consume your crawl budget, the number of pages a search engine will fetch from your site in a given period. Blocking a crawler stops it from discovering new pages and refreshing the ones it already knows, but a URL that’s already in the index can stay indexed anyway. To keep a page out of search results, leave it crawlable so the crawler can see your noindex tag, and use the search engine’s removal tool for urgent, temporary removal.
Chatbots and AI bots
Businesses use chatbots to answer customer questions automatically. Simple rule-based bots can’t handle complex queries: they serve pre-written responses, point customers to knowledge base entries, and pass harder tickets to human support agents. Modern AI-powered chatbots go further and use natural language processing (NLP) to understand a question and answer it directly.
AI has also created a new category of bot on the open web. AI crawlers and fetchers gather content for model training and for answering user prompts, while AI agents browse and act on a user’s behalf. The Thales report now counts AI agents as a third category of automated traffic alongside good and bad bots, and it measured AI-driven bot activity growing more than tenfold (12.5×) in 2025. Allowing or blocking AI crawlers is your call as a site owner, and you typically make it with rules in robots.txt.
Good bots vs. bad bots
The dividing line is behavior, and you can usually see it in your server logs:
- Good bots (search crawlers, uptime monitors, feed fetchers) identify themselves in their user-agent string and obey robots.txt.
- Bad bots (content scrapers, spambots, credential-stuffing bots) disguise themselves as regular browsers and ignore robots.txt entirely.
Bad bots often spread as malware: infected devices are pooled into botnets that brute-force logins, stuff stolen usernames and passwords into sign-in forms, or flood a site with junk traffic in a DDoS attack. Treat the two kinds differently. Blocking good crawlers costs you search visibility; tolerating bad ones costs you bandwidth and security.
Managing bot traffic on your site
Welcome the good bots: let search crawlers reach the pages you want indexed. Manage the bad ones with rate limiting, CAPTCHAs, or a web application firewall, because robots.txt is voluntary and bad bots ignore it. Watch for the warning signs: bandwidth spikes and login pages hammered with failed attempts. For step-by-step help, see our guides on preparing your site for bot traffic and stopping crawlers from slowing down your website.
Frequently Asked Questions
How can you tell if website traffic is from a bot?
Check your server logs for bot user-agent strings, sudden traffic spikes with near-zero engagement, and requests arriving faster than any human could click. Logs alone aren’t proof, though: sophisticated bad bots impersonate real browsers, most commonly Chrome.
Are bots illegal?
No. Bots are legal tools; what they do determines legality. Crawling public pages is generally lawful, while credential stuffing, ad fraud, and ticket-scalping bots are illegal in many jurisdictions. If you run bots, respect each site’s robots.txt rules and terms of service.
What does bot mean on social media?
On social media, a bot is an automated account that posts, likes, follows, or replies without a human behind it, often used to spread spam, scams, and fake engagement. It’s a different context from web crawlers, but the same underlying idea: automated software acting at scale.
