Other Crawlers
299 bots documented
Directory of specialized web crawlers. Research bots, data scrapers, and miscellaneous crawlers from various organizations and projects.
The web is crawled by many specialized bots beyond AI companies and search engines. These include research projects, data initiatives, and specialized services. Examples of specialized crawlers: - Common Crawl: Open web corpus for research - Archive bots: Web preservation projects - Academic crawlers: University research projects - Commercial scrapers: Various data companies Our other crawlers directory helps you identify and manage these specialized bots, ensuring you understand who is accessing your content and why.
All other crawlers bots
2ip bot
2ip.ru's miscellaneous & unknown
2ip.ru bot
2ip.ru's miscellaneous & unknown
360Spider
Qihoo 360 Technology Co.,Ltd's search engine crawlers
advanced_crawler
Unknown Author's miscellaneous & unknown
Adventurer
Unknown Author's miscellaneous & unknown
AGAKIDSBOT
AGAKIDS's miscellaneous & unknown
alienfarm
Unknown Author's miscellaneous & unknown
ALittle Client
Unknown Author's miscellaneous & unknown
All Africa Crawler
AllAfrica's content aggregators
AntBot
Unknown Author's miscellaneous & unknown
Aranea
Vladimír Benko's miscellaneous & unknown
Authory
Authory's content aggregators
BDBot
Unknown Author's miscellaneous & unknown
BeeperBot
Automattic Inc.'s miscellaneous & unknown
BestChange
BestChange's financial / cryptocurrency
Birdcrawlerbot
Swoppen Systems GmbH's miscellaneous & unknown
bitlybot
Bitly, Inc.'s miscellaneous & unknown
BotPoke
Unknown Author's miscellaneous & unknown
Brave
Brave Software, Inc.'s search engine crawlers
Bushbaby
Unknown / Unclear's miscellaneous & unknown
butterfly bot
Topsy Labs's miscellaneous & unknown
Cốc Cốc
Cốc Cốc Company Limited's search engine crawlers
centurybot9
Unknown Author's miscellaneous & unknown
certytags
Francesco Tacconi's miscellaneous & unknown
charlotte bot
SearchMe.com's miscellaneous & unknown
Chatwork crawler
Chatwork Co., Ltd.'s miscellaneous & unknown
Cincraw
Cincraw's miscellaneous & unknown
CirrusExplorer
Cirrus Europe SRL's miscellaneous & unknown
Cloudtrellis
Cloudtrellis Inc.'s miscellaneous & unknown
coccocbot-web
Cốc Cốc's search engine crawlers
coccocbot
Coc Coc Company Limited.'s search engine crawlers
Coveo
Coveo Solutions Inc.'s search engine crawler
Crawler
Unknown Author's miscellaneous & unknown
Crawlson
Crawlson's search engine crawler
Crissy Field bot
Crissy Field GmbH's miscellaneous & unknown
Critical CSS
Critical CSS's other
Cronless
Cronless's miscellaneous & unknown
crusty bot
Unknown's miscellaneous & unknown
crusty crawler
Sergey F's miscellaneous & unknown
Cxense
Cxense's search engine optimization
CXK_Bot
Unknown Author's miscellaneous & unknown
Dark Visitor Client
Dark Visitors's miscellaneous & unknown
Dataprovider.com
Dataprovider.com B.V.'s search engine crawler
Dawap bot
Dawap's miscellaneous & unknown
Deskyobot
Gloriot Stéphane's miscellaneous & unknown
Devin
Devin AI's ai assistant
DF Bot
Unkown Author's miscellaneous & unknown
DigiCert DCV
DigiCert, Inc.'s search engine optimization
DingTalkBot
Alibaba Group's miscellaneous & unknown
Dmbot
Unknown Author's miscellaneous & unknown
DnBCrawler
Unknown Author's miscellaneous & unknown
EasyBill Import Manager
easybill GmbH's other
Echobox
Echobox's ai crawler
Ecovadis bot
Ecovadis SAS's miscellaneous & unknown
edansbot
Unknown Author's miscellaneous & unknown
Edge Crawler
Viktor Lofgren's search engine crawlers
EFF crawler
Electronic Frontier Foundation's miscellaneous & unknown
Elastic crawler
Elasticsearch B.V.'s miscellaneous & unknown
Elementor bot
Elementor Ltd.'s miscellaneous & unknown
Elisabot
Unknown Author's miscellaneous & unknown
EMDASH bot
EMDASH (FONTS NINJA)'s miscellaneous & unknown
eMoney Advisor
eMoney Advisor, LLC's aggregator
Epivoz Crawler
Techmeme's aggregator
ev-crawler
Headline's miscellaneous & unknown
Ezgif
SIA OPEN IDEA's miscellaneous & unknown
Fake bot
Unknown Author's miscellaneous & unknown
FAST-WebCrawler
Unknown Author's search engine crawlers
favicon bot
Google's search engine crawlers
FedReporter
Fed Reporter, Inc.'s aggregator
fidget-spinner-bot
Unknown Author's miscellaneous & unknown
FirmoGraph
firmograph.io's miscellaneous & unknown
fluid
leak.info's miscellaneous & unknown
Freespoke
Freespoke, Inc.'s search engine crawler
friendly-spider
Unknown Author's miscellaneous & unknown
Friendly testing bot
Unknown Author's miscellaneous & unknown
FriendlyCrawler
Unknown Author's miscellaneous & unknown
Funnelback
Squiz's search engine crawler
Gabanzabot
Gabanza's miscellaneous & unknown
GDNP crawler
Global Digital Network Plus, LLC.'s miscellaneous & unknown
GeedoBot
Geedo.com's miscellaneous & unknown
getthit.com
GettHIT.com's miscellaneous & unknown
GG bot
Fintecom Sp. z o.o.'s miscellaneous & unknown
Ghost Inspector
Ghost Inspector, Inc.'s automation / testing
Gigabot
Gigablast, Inc.'s miscellaneous & unknown
github-camo
Corey Donohoe's miscellaneous & unknown
GlobalWebSearch
Unknown Author's miscellaneous & unknown
Golfe
Unknown Author's miscellaneous & unknown
Google-AdWords-Express
Google LLC's search engine optimization
Google-Adwords-Instant
Google LLC's search engine optimization
Google-Area120
Google Inc.'s miscellaneous & unknown
Google Favicon
Google LLC's search engine crawlers
Google Images
Google LLC's search engine crawler
Google-Safety
Google Inc.'s search engine crawlers
Google Videos
Google LLC's search engine crawler
GoogleAssociationService
Google Inc.'s search engine crawlers
Grammarly
Grammarly Inc.'s miscellaneous & unknown
gregcrawler
Unknown Author's miscellaneous & unknown
grover bot
Unknown Author's web scraper
HappyWing
Unknown Author's miscellaneous & unknown
harsilbot
Universal Grammar Limited's miscellaneous & unknown
HawaiiBot
Unknown Author's miscellaneous & unknown
hCardValidator
Kornel Lesinski's miscellaneous & unknown
Headless Chrome
Unknown Author's miscellaneous & unknown
Headless Edge
Unknown Author's miscellaneous & unknown
HeadlessChrome
Google's search engine crawlers
headline bot
e.ventures Managementgesellschaft mbH's miscellaneous & unknown
HeartRailsBot
HeartRails Inc.'s miscellaneous & unknown
Hello World
Unknown Author's miscellaneous & unknown
HelloWork
HelloWork Group's other
HIFI
HIFI's other
HSTS preload
Chromium's other
HubSpot Crawler
HubSpot's other
Huckabuy
Huckabuy's search engine optimization
Hype Machine
The Hype Machine, Inc.'s search engine crawler
hyperhost.ua crawler
HyperHost Ukraine's miscellaneous & unknown
iaskspider
SINA Corporation's search engine crawlers
IBM Crawler
Unknown Author's miscellaneous & unknown
ICC Crawler
NICT's ai crawler
ImageMind
Unknown Author's miscellaneous & unknown
ImagesiftBot
Castle Global, Inc.'s miscellaneous & unknown
impendoom-bot
Mohammad Mustakim Ali's miscellaneous & unknown
IndeedJob
Indeed's aggregator
inetdex bot
Cyberserver Ltd.'s miscellaneous & unknown
Instapaper
Instant Paper, Inc.'s other
IonCrawl
Ion Channel's search engine crawler
IONOS bot
IONOS SE's miscellaneous & unknown
IPIP.NET crawler
Beijing Tiantexin Tech. Co., Ltd.'s miscellaneous & unknown
ips-agent
VeriSign, Inc's miscellaneous & unknown
jobswithgptcom-
jobswithgptcom's search engine crawler
Jugendschutzprogramm-Crawler
JusProg e.V.'s miscellaneous & unknown
Kagi
Kagi Inc.'s search engine crawler
Kangaroo Bot
Kasada's miscellaneous & unknown
kazbtbot
caspicat's miscellaneous & unknown
KeyCDN bot
proinity GmbH's cloud & cdn bots
keys-so-bot
Unknown Author's miscellaneous & unknown
KOCMOHABT bot
PAGEPEEKER SRL's miscellaneous & unknown
Kukei.eu-Bot/0.2
Szymon Nowicki's miscellaneous & unknown
LastModBot
Pavel Bychko's miscellaneous & unknown
Level9Search
Level 9's search engine crawler
line-poker
Line Corporation, Ltd.'s miscellaneous & unknown
LinerBot
Aurum Planet Co., Ltd.'s miscellaneous & unknown
Linespider
LINE Corporation's search engine crawler
LivelapBot
LiveLap's miscellaneous & unknown
MADBbot
MADB Project's miscellaneous & unknown
magicsearchdev
Unknown Author's miscellaneous & unknown
Magnetme
Magnet.me B.V.'s aggregator
Mail.Ru bot
VK Group's miscellaneous & unknown
Mail.RU_Bot
VK Company Limited's search engine crawlers
Marginalia Search
Marginalia's search engine crawler
marketgoo
marketgoo's search engine optimization
Mars Finder
Mars Flag's search engine crawler
Matrix.org crawler
Matrix.org Foundation's miscellaneous & unknown
Mattermost-Bot
Mattermost, Inc.'s miscellaneous & unknown
MB-SiteCrawler
Marco Beierer's miscellaneous & unknown
mbot bot
Unknown's miscellaneous & unknown
MediaWiki crawler
Wikimedia Foundation Inc's miscellaneous & unknown
MelonMesa
unknown's other
mindUpBot
Intelligence GmbH's miscellaneous & unknown
MirrorWebCrawler
MirrorWeb Ltd's aggregator
MojemojekBot
Mojeek Ltd.'s search engine crawlers
Monibot
Unknown Author's miscellaneous & unknown
Monsidobot
Monsido, Inc.'s miscellaneous & unknown
more bot
Larry Hamel's testing & qa crawlers
moreover bot
Moreover Technologies, Inc.'s miscellaneous & unknown
MotoMiner
Motominer's aggregator
MRG
MRG Web Services srl's search engine crawler
MSN
Microsoft's search engine crawler
my-tiny-bot
Unknown Author's miscellaneous & unknown
MyBot
Unknown Author's miscellaneous & unknown
nbertaupete95
Unknown Author's miscellaneous & unknown
Neeva
Neeva, Inc.'s search engine crawler
netEstate Crawler
netEstate GmbH's miscellaneous & unknown
netEstate Imprint Crawler
netEstate GmbH's ai crawler
Nextcloud Crawler
Nextcloud GmbH's miscellaneous & unknown
ng bot
Various developers's miscellaneous & unknown
Nicecrawler
Unknown Author's miscellaneous & unknown
Nitro
NitroPack Ltd's search engine optimization
nomore404.com robot
nomore404.com's miscellaneous & unknown
Nota
Helpfeel's search engine crawler
Notion bot
Notion Labs, Inc.'s miscellaneous & unknown
Novaact
NICE Actimize.'s miscellaneous & unknown
Novellum AI Crawl
Novellum's ai crawler
Nutch
Apache Software Foundation's search engine crawlers
nvdorz
Unknown Author's miscellaneous & unknown
Odin
Cyble, Inc.'s miscellaneous & unknown
omgilibot
Webz.io's miscellaneous & unknown
Online Domain Tools
AITIS s.r.o.'s miscellaneous & unknown
Online Webceo
WebCEO's search engine optimization
OpenindexSpider
Openindex.'s search engine crawlers
Orbbot
's miscellaneous & unknown
OrgProbe
Unknown Author's miscellaneous & unknown
PanguBot
Pangu Lab's miscellaneous & unknown
Panscient crawler
Panscient, Inc.'s miscellaneous & unknown
Paqlebot
Paqle A/S's miscellaneous & unknown
PDF24 crawler
geek software GmbH's miscellaneous & unknown
PhantomJsCloud.com
Novaleaf Software Co. Ltd.'s miscellaneous & unknown
Photon
Jonas Ohlsson Aden's miscellaneous & unknown
PhxBot
Unknown Author's miscellaneous & unknown
Potions
Potions's search engine crawler
PrintFriendly.com
PrintFriendly Team's miscellaneous & unknown
PS_Daily
Unknown Author's miscellaneous & unknown
PWABuilder
Microsoft's other
qbot bot
Unknown's web scraper
Qualified
Qualified.com, Inc.'s ai crawler
Rakuten Image extraction
Rakuten, Inc.'s other
Rambler bot
Rambler&Co's search engine crawlers
RavenCrawler
Raven Internet Marketing Tools's miscellaneous & unknown
REDbot
Mark Nottingham's miscellaneous & unknown
remove.bg
Kaleido AI GmbH's miscellaneous & unknown
Retool
Retool, Inc.'s other
Revvim
Revvim's search engine optimization
Riddlerbot
F-Secure Corporation's miscellaneous & unknown
RobotsChecker
Open Rights Group's miscellaneous & unknown
s4a
Capsule Code SAS's miscellaneous & unknown
SafeDNSBot
SafeDNS, Inc.'s miscellaneous & unknown
SandobaCrawler
SANDOBA's miscellaneous & unknown
Searcherweb
Unknown Author's miscellaneous & unknown
Searcherxweb
Unknown Author's miscellaneous & unknown
SearchExpress
Unknown Author's miscellaneous & unknown
SEBot-WA
Unknown Author's miscellaneous & unknown
seeker bot
Various developers's utility & game bots
Seekport Crawler
SISTRIX GmbH's search engine crawlers
SeekportBot
Seekport GmbH's search engine crawlers
SentiBot
SentiOne's miscellaneous & unknown
ServerHunterSpider
Server Hunter's other
Sidetrade crawler
Sidetrade SA's miscellaneous & unknown
silk bot
Slider.com's miscellaneous & unknown
Sincera bot
Sincera Technologies, Inc.'s miscellaneous & unknown
SkyworkSpider
Unknown Author's miscellaneous & unknown
SMTBot
SimilarTech Ltd.'s miscellaneous & unknown
SMTnet PM
SMTnet's search engine crawler
Sohu
Sohu's search engine crawlers
Sora Caisse POS
Sora Websoft's other
Spawning bot
Spawning, Inc's miscellaneous & unknown
Spider
Unknown Author's miscellaneous & unknown
SpiderLing
Natural Language Processing Centre's miscellaneous & unknown
Splash
Zyte's miscellaneous & unknown
SSSSBot
Unknown Author's miscellaneous & unknown
startpage bot
Startpage BV's search engine crawlers
Stata
StataCorp LLC's aggregator
StatistikAustria
Statistik Austria's other
StatsDrone
StatsDrone Inc.'s aggregator
SurdotlyBot
Sur.ly's miscellaneous & unknown
Swisscows Crawler
Swisscows AG's search engine crawlers
t3versionsBot
Torben Hansen's miscellaneous & unknown
TactiScout
Philippe Vincent's miscellaneous & unknown
tchelebi
Black Kite Inc.'s miscellaneous & unknown
test-bot
Unknown Author's miscellaneous & unknown
Test crawler
Unknown Author's miscellaneous & unknown
TestLocally
WonderNetwork's miscellaneous & unknown
Testomatobot
Testomato s.r.o.'s miscellaneous & unknown
TestURI crawler
testuri.org's miscellaneous & unknown
TheInternetSearch
Unknown Author's miscellaneous & unknown
Trellis-Services
Mediavine's search engine optimization
trentwiles bot
Trent Wiles's miscellaneous & unknown
um-LN
ThreatMetrix's miscellaneous & unknown
uMBot
uberMetrics Technologies GmbH's miscellaneous & unknown
Unknown bot
Unknown Author's miscellaneous & unknown
Unshorten.It
Cameron Gray's miscellaneous & unknown
upday
upday GmbH & Co. KG's aggregator
useeBookChecker
usee.pl's miscellaneous & unknown
vagabondo bot
Wise-Guys / Bilbo's miscellaneous & unknown
vebidoobot
vebidoo GmbH's miscellaneous & unknown
videootv
Digital Green's aggregator
voltron bot
Unknown Author's miscellaneous & unknown
VsuSearchSpider
Unknown Author's miscellaneous & unknown
W3 Validator Services
World Wide Web Consortium's search engine optimization
W3C_CSS_Validator
World Wide Web Consortium (W3C)'s miscellaneous & unknown
WARD
WEBSPARK's ai crawler
Webflow bot
Webflow, Inc.'s miscellaneous & unknown
webprosbot
WebPros's miscellaneous & unknown
WebSniffer
Karen Tamrazyan's miscellaneous & unknown
WebSpiderMount
AspenTechLabs Inc's aggregator
Webzio
Webzio is operated by Webz.io.'s miscellaneous & unknown
WellKnownBot
Braedon Vickers's miscellaneous & unknown
wesee bot
WeSEE Ltd's miscellaneous & unknown
WGETbot
Alan Reed's miscellaneous & unknown
WhatsMyIP.org
WhatsMyIP.org's miscellaneous & unknown
Wheregoes bot
Adfluent Media Inc.'s miscellaneous & unknown
WMF Citoid
Wikimedia Foundation's other
WMF Zotero Translation Server
Wikimedia Foundation's other
WordPress Namecheap
Namecheap, Inc.'s other
Wordup-1
Unknown Author's miscellaneous & unknown
Workona bot
Workona Inc.'s miscellaneous & unknown
WorldBot
Unknown Author's miscellaneous & unknown
wpbot
Unknown Author's miscellaneous & unknown
WRTNBot
Wrtn Technologies Co., Ltd.'s miscellaneous & unknown
XMCO bot
XMCO's miscellaneous & unknown
yacybot
Michael Christen's miscellaneous & unknown
Yahoo! JAPAN
LY Corporation's search engine crawler
Yahoo!
Yahoo! Inc.'s search engine crawlers
YouBot
SuSea, Inc.'s miscellaneous & unknown
Yuuperbot
Unknown Author's miscellaneous & unknown
_zbot
Unknown Author's miscellaneous & unknown
zeno bot
ListPremier's miscellaneous & unknown
Zombiebot
Zombie Domain's miscellaneous & unknown
Zoom crawler
Zoom Video Communications, Inc.'s miscellaneous & unknown
Zum
ZUM internet Corp.'s search engine crawler
zzhbot
Unknown Author's miscellaneous & unknown